Technologies

Apple iPhone Spatial Video Arrives in Beta and Looks Amazing on Vision Pro

I tried the new iPhone 15 Pro camera feature on Apple’s upcoming AR-VR headset. The 3D is undeniably vivid.

I’m looking at a plate of sushi hovering in front of me in 3D. The chef finishes off toppings on yellowtail rolls and tuna, talking to me as she works. It looks vivid. It looks real. The amazing part is that I just shot this video myself, moments earlier, on an iPhone 15 Pro. And now it’s a VR experience I’m watching in beautiful 3D on Apple’s Vision Pro headset.

I swipe with my fingers and watch other ghostly videos Apple provided. Families in a home, walking through grass. Cuddling together. All in lifelike 3D. I feel like I’m peeking in on their lives, which is weird and intimate. But the vividness is undeniable.

I’m experiencing Apple’s new Spatial Video iPhone camera feature, now available in Apple’s new iOS 17.2 public beta, with the final version rolling out in 17.2 later this year. It allows you to record 3D videos, but to use it you’ll need Apple’s best phone, the $1,000 iPhone 15 Pro. And to watch the videos in 3D you’ll need the $3,500 Vision Pro headset, which launches early next year.

The videos look great and the 3D is compellingly realistic. They’re also easy to record, and can save as videos that will play back in 2D in a normal video format. But ultimately this feature is made for a 2024 product that, at $3,500, it’s safe to say most people won’t buy anytime soon. Still, the experience is impressive.

Vision Pro impressed me once again

I first used the Vision Pro at Apple’s WWDC conference in June, and once again I’m reminded of its extraordinary display capabilities. Photos and video look fantastic on Apple’s headset. I’d even venture to say the Vision Pro beats any display I have anywhere in my home right now. Yes, I’d want to look at videos and photos — 3D and otherwise — if I had a Vision Pro.

My second dive into the Vision Pro, complete with fitted prescription lenses that matched my needs, reminds me how much smaller the headset was than I remembered. And also, how effortless the interface is.

Setting up eye tracking was a rapid process; I looked at dots around a circle and tapped them. I also tried a new set of zoom gestures, pinching my fingers and stretching them to expand photos. The gesture works with eye tracking, so wherever I looked, the image expanded. It felt like telepathy. I got to look through a few test photos in the Photos app, and play back photos in Memories. After having used a Quest 3 recently, Apple’s passthrough cameras and display resolution are on another planet.

Panoramic photos were a surprise. They opened up and wrapped around me, felt like windows into other places — almost 3D, in fact. And spatial videos look nice, too. Their 3D, almost ghostly playback quality feels like it’s aiming for an immersive memory more than a straight-up video playback. But I found some limits even in my brief demo of the experience.

Unfortunately, there aren’t any captures of what I saw or recorded here: the photos are all ones Apple provided, so you’ll have to just follow along and read about my experience.

02-iphone-15-pro-cameras-for-spatial-video

Spatial videos on iPhone 15 Pro: How they work

Spatial videos need to be recorded by holding the phone horizontally. The iPhone 15 Pro uses its main and ultrawide cameras side-by-side in landscape mode to create the 3D video. It records two 1080p, 30fps videos at once, resolving the lens and distance differences with computational photography. The file is saved in HEVC format, similar to other Apple video files, with a storage size of about 130MB a minute. That’s not too bad, likely because the files are limited to that relatively low 1080p and 30fps resolution and framerate.

Spatial video clips can be shared via Messages or AirDrop, but there could be cases where files are compressed down in other apps, losing the 3D data. Still, the video clips should be usable anywhere as an everyday video file, which is great news for anyone who wondered (myself included) whether you’d have to choose a «special» 3D video format.

You do have to toggle Spatial Video mode on, either inside Apple’s Camera app settings or with a Vision Pro-mask-shaped toggle in the Camera app itself. That limits your video resolution and frame rate, and also means you have to shoot in landscape. Sorry, no 3D vertical videos.

The camera app makes recommendations on turning the camera sideways, and staying a certain distance from a subject. I was told to stay within 3 to 8 feet of what I was shooting for a good spatial video, but when I shot my test recording of someone making sushi at a table I got up closer and it looked perfectly fine. I also recorded in a well-lit room, but apparently the spatial video recording mode prevents adjustments on brightness and contrast, which means low-light recording may end up grainier than normal videos.

03-spatial-video-capture-on-iphone-15-pro

How will spatial video evolve or be edited?

There’s no API for Apple’s 3D HEVC video format, which means the files are not designed so that third-party apps can recognize them yet. But it’s possible app developers will figure out some sort of a solution. Also, it’s disappointing that there’s no other way to view the videos in 3D than on a Vision Pro.

The stereo 3D recording theoretically sounds like a process that could be adapted for other VR headsets like the $500 Meta Quest 3. It’s straightforward stereo video, and doesn’t use lidar or depth sensing for any sort of full-movement spatial capture. Unfortunately that’s not currently in Apple’s plans.

There’s no recording limit on the videos, so you could theoretically make a spatial video 3D feature-length film. But the clips will only be able to be trimmed, either in the iPhone or Vision Pro. Final Cut Pro, Apple’s video editing software, will get Spatial Video editing support, but not until sometime next year. Meanwhile if you do end up trying to edit these video files in a Mac or iOS video editor, they’ll end up being converted over into 2D-only files.

It’s also a tiny bit disappointing that the videos can’t be recorded in 4K resolution. Apple’s Vision Pro headset has astounding quality and resolution, which became clear once again as I looked at iPhone photos in the headset and zoomed in on them, or viewed panoramic photos in a wraparound mode that made it feel like I was in a vivid immersive recreation of a location, similar to a 360-degree photo. The spatial videos look really nice, but I felt the desire to see them in more fluid 60fps, 4K or both. Maybe someday.

Memories for a 3D future

Apple often seems like it’s trying to become not just a lifestyle, but a memory company. Automated photo collections are already called «Memories.» And that’s how 3D spatial videos feel on Vision Pro: hazily bordered, like digital recollections in some Minority Report or Bladerunner future. It reminded me, too, of some of the immersive 3D realism I had using Google’s real-time 3D light field conversations in Project Starline, but in this case I was just watching videos.

Apple’s choice to frame the spatial videos in a fuzzy sort of frame makes it look more like the videos are almost holographically being cast into the room, and the edges sort of dissolve a bit. But I’d love to see a way for the videos to have a standard frame around them, too.

Apple’s spatial video format shows up in a new category of its own on the Photos app, and won’t show up in Memories yet. Maybe this new spatial video tab in Photos becomes the go-to repository for new 3D experiences. Right now, Apple isn’t allowing 3D spatial photos on the iPhone 15 Pro, just videos, although the Vision Pro can capture photos itself. But Apple’s clearly aiming for everyone to hopefully start recording in the format before the headset arrives, building a library of capable videos.

That means you’ll have to remember to toggle spatial video on, and care to use it. Future Vision Pro owners will have that feature toggled on by default on the iPhone 15 Pro, but that’s a tiny portion of iPhone owners. I’ll be turning it on, because I want to see what my videos will look like next year on a Vision Pro. But unless you’re planning on buying a Vision Pro yourself, there’s little reason to record spatial video now, even though it’s possible, someday way down the road, you might regret not shooting that birthday party footage in 3D. I’m already thinking about I shot video footage at my niece’s bat mitzvah last week, and what it would have been like if I could see it in 3D, too. That new spatial video camera toggle is already playing on my future FOMO.

I Took 600+ Photos With the iPhone 15 Pro and Pro Max. Look at My Favorites

+31 more

See all photos

Technologies

Google races to put Gemini at the center of Android before Apple’s AI reboot

Google is using its latest Android rollout to position Gemini as the AI layer across phones, Chrome, laptops and cars.

Google is using its latest Android rollout to make Gemini less of a chatbot and more of an operating layer across the phone, browser, car and laptop, just weeks before Apple is expected to show its own Gemini-powered Apple Intelligence reboot at WWDC.
Ahead of its Google I/O developer conference next week, the company previewed a number of Android updates, including AI-powered app automation, a smarter version of Chrome on Android, new tools for creators, a redesigned Android Auto experience, and a sweeping set of new security features.
Alphabet is counting on Gemini to help Google compete directly with OpenAI and Anthropic in the market for artificial intelligence models and services, while also serving as the AI backbone across its expansive portfolio of products, including Android. Meanwhile, Gemini is powering part of Apple’s new AI strategy, giving Google a role in the iPhone maker’s reset even as it races to prove its own version of personal AI on the phone is further along.
Sameer Samat, who oversees Google’s Android ecosystem, told CNBC that Google is rebuilding parts of Android around Gemini Intelligence to help users complete everyday tasks more easily.
“We’re transitioning from an operating system to an intelligence system,” he said.
As part of Tuesday’s announcements. Google said Gemini Intelligence will be able to move across apps, understand what’s on the screen and complete tasks that would normally require a user to jump between multiple services. That means Android is moving beyond the traditional assistant model, where users ask a question and get an answer, and acting more like an agent.
For instance, Google says Gemini can pull relevant information from Gmail, build shopping carts and book reservations. Samat gave the example of asking Gemini to look at the guest list for a barbecue, build a menu, add ingredients to an Instacart list and return for approval before checkout.
A big concern surrounding agentic AI involves software taking action on a user’s behalf without permissions. Samat said Gemini will come back to the user before completing a transaction, adding, “the human is always in the loop.”
Four months after announcing its Gemini deal with Google, Apple is under pressure to show a more capable version of Apple Intelligence, which has been a relative laggard on the market. Apple has long framed privacy, hardware integration and control of the user experience as its advantages.
Google’s Android push is designed to show it can bring AI deeper into the device experience while still giving users control over what Gemini can see, where it can act and when it needs confirmation.
The app automation features will roll out in waves, starting with the latest Samsung Galaxy and Google Pixel phones this summer, before expanding across more Android devices, including watches, cars, glasses and laptops later this year.
The company is also redesigning Android Auto around Gemini, turning the car into another major surface for its assistant. Android Auto is in more than 250 million cars, and Google says the new release includes its biggest maps update in a decade and Gemini-powered help with tasks like ordering dinner while driving.
Alphabet’s AI strategy has been embraced by Wall Street, which has pushed the company’s stock price up more than 140% in the past year, compared to Apple’s roughly 40% gain. Investors now want to see how Gemini can become more central to the products people use every day.
WATCH: Alphabet briefly tops Nvidia after report of $200 billion Anthropic cloud deal

Technologies

Waymo recalls 3,800 robotaxis after glitch allowed some vehicles to ‘drive into standing water’

Waymo issued a voluntary recall of about 3,800 of its robotaxis to fix software issues that could allow them to drive into flooded roadways.

Waymo is recalling about 3,800 robotaxis in the U.S. to fix software issues that could allow them to “drive onto a flooded roadway,” according to a letter on the National Highway Traffic Safety Administration’s website.
The voluntary recall is for Waymo vehicles that use the company’s fifth and sixth generation automated driving systems (or ADS), the U.S. auto safety regulator said in the letter posted Tuesday.
Waymo autonomous vehicles in Austin, Texas, were seen on camera driving onto a flooded street and stalling, requiring other drivers to navigate around them. It’s the latest example of a safety-related issue for the Alphabet-owned AV unit that’s rapidly bolstering its fleet of vehicles and entering new U.S. markets.
Waymo has drawn criticism for its vehicles failing to yield to school buses in Austin, and for the performance of its vehicles during widespread power outages in San Francisco in December, when robotaxis halted in traffic, causing gridlock.
The company said in a statement on Tuesday that it’s “identified an area of improvement regarding untraversable flooded lanes specific to higher-speed roadways,” and opted to file a “voluntary software recall” with the NHTSA.
“Waymo provides over half a million trips every week in some of the most challenging driving environments across the U.S., and safety is our primary priority,” the company said.
Waymo added that it’s working on “additional software safeguards” and has put “mitigations” in place, limiting where its robotaxis operate during extreme weather, so that they avoid “areas where flash flooding might occur” in periods of intense rain.
WATCH: Waymo launches new autonomous system in Chinese-made vehicle

Technologies

Qualcomm tumbles 13% as semiconductor stocks retreat from historic AI-fueled surge

Semiconductor equities reversed sharply after a broad AI-driven advance, with Qualcomm suffering its worst day since 2020 amid inflation concerns and rising oil prices.

Semiconductor stocks fell sharply on Tuesday, reversing course after an extensive rally that had expanded the artificial intelligence investment theme well past Nvidia and driven the industry to unprecedented levels.

Qualcomm plunged 13% and was on track for its steepest single-day decline since 2020. Intel shed 8%, while On Semiconductor and Skyworks Solutions each lost more than 6%. The iShares Semiconductor ETF, which benchmarks the overall sector, fell 5%.

The sell-off came after a key gauge of consumer prices came in above forecasts, and as conflict in Iran pushed crude oil higher—prompting investors to shift away from riskier assets.

The preceding advance had widened the AI opportunity set beyond longtime industry leader Nvidia, which for much of the past several years had largely carried the market to new peaks on its own.

Explosive appetite for central processing units, along with the graphics processing units that power large language models, has sent chipmakers to all-time highs.

Market participants are wagering that the shift from AI model training to autonomous agents will lift demand for additional AI hardware. Among the beneficiaries are memory chip producers, which are raising prices as supply remains tight.

Micron Technology slid 6%, and Sandisk cratered 8%. Sandisk’s stock has surged more than six times over since January.