Apple Immersive Video Sound: ASAF Recording and Mixing

An off-road drive is one of the few experiences that is almost entirely physical. You feel the surface through the seat, you hear the suspension take a hit before you see the rock, and the engine alone tells you how hard the vehicle is working. Put that on a flat screen and all of it disappears. The picture survives, the experience does not.

This project was about bringing it back, as an Apple Immersive Video, for a driving experience shot once on a mountain and once on a closed test track. I recorded the sound in the Apple Spatial Audio Format and mixed the individual scenes, including the additional sound design.

Why an off-road drive needs sound first and picture second

The picture side is solved. Stereoscopic 180° video on the Vision Pro is sharp enough, fast enough and stable enough in head tracking that the image stops being the limit. What remains is the part nobody notices while it is there.

Watch a vehicle cross a gravel track with the sound off and you see a car in motion. Add the sound and something else happens. The crack of stones under the tyres tells you the surface is loose. The rattle of the suspension tells you how hard the springs are working. The brakes squeal, the tyres scrub as the rear steps out, the engine howls when the driver asks for it. None of this is decoration. This is the information your body uses to decide whether it is sitting in something robust or something fragile.

That is why the sound here cannot come from a library. A generic gravel loop sounds like gravel. It does not sound like this vehicle on this surface at this speed. The moment those two come apart the illusion breaks, and on a headset it breaks harder than on a screen, because the viewer is sitting inside the scene rather than in front of it.

Apple treats sound in this format as equal to picture, and the Apple Spatial Audio Format gives you the precision to place it accordingly. That only helps if the recordings underneath are worth placing.

How the drive was recorded under extreme conditions and mixed in ASAF

Several vehicles were shot at the same time, on the mountain and on the test track, and the sound had to survive both.

Rigging on and inside the vehicle. Microphones went onto the body and into the cabin with suction mounts, so the same run delivered the exterior and the interior perspective. That matters because the two tell you different things: from outside you hear a machine working, from inside you hear what the driver feels. The finished piece needs both, and they have to line up sample-accurately.

Record in the format, do not convert into it. The material was captured for the Apple Spatial Audio Format from the start. Converting a conventional multichannel mix afterwards costs exactly the spatial precision the format exists for, and on a headset that loss is audible.

Boom on foot. Not everything can be rigged. A water crossing, a stretch of loose ground, a particular tyre sound in a particular corner: that gets recorded from the bank, by hand, as the vehicle passes. This is where the extreme conditions hit. Water, dust, cold, and vehicles that drive a given passage only once or twice.

Perspective from the air. A drone carried the stereoscopic immersive camera rig for the wide passes on the mountain. Sound anywhere near a drone is unusable, so those perspectives are built in the mix from the ground recordings rather than captured with the picture.

Delivering into the Apple ecosystem. The finished mix leaves post production in the Apple Spatial Audio Format and travels in the Apple Positional Audio Codec (APAC). For the Apple Vision Pro that means the object positions survive all the way to the headphones, head tracking computes against the recorded position rather than one estimated afterwards, and a bed of higher order ambisonics carries the environment while the individual objects sit on top of it.

Mixing scene by scene. Every scene got its own mix instead of one global setting. A climb, a water crossing and a fast run on the test track have nothing in common acoustically, and treating them alike is the quickest way to make an immersive piece feel flat. On top of the recordings came additional sound design wherever reality was either unavailable or unreadable: reinforcing an impact the microphone caught too politely, giving a movement a clear direction, clearing a passage so the one detail carrying the moment is not buried under everything else.

What came out of it

The result is an Apple Immersive Video in which the drive is something you sit inside rather than something you watch. Turn your head towards the wheel that just found the rock, and the sound is already there, because the position was recorded and not estimated afterwards.

Three things this gives a brand experience that a conventional film does not:

Capability becomes audible. A spec sheet says what a vehicle can do. The sound of a suspension taking a hit and staying calm shows it, and the viewer draws the conclusion instead of being told it.

The body reacts. Tension in the shoulders on the steep section, a flinch at the first stone strike. Physical reactions stick, and sound triggers them far more reliably than picture.

It works in a controlled space. A headset experience puts the viewer on the mountain from a showroom, a trade fair stand or the shop floor. No test track, no weather, no risk, and the same experience for everyone.

The technical background on the format is at Apple Spatial Audio Format (ASAF) plugins in Pro Tools, and how ASAF relates to other spatial formats is covered at MPEG-H Audio and What is Dolby Atmos.

Planning an Apple Immersive Video or a 360° project?

If you are producing for the Vision Pro and the sound should carry the experience rather than accompany it, that is the part I do.