VR Cinema for INVR.SPACE – 25 Films, Five Audio Formats, One Room

A VR cinema is a room full of people all watching the same film and yet not the same thing. Everyone wears a headset and turns their head wherever they like, and that is precisely where the appeal of the format lies.

For the sound it is a problem. At the end of 2018 around twenty-five 360° films from as many productions came together at INVR.SPACE, all of them meant to run in one room. Not one of them sounded like the next. I brought them into a shape the system in the room can handle.

One of the twenty-five films, shown above in the title image: a flight simulator of the Dornier Do X, which I also did the sound for.

Why a VR cinema takes the sound apart instead of blowing it up

Spatial sound for 360° video is built for headphones and head tracking. The sound field sits fixed in the room, the viewer turns inside it, and the software recalculates for every head position so that the sources stay put while you move.

In a cinema exactly that part falls away. The picture stays individual, everybody looks into a different corner, while the sound becomes collective, because there is one set of speakers in the room playing for everyone at once, and it cannot be turned differently for twelve heads.

That reverses the direction of the work. Normally I take a mix from a fixed format into a movable one. Here every movable sound field had to become a fixed floor, a 5.1 bed that stays put while the audience turns.

In passing, a track that is usually the afterthought gains unexpected weight. Many 360° mixes carry a second, head-locked stereo track next to the spatial field, for music and narration, for everything that is not supposed to sit in the room anyway. In a cinema that is exactly the part which already works correctly by itself. Everything else has to be carried there.

How five formats became one room

What arrived was the full range of what was in circulation in 2018. Eight-channel TBE from Two Big Ears, often with the head-locked stereo track inside the same file. Four-channel AmbiX, the format YouTube accepts. Finished binaural, that is a headphone version that can no longer be taken apart for loudspeakers. And plain stereo, on some films as the only version at all.

The first step was manual and fairly thankless. Many films came as MP4 files with several audio streams sitting next to each other, eight channels here, two channels there. You cannot simply gather channels from different streams into one file in a single pass, the standard tools do not allow it. So you separate the streams first, merge them again afterwards, and check every time whether the second track has any content at all, because it is often simply empty.

Then came the decoding. The eight-channel sound field becomes a 5.1 bed with left, right, centre, low frequency and the two surrounds. With binaural versions that is no longer possible, and all that remains is to place the existing stereo version cleanly in the room and be honest that there is no more in it. A format is only ever as good as the playback path it finds in the end.

Then there was the question of level, which matters more across a series of films than for a single one. Twenty-five productions by twenty-five teams are twenty-five different ideas of how loud a film may be. Run that unchecked one after another and the audience sits alternately in a whisper and a pneumatic drill, and after the third jump somebody turns the system down, which makes the rest of the evening too quiet. So I related the versions to one another and recorded for each film where it lands.

Finally the part you must not skip. I ran the EBU’s BLITS test signal, a signal that announces channel by channel where it is currently playing, along with a loudspeaker file of my own. Only after that do you know whether what sits rear right in the file really is rear right in the room.

On a single film a swapped surround pair goes almost unnoticed. Across twenty-five films in a row the audience notices immediately, because the error repeats with every title.

Then there is loudness, which matters more across a series than on a single film. Twenty-five productions from twenty-five teams are twenty-five ideas of how loud a film should be. With one title you nudge the room fader. Across a series you cannot, because nobody adjusts between films, and a quiet entry straight after a loud one simply does not land. So a common loudness had to be laid across every version before anyone entered the room.

What came out of it

In the end a series of around twenty-five 360° films ran in one room, all on the same system, all in a uniform 5.1 bed, all at comparable loudness. For the audience that is invisible, and that is exactly the success: nobody notices the films came from five different audio formats.

What I took from this work concerns sequence more than technology. A film rendered as a finished headphone version cannot be taken apart again. Whoever knows while mixing that the film might one day run on loudspeakers delivers the sound field as well and keeps every route open. Whoever only has the finished binaural version made that decision long ago without noticing.

The same pattern turns up with domes and permanent installations: think spatially once and the same mix scales from headset to auditorium. What that looks like on a fulldome production is shown by Emergence in Nature at the Deutsches Museum, and on a single film with many deliverables by the case on Bauhaus in Bavaria.

Planning a film series or a VR cinema?

As soon as several productions have to run in one room, the work is decided by the formats and not by the individual mix. Let’s talk early about what has to be delivered.