A History of VR – Sound Design for an Interactive VR Prototype

Telling the history of VR inside VR is an obvious idea and a tricky one. A museum can put up display cases, a film can show archive footage. Inside a headset the audience sits in the middle of it and has hands that want to grab things rather than listen.

I did the sound for the prototype “A History of VR”. The narrator of this piece is not a person but reality itself, a voice with opinions and a short temper. You never see her. You see a light that swings with her voice.

Why a narrator without a body makes sound the lead role

That inverts the usual order. In a film the narrating voice is a comment over pictures that would work without it. Here the voice is the character, and the picture is only its display: a constellation whose glow follows the spoken word, an equaliser with a personality. What happens in the recording therefore decides what you see.

On top of that, this narrator reacts to you. She asks you to touch a floating object. If you do not touch it, she gets impatient. Touch it a second time and a different take plays, because she is slowly getting tired of it. Put the object down and her comment breaks off mid-sentence, with a new one starting as soon as you pick something up again.

For the sound that means something very concrete. In a film I mix a sequence that always runs the same way. Here there is no sequence, only a set of lines, each of which has to be able to start, switch or break off at any moment. A line that only works in context is useless inside an engine. Why that way of thinking from game audio is now reaching film as well, I wrote up in how game audio influences film sound design. It is the same difference that shaped the whole job on the VR prototype about the Dornier Do X flying boat.

How fifty takes became a character

I recorded the narrator in June 2021 with an actress, in roughly fifty takes. The script is not a text to be read out but a collection of reactions. “Go on, be brave.” “Are you scared?” “Wow, you really are a slowcoach.” Lines like that sit in the script without a fixed place, because only the player decides whether they come up at all.

For the recording that means a different way of working than a voice-over. Every line needs its own beginning and its own end, because it has to start cold and stop hard. You cannot record a continuous monologue and cut it up afterwards, because the phrasing then tells of a context that does not exist in the piece. So you record line by line and make sure the character still stays the same person across all of them.

Alongside the human voice there is a second one that comes out of a signal generator. It is the counterpart: mechanical, even, without attitude. Only the contrast makes it audible how much the narrator is a character rather than an announcement.

A second problem comes from the fact that nobody knows what else is running at the time. When the narrator starts speaking while the player drops a coin into a bowl and music plays underneath, she still has to remain intelligible. In a film I solve that in the mix, because I know what is layered on what. In an engine the rule has to be fixed in advance and hold for every conceivable combination. In practice that means a clear order of who gets out of whose way, and effects laid out from the start so that they leave the speech range free.

The mix itself was half localisation. The voice has to come out of the light, otherwise the idea collapses immediately. Sitting neutrally in the head, it is a comment from outside again, and the constellation becomes decoration. Coming audibly from where the light is, the audience turns towards it by itself, and the character exists.

The music carried its own curve tied to the three scenes. For Morton Heilig’s workshop the direction was swing and jazz of the fifties and sixties, for the time tunnel synthesizers in the manner of Stranger Things. The main theme came from Simon Michel, I spread it in the room and fitted it into the transitions.

What came out of it

The result is a playable prototype for the Meta Quest 2 with three scenes. You start in a black room under a starry sky, land in Morton Heilig’s workshop in front of the Sensorama of 1962, drop in a coin, look inside and get pulled into a time tunnel where a TRON bike, a Virtual Boy and an arcade cabinet fly past and are explained on the way.

The Sensorama at the centre of this story is incidentally the most fitting object anybody could pick for a project like this. Morton Heilig built the machine in the early sixties and had it patented in 1962, and from the start it was more than a picture device: stereoscopic image, stereo sound, wind, vibration in the seat and smells. Sound was a matter of course there long before anybody used the words virtual reality. That you drop in a coin and look inside while a voice watches you do it closes a rather nice circle.

Fourteen separate sound elements hang off these three scenes, from the coin drop through the spotlight change to the pull into the tunnel. That sounds like very little for a film. For a prototype that had to stand within a month it is a lot, because each of them has to do more than sound right: it has to be triggered, interrupted and picked up again.

The prototype served as the basis for the funding application to the FilmFernsehFonds Bayern, for which I contributed a letter of intent as sound partner. The application succeeded: FFF Bayern approved “A History of VR (in VR)” on 24 November 2020 with €30,000 in project development funding. Vertical slices like this are the usual currency in XR production: a short but complete stretch that demonstrates how the whole project would work.

It was made by the straightlabs team around Fabrizio Palmas, together with Matthias Leitner and Christian Schiffer, design by Marcel Rühmer, music by Simon Michel. I was responsible for the voice recording, sound design, mix and localisation.

Planning a VR prototype for a submission?

A prototype has to convince on the first run-through, with people who do not know the project. Let’s talk early about which sounds have to sit as separate assets for that.