Virtual Reality Sound – Benefits you must know

Virtual reality places entirely new demands on the sound, be it from a technical or creative point of view. Simplified it can be said that the virtual reality sound is supposed to create a three-dimensional audio impression, although it is usually only played back via headphones.

It can be guessed that this is not at all easy, although I don’t want to go into too much detail with technical terms like “binaural” and “Head-Related Transfer Function”. In order to get a good overview of the sound of 360° videos, the most important facts are summarized here.

What is VR audio?

VR audio is sound that surrounds you as a sphere instead of coming from a flat stereo image. In a headset it plays over headphones, and binaural rendering with head tracking keeps every sound in its place when you turn your head, so you hear what is behind you before you see it.

Once you know the benefits, the next step is choosing the right format: see virtual reality audio formats.

How does sound work in VR?

In VR the sound is almost always played back over headphones and has to produce a three-dimensional impression from two channels. Binaural rendering supplies the directional cues, object-based audio positions individual sources in the scene, and head tracking keeps that scene in place when the viewer turns their head.

Which formats does VR audio use?

The two formats you meet most often are binaural stereo and Ambisonics. Binaural stereo plays on every platform like a normal stereo file, but it only gives you the sound of one fixed viewing direction and no head tracking.

Ambisonics has 4, 9 or 16 channels depending on its order. It carries a full sound bed and can add head-locked audio, but it has no objects, so it always sounds a little diffuse. An optional head-locked stereo track carries voice-over or music that should not turn with the scene. YouTube still plays Ambisonics for 360° video, although that support has grown thinner year by year.

Overlook, yes – Overhear, no!

A decisive difference between the visual and auditory levels is the following fact: video elements can easily be missed, but sound elements not. Only the feel of direction varies, which means that the sound is the only way to subtly lead the viewer in one direction. You direct its gaze to the important elements in the picture and thus you can ensure that no major details of the scene remain undiscovered.

This approach is particularly interesting if the 360° video concept takes this point into account in advance, and like that creates completely new ways of playing the plot, which is essential for storytelling in virtual reality.

It doesn’t have to be a copy of reality

Since the beginning of surround formats, the goal has often been to create a sound image that is as close to reality as possible for the listener. Virtual reality sound, in particular, seems to be also ambitious about it.

But I disagree. Through the targeted guidance of the viewer’s attention, it becomes clear that it is not always ideal to take reality as a guide because too many sounds quickly overload the viewer. This way, it can be decided what is important for a scene, influencing the line of vision and not distracting attention with unimportant sounds.

In 99 out of 100 cases it is enough that you hear what you see. The more important question is whether a sound should be spatial at all.

Spatial Audio isn’t just “nice to have”

Cinema-goers are ready to enjoy a surround-sound experience at a good price. But whether it’s stereo or surround, at the end of the day it’s a nice add-on (and I say that as an enthusiast for immersive audio). However, this setting is not transferable to 360-degree videos, since the entire surroundings of VR are portrayed in spherical form, and not, as in the cinema, only a section of the scene is shown.

On the contrary: the stereo sound can actually be noticed as a broken sound, which can destroy the immersion that one has tried to build up with the image. There are very few good reasons to fall back on static sound for virtual reality sound experiences and like that it is very obvious that the potential of the medium has not been exhausted by far.

Sure, that’s what the sound guy says, but imagine this: You wear a VR headset and turn your head: the image rotates with head-tracking, but the sound does not… even the layman notices quickly, something is not right here 😉.

VR audio in practice: #EUsavesLives in Kakuma

For the European Commission I recorded the sound for a 360° documentary in the Kakuma refugee camp in Kenya, in a team of four people at 40 °C. Wireless microphones captured the protagonists, Ambisonics microphones the sound field around them. The clip reached almost 5 million views on Facebook 360.

Let’s talk about your project

Are you working on a VR experience and the audio still feels flat?