Thanks to VR Videos, it is now necessary to ask ourselves things, that seemed to be clear for ages – using music for example. The problem: if you hear music in a 360° space, you should see musicians somewhere in the frame, right? So if you use it anyways, is the music supposed to rotate with the viewer’s line of vision, or should it stay as a static stereo?
These questions can not be answered that easy because they depend on a lot on the subject. That’s why I want to discuss the typical scenarios.
Note: This article is based on linear content only. Music for interactive VR content is a completely new topic that I will have a look at sometime.
spatial or non-spatial – that is the question
The simplest case is probably the recording of a live performance. You can see musicians, usually on a stage, so it is pretty clear, from which direction the virtual reality music is supposed to come from, while the applause is coming from the audience for instance.
But be careful with realism: In reality, the sound source would be the loudspeakers. But here, it would make even more sense, if the sound of a guitar, for example, can be located from the actual position of the guitarist.
Music videos are like icing on the cake and are constantly popular among listeners. The new trend is – who would have thought – 360° Videos. Out of personal experience, I can say: for good reason! With VR goggles in front of your eyes and music on your ears, you can really dive even more into music. It is a bit like you close your eyes and get lost in the music. But this time, imagination gets beaten by a video that shows where the visual journey goes.
Here it is hard to tell if a static stereo-mix or an immersive mix would be the right way to go for virtual reality music.
It is obvious to go for the mix that is also running on the radio. For me, this is one of the few occasions, where virtual reality videos may resign spatial sound. The advantage is, that the musical piece will always sound like as it was supposed from the producer as there is only one version of the track that is supported by every platform and device.
This immersive version of the song requires an extra mix, which increases the cost and effort of the production. I just realize that I would have to do an entire blog post on that subject to really highlight all (dis-)advantages. But for now, you can form your own opinion with the following example:
For this song, the effort was taken to create an individual mix for all kinds of VR-platforms. To be exact, YouTube with Ambisonics, and JauntVR with Dolby Atmos VR. But of course, there is also the original Stereo-Mix on YouTube.
So, do your own comparison, with JauntVR on GearVR and the YouTube-App on Smartphones or Chrome-Browser on desktop devices.
Personally, I like the sense of depth and space you get from a spatial sound mix, because the rotation of the sound supports the movement of the pictures and therefore improves immersion. But I also enjoy the static stereo version. It feels somehow closer in your head, just like you know it from your walkman.
There is a lot that can be said about this topic, but I want to break it down into core facts.
Using film scores in 360° videos must be re-thought from the base because virtual reality music can rarely be reflected by a visible sound source (musicians, loudspeaker, monitors, etc.)
But you can take the easy way out, and use film scores just like it is being done for non-VR video. The audience has learned from the cinema, to be emotionally guided by the music and does not think “I hear an orchestra, so where is Hans Zimmer?!”.
For me, I mostly decide to be spare on using music, but it really makes sense if you have a couple of 360° shots in a row where music will not overload the mental impressions of the viewer. For this, I stay with plain stereo. As a composer, I usually don’t want the listener to have any influence on virtual reality music. But there is also the possibility to direct the attention with music in excess of sound. For instance with individual instruments that are positioned in 3D space and can, therefore, be located.
There is a last determination I want to make. Mixing music in surround (5.1, Dolby Atmos, Auro 3D etc.) has been common for quite some time. But the approach to mixing cannot easily be adapted to VR content. This is the reason why:
For surround mixes, you usually have only one fixed line of sight because the audience usually sits in the middle of a surround loudspeaker setup with a screen in front of it. So it is common to please the vocals or snare drum for example in the center so it sounds as if it is in front of the listener. But if you carry this mix over to 360° playback on headphones, it would lead to the case that e.g. the vocals can only be heard on one ear as soon as you rotate the head by 90°.
To simplify the consequences it can be said that for virtual reality music the individual instruments should be placed more carefully in 3D because they can easily start to stick out as soon as head-tracking is used.
There are no general answers. They really depend a lot on the scenario. Personally, I prefer music in static stereo because it keeps the music production classic and does not introduce a lot of surprises. However, there is still a lot of unexplored space and for myself decided to experiment more with virtual reality music and I’m excited what follow content creators will come up with the new possibilities.More 3D music infos
Music moves virtual spaces - VR Music
360 Music Video - Why the 3D audio potential was never reached?
ByteFM Interview Part 1: The Three Pillars of 3D Music in virtual reality
ByteFM Interview Part 2: Climate and Accessbility with Music in VR
ByteFM Interview Part 3: Interactive and Immersive Game Music