“Immersive Audio” is More Than Just “3D Music”

Immersive Audio and 3D Music? There is hardly a word in the audio scene that is as hyped as “immersive”. Yet “immersive audio” is often used as a synonym for “3D audio” and many people think only of music.

This undercuts the fact that even mono playback can be immersive with a stirring story. And even music in good old stereo very much invites you to close your eyes and immerse yourself in a world of sound. 

Even more, the fuss about 3D music aka Dolby Atmos Music and Sony 360 Reality Audio undercuts the following: Namely, in my immersive audio world, both audio contents are just pretty nice gimmicks! Yes, I was allowed to hear very nice 3D music mixes from colleagues at the Tonmeistertagung. But what is currently being mixed through the 3D meat grinder on Apple Music, I do not want to comment on further.

For me, 3D audio is not just about putting audio objects into a (half-)sphere. In my 3D audio bubble, the technology is there to overcome the creative and technical limitations of stereo. For over half a decade I’ve been producing almost exclusively immersive audio in a variety of contexts from AR, VR, and XR with and without head tracking or image reference. I would like to show you what kind of applications these are with the help of my latest projects.

How’s it coming, where’s it going?

Artificial head stereophony has been a popular recording method for playback on headphones for decades. Currently, however, there is a lot of momentum coming back into the field as sound is now becoming more accessible as a three-dimensional event for a larger audience with speakers and headphones. Furthermore, additional technologies such as head trackers, data glasses and real-time renderings are popping up in a wide variety of areas, giving us an ever more realistic impression of hearing that we are used to from our natural environment.  

Put simply, this means that the more natural auditory impression can trigger emotions, feelings of presence and perceptions in us humans that are deeply linked to our own experiences. The whole thing happens more immediately than mono could, for example, because three-dimensionality eliminates a level of abstraction and is easier for our brain to process. Therefore, the keyword immersion, as the immersion in a virtual world environment, hits it quite well.     Of course, this changes everything – doesn’t it?! The euphoria among sound engineers is great. But where and when can 3D audio add any value at all? As I mentioned earlier, I’ve become quite radical in my choice of projects over the years and don’t do anything that isn’t immersive. So in that sense, I’ve moved quite far away from the traditional music and film sound industry to be able to implement my own ideas more freely.

But precisely because of my enthusiasm, I like to question whether 3D sound always has to be the first choice. Because I would like to warn against mentally storing “3D audio” in the drawer “better than stereo”. So it’s a matter of filtering out where the new technology works best and what might be promising areas of application. Since it’s hard enough to do justice to the topic in its entirety, we still want to scratch the surface as best we can here. 

Content is King 

Fortunately, there are now a number of manufacturers who have jumped on the immersive bandwagon: Microphones, plug-ins, and distribution platforms. Nevertheless, there is still a lot of research to be done in the area of next-generation audio, personalized listening, binauralization, etc. It feels like every discussion at relevant conferences ends with the keyword HRTF (Head-Related Transfer Function). But I would argue that what we currently lack is not so much the tools and theories, but the knowledge of what we can do with them in creative practice and what of it makes any sense at all.    I know at least a handful of colleagues who jumped at the chance to use the Ambeo VR mic in combination with a Zoom F8 as soon as the product was released, wisely anticipating that all the 360° requests would come soon. Years later, it’s now clear: that never happened. But how can it be that in other areas (like VR) there is still so much being produced in 3D audio than ever before? But before we dive into virtual reality, let’s take a listen to the world of moving images.

Video production with 3D audio 

The typical approach is to use 3D audio which has already worked well with surround. The fact that over a thousand movies have already been mixed in Dolby Atmos is no longer a secret, and this fact alone is remarkable. While you can currently enjoy the listening experience almost exclusively in cinemas or friendly studios, the three-dimensional mix will presumably soon find its way increasingly into the living room at home via soundbars.

Or even simpler: in virtually every household there are headphones that make three-dimensional audio reproduction in the form of binaural stereo accessible to consumers. As a listener, you increasingly get the feeling that you’re not just watching a movie, but that you’re part of the action. Mixing engineers can also be happy about the better transparency because with the spatial distribution of the different sound levels in the room fewer compromises have to be made than with stereo. For me as a “critic”, besides the immersion, the search for advantages of 3D in this context is getting pretty thin. Don’t worry, there’s more audio gold buried in other areas.

Rethinking sound and image is called for

Our ears, by the way, have unexpected allies: our eyes. As humans, we find it much easier to interpret what we hear when we also see it. In the 3D audio context, I advocate not subordinating sound to the image or, conversely, image to sound, but creating a symbiosis that must be approached differently for 3D audio than we are accustomed to with classic film sound. 

It’s nice to have an immersive mix for a motion picture. But if this were stereo only, the narrative would probably work just as well and only the physical immersion experience would suffer.   That’s not a bad thing, but it makes it difficult to get viewers excited about 3D audio. This is where the next hype word comes in: immersive storytelling. It’s a complex topic, but the principle is simple: find an application where spatial audio offers added value beyond just sound. 

So what would a video have to look like that benefits from 3D audio and vice versa? This is exactly the question I asked myself, and the result was the online marketing campaign “Sounds of Germany”. The client was the German National Tourist Board and I was able to win Sennheiser Ambeo as an industry partner.

Sounds of Germany

We all know those typical glossy tourism clips with happy people and beautiful landscapes. The task of this project was to produce four videos in four German cities for an online marketing campaign, which gives exactly that. But with 3D audio – there’s never been anything like it before, and in marketing, you always want to inspire people with something new. But the first thing I advised against right away was to simply go with the same style as always and just add “nicer” sound, because such images are rarely suitable for 3D audio effects, such as drone shots, panoramas, etc…

So I turned the tables and thought about which sounds would work well binaurally over headphones for all listeners* at home, arouse curiosity and provide the desired wow experience. Only in the next step did I talk to authors and cameramen about how such clips should be visually resolved. So, to bring out the full potential of 3D audio, I didn’t add sound to images, but visualized the 3D sound, if you will. So during the whole process, I had several hats on at the same time – sound engineer, director and editor.

This is how the concept of traveling through a city as an immersive tourist came about. In the process, we collect characteristic sounds, which are combined to form a beat. And in the end, one of Beethoven’s most famous symphonies is created for the 250th anniversary of his birth – I hadn’t yet gone far enough with a completely experimental concept. The four clips can be found on my homepage with further bizarre behind-the-scenes info: Sounds of Germany in 3D Audio.

Leipzig was so taken with it that they commissioned a second clip. To question whether the use of 3D audio is really well received, I submitted the video to some film festivals. Conclusion after one year: “Leipzig in 3D Audio” ran at 20 festivals in 10 countries and won 5 of them.

360° Videos and VR (Virtual Reality), Headtracking 

Let’s talk about VR: First of all a big “Attention”: Most sound engineers talk about VR, but actually mean 360° videos. However, such 360° productions are only a very small niche of the whole virtual reality and should rather be considered as a special case. There are, in fact, two major limitations: 360° videos are linear in time, just like conventional videos, and also only allow rotation of the viewing direction around three axes: X, Y, and Z. Also known as three degrees of freedom (3DoF). The hype around such productions has already died down and will probably only play a minor role in the future – even if the flood of tools and plug-ins suggests a need that hardly exists on the market. 

360 videos with head tracking

Nevertheless, I am a strong advocate of dynamic audio head tracking – a topic that is currently causing a lot of disagreement. Classically, surround formats with a center channel assume that the user is sitting in front of a screen and always facing forward. In my opinion, however, head tracking solves front-back confusion better than a personalized HRTF could. Apple supports exactly this technology on Apple Music, which they have already placed with foresight in the marketing of the AirPods. But all the Dolby Atmos Music tracks were mixed without head tracking – whoops! There are a lot of thoughts about this on my blog, so as a short conclusion: I don’t want to say now that all 3D music mixes are for the barrel with this … but head-tracking can be super if the application uses it correctly. In combination with movies, it’s also really fun to experience the cinema experience on headphones. The potential is even better exploited with 360° videos.

How could it be otherwise, it’s not only the picture that’s a round thing, but also the sound. The Ambisonics format has established itself as a quasi-standard for 360° videos and has been officially supported by YouTube and Facebook, among others, since 2016. In addition, there is a head-locked stereo track that, in contrast to the Ambisonics sound field, does not change when the head is moved. This additional, optional audio stream, is preferably used for voice-overs or music to narrate audio content that is not in virtual space. This blurs the boundaries between diegetic and non-diegetic content.

Ambisonics tends to get a bad rap in the audio scene because it has certain drawbacks, such as localization, that even higher orders can only partially solve. Nevertheless, due to its practicability and in combination with object-based audio in the form of mono sources, it absolutely has its raison d’être here. But since this is mainly about the content part and less about the constantly developing technology behind it, now on in the text: 

Of course, I would love to take an ORTF-3D microphone with me everywhere. But that was only possible on a handful of productions. These were shot with cinema cameras that were so large that the sound engineer and microphone array could simply hide behind the camera. One example is the post-production of the Anima Mentis project, which can be read in the VDT magazine 2019, issue 2.

Otherwise, my reality tends to be that I install as many small mics as possible throughout the scene and hope that they get as little noise as possible. Simply because the camera is looking in all directions and thus microphones have to be installed in the smallest possible space somewhere under the camera tripod or in the camera shadow. Otherwise, microphones will suddenly be visible later in VR, destroying the immersion that you’ve just painstakingly tried to build up.

In addition, the shooting location is usually anything but normal – after all, you want to show new angles. That’s why I quickly found myself in the desert in Bangladesh, in knee-high deep snow on skis in Ischgl or on a floating SUP in the middle of Lake Chiemsee. The often discussed “perfect” microphone arrays in all honor, but they often refer more to laboratory conditions or classic (audio-only) use cases, and less to the actual reality in a cross-media production.

VR

VR applications go one step further and should be considered an interactive experience. VR productions have more in common with computer games than with film productions. Here you can’t get around game engines like Unity or Unreal, and middlewares like Fmod or Wwise. This requires programming skills and a certain rethinking of audio production. However, the true potential of VR and 3D audio lies in this interactivity, which does not have to be linear in time.

Furthermore, 3D audio gets another component, since, in addition to the axis rotation, the transformation based on the three spatial axes is added. Thus, users can not only rotate from a fixed position, i.e. look around (3 DoF), but also move freely in space. This adds three more degrees of freedom and is thus also referred to as experiences with 6 DoF (six degrees of freedom). Thus there is no longer one optimal listening position, as it is taken as a basis for audio mixes. Rather, it must be taken into account that the location of the playback can be anywhere in virtual space. This opens up an incredible number of possible applications for rethinking sound.

In contrast to 360° video, where reality is usually filmed, I can add sound to precisely those experiences that would not be possible in reality. A flight simulator of the legendary flying boat DO-X is worth mentioning here. Here, for example, the task was to create engine sounds that were as authentic as possible or to record the actors in my studio using motion capture so that they could later breathe life into the scene as avatars. While we as VR users are part of the pilot crew and later have to take the wheel ourselves.

When the ears make eyes… 

Enough of the newfangled eyeglass technology. Let’s go in a completely different direction, in which I have been poking around in recent years. Namely, experiences that are entirely devoid of visual components. You could also simply call them binaural audio games. Here, more and more comparable formats like 3D audio podcasts are popping up, about which I already wrote a rather detailed article a year ago, see vrtonung.de/blog.

Often it seems as if in such productions with DearVR Pro, the speakers are merrily stretched through the area, whereby no real concept is recognizable. At best, binaural atmos were still integrated. A question that immediately arises for me as a listener: Couldn’t it have been done in stereo right away?!

A podcast or radio play benefits immensely from even the simplest sound design like jingles, separators, atmos, etc… Why they should work much better in 3D (which is still binaural audio in the end) is not really clear from the advertising texts of such formats. At the end of the day, the nice sound design in stereo is just a bit nicer when it is binaural. You might be able to better match the voices to the characters, but crazy thought: You could do that with stereo, too, and there’s no one panning the speech left and right. So why should binaural be so much different?

BKW Engineering Live Show

And now the enthusiast in me comes out again: How about finding an application where exactly the above panning makes perfect sense? That’s why I’m a fan of storytelling because it puts audio into a more comprehensible context. So if you can find a story that tells directly why the whole scene is using three-dimensional audio, you don’t think about the technology so much as a user. You accept it, and that’s exactly where we want to go: People should forget about the technology and be able to focus on the story. That’s probably the secret to the success of the ever-popular and rarely achieved “Virtual Barbershop.”

Driven by this thought, I produced several radios plays that took this idea to the extreme and got my artistic head out of the studio and into unknown territory. One was for the Swiss energy company BKW. The task was to produce an audio of about 7 minutes, which reflects the values of the company (sustainability, innovation, network, etc.) and is entertaining at the same time. This was because the whole thing was built into a live stream, which was broadcast corona-compliant instead of the planned large event with 3000 viewers*. So 3D audio can be a welcome change for the sound system at home, and everyone has headphones handy anyway.

While I was writing the story, I was asked if I could imagine an interview in the show, in addition to the radio play, in which I explain a bit about 3D audio technology. Sure – I’m always happy to do that. And so, from the author’s point of view, it suddenly made sense for me to become a protagonist in my own radio play. Because in the interview I’m already introduced as a character, and before I knew it Hazel Brugger was grilling me about 3D audio in the Livestream.

Immersive theater

Thanks to funding in the cultural sector, which was particularly hard hit by the Corona pandemic, there were more opportunities to experiment with immersive technologies in the theater. Worth mentioning are the theaters in Augsburg and Essen, which both use VR glasses to give their audiences new opportunities to immerse themselves in the theater. Here, too, I had my fingers in the pie, since the Chinese VR glasses from Pico do not support Ambisonics sound out of the box. So we developed our own app that solved the problem.

But VR here is just one of the myriad ways technology can be used to add value, not just bells and whistles. Here I was able to work with director Christiane Mudra, who has been adding video, light and sound elements to her plays for many years. This time, the focus was – how could it be otherwise – on 3D audio.

Why actually? Perhaps a few words about the content, because the play “Holy Bitch” is about sexualized violence against women. Mudra chooses an investigative approach, which questions, does not impose an opinion, but makes the listeners think and awakens empathy. So it’s not about the clumsy imitation of violence, but about making tangible how sexism sounds in everyday life – from real life to the Internet.

The live audience wore a wireless headphone system throughout the play. On the one hand, the live play of the actors could be heard through their headsets. But we also recorded several passages in advance, which were played live binaurally. This allowed us to blur the narrative levels, so that actors, for example, were in dialogue with their inner selves from the past. Thanks to the 3D audio technology, it was possible to create confusion on the one hand when things got wild – but on the other hand, very personal moments could be heard very presently, as if one were part of the victim’s privacy.

Outlook 

For immersive 3D audio to be a long-term success, it’s not enough to install a few plug-ins and speakers in your studio. In my opinion, the position of “sound man” has to be extended during the production; from mere recording and mixing in a certain way to the directing department and project management. Only in this way can immersive sound also be given the necessary appreciation. So it’s worth turning the tables and looking beyond film and music to the industries and applications that hold unimagined potential. True to the motto “Sound First.” 

That’s also why I like to call myself a “sound enthusiast”, and write about the small, big, old and new topics of the sound world on my blog. I want to try to do my part to help 3D audio get the future it deserves. I feel that there is a lot of interest among audio engineers to get a piece of the 3D pie. But I think the cake has to be baked first, and that will never happen if our only ingredient in the cake is, say, AC-4. There are a lot of myths and misinformation in the field of immersive audio – and I’m always eager to expose and question them, even if I sometimes make myself vulnerable. As audio professionals, we also have a certain responsibility to pave the way for the next generation. This is also one of the reasons why I am sitting as an expert in the reorganization of the professional training “immersive media designer”, as you can read more about this here: Professional education as immersive media designer.

In the midst of all the immersive hype, however, we must not lose sight of the whole big topic of sound. Already, the next hype topics are hitting: Artificial Intelligence, Blockchain, Voice Assistants, Smart speakers etc. What at first sounds like buzzwords that are far removed from our film sound world are actually very hot topics that sound engineers should take a close look at.   It sounds a bit abstract, but the wonderful thing about sound is that it’s a cross-cutting technology that can be used to dock onto virtually any other topic – and that goes far beyond immersive audio. So now it’s time to become a doer yourself, to leave your comfort zone and venture into unimagined audio territory. Because it’s up to us to shape the music of the future, and it has more than just three dimensions.

Interested in even more immersion? I offer to consult and coach on request, both for self-employment as a sound engineer (market positioning, away from price dumping) and for entering the immersive audio world.

And what could only be touched upon here goes into more depth on my blog: