VR Sound Design with Spatial Audio – Gaming, Metaverse and more!

This blog post is based on the interview I did with Patrick Schappert of GrobiTv on the topic of 3D audio applications. Here is the video about it on YouTube, which was created from the livestream:

But I wanted to bring the most important points again … to the point:

What is 3D audio in the classical sense?

Today we want to report on an interesting aspect of 3D sound, which is not film. Or is also not classical music over at least nine speakers as with Auro 3D and Dolby Atmos.
Instead, we’re talking to someone who is essentially concerned with getting the 3D sound experience over headphones.

Exactly, I’m totally committed to making sure that 3D audio really comes into its own. Unfortunately, it’s almost always forgotten that you can do music and movies, but you can also do a lot of other great things. We’ll find out today what these are and how 3D audio can work for all headphones.

My goal is to “enlighten” the people out there a bit. There are a lot of different terms like spatial sound, 3D sound, binaural, spatial audio. There is a lot of talk about it, but most of the time it is only thought of for the mainstream. Of course, movies and music currently reach billions of people. But the added value and the potential of this immersive sound lies elsewhere. So stay tuned.

How can 3D audio be used in VR?

Virtual reality had a pretty bad image with me in the beginning. At that time I only ran away from zombies which is not the only thing VR can do.
Now I know the guys from Magic Horizons, for example, they have developed the whole thing on the topic of relaxation. They also put a lot of emphasis on the acoustics, after all, the whole thing has to form a unit, right?

That’s right, everyone is always talking about immersion, i.e. immersion in virtual worlds. All of that with just a pair of data glasses that you have to put on. But if your ears perceive something different than your eyes would expect, then the brain says, “Wait, something’s wrong here.” That’s why it’s important that the sound supports the image. Only then you can really talk about immersion.

I’m also totally with you that VR had problems in the beginning and also the last few years.

• The HMD devices are clunky
• the image resolution is pixelated
• one gets sick somehow relatively fast
• the costs are not exactly favorable

Oculus aka Meta and the Metaverse

However, an insane amount has happened in the meantime. The Oculus Quest II (now: Meta Quest) is now on the market. Unfortunately, it’s not sold in Germany for privacy reasons. It’s really the first device that I would really recommend. Because as soon as you put it on, you’re immediately inside VR.

The setup is super simple. It has cameras that always track your room. You just have to draw markers on the floor so you don’t run into the wall and you’re good to go.

VR content is just gaming, isn’t it?

I remember five years ago when I first put on a pair of glasses and thought: “Great, I can now experience VR and look around… Only to ask myself where everything is? There was nothing! No content!” You had to produce content yourself, or what was available was wasn’t that well produced, because it was just insanely costly.

Even simple games like on the PS5 today didn’t exist back then. Because for A-level titles, you first have to spend an insane amount of money. Currently, you don’t have a large target group. Only for a few supposed nerds who have VR glasses. There are now a few million nerds. But that’s still not a huge market, one would argue.

What are the VR applications in the industry?

That’s why VR has developed more as an industry medium in recent years. I know many who use the Metaverse medium for onboarding new employees. Instead of boring training, there is an immersive experience. Take, for example, a trainee who needs to learn how to operate a crane. You no longer have to be placed in a real crane. Instead, the entire simulation runs digitally. This means that serious mistakes can no longer be made if the wrong lever is pressed. In reality, one wrong push of a button can mean a loss of millions.

All of a sudden, the costs to create such a VR project are not that high in relation to the potential damage. Be it not only for inanimate buildings, but also in the medical field like surgeries. If you can practice surgeries in VR as often and as long as you want. That is much better than on a living human being.

You’ve already added value with a medium that most people only know from entertainment or gaming.That’s what it’s all about: finding applications where virtual reality adds value. I have written an article on such VR training. So more info on how 3D audio can be used you can find here.

What are the VR hardware requirements?

Okay, let’s get to the hardware. As I said, VR is a pair of glasses that I put on. What can Oculus Quest II do more than the previous models? Why Oculus aka Meta and not HTC Vive, Pico and the like?

Which VR glasses are suitable for beginners?

Let’s take a quick look at the devices. You can buy the Oculus Quest 2 via Amazon France and use it legally in Germany. It has four cameras built into the front. That means you’re in VR, but you can also switch to the cameras at any time to see reality again. So it warns you how far away the wall is so you don’t run into it. Then you draw a virtual marker. You set it up within 10 seconds, and then the glasses jump from mixed reality mode back to VR. Then you can start right away.

Meanwhile, they make an update every month. Now, for example, hands tracking has been enabled. That means you see your hands in VR. That means in the long run you don’t even need controllers in each hand to navigate. Has a bit of magic, if overnight the VR goggles have suddenly learned. It’s also crazy to see your hands virtually all at once.

But I don’t want to talk so much about pixels, cables and computers. The content and applications are still crucial. In the future, however, we can look forward to lighter, better and cheaper devices. While in the past you had to connect glasses for about 700€ with several cables to a powerful PC starting at 2000€. Now you can already buy said all-in-one solution for just over 450€.

Metawhat? Metaverse!

Meta (formally Facebook and Oculus) are developing a lot. Of course, if you’re not in this world, you don’t even notice it. But in the meantime, they’re going full throttle. They also want to create 10,000 jobs in Germany alone for this “metaverse”. The question is: How can VR still be used in the workplace? Maybe with a lot of virtual screens that you bring to your desk. A bit like Minority Report.

How can VR be used for better communication? It’s less tiring than a Zoom Call. On the one hand, for brainstorming, that you’re meeting with people in VR and you can place 3D objects as you never could in real life. The whiteboard has had its day. So you have a lot of possibilities, but to be honest, only a few of them really make sense. We are still a bit in this experimental phase. Okay, now we have these glasses, but how can we really use them in a creative and meaningful way?

Rumor has it that Apple will soon jump on the bandwagon. That’s why Meta is stepping on the gas so much right now. To get a head start. In the background, many managers and employees are being bought up, startups are being purchased, and so on. Everyone is looking for exciting demos for the technology. That’s exactly the approach I’m using with 3D Audio. The technology is great. But what can you actually do with it in a value-added way?

Apple is already doing a lot in the area of “Spatial Audio”. In addition to the Airpods, almost all devices such as the iPhone, AppleTV, etc. now support spatial sound. This technology can also be adapted for VR. An overview of the supported hardware and software is available here.

Furthermore you can read here about the benefits of spatial audio for video calls.

How can 3D sound be heard in VR gaming?

So we are looking for content. At the end of the day, we have the glasses and the technology is quite far along. But the private use is still very much oriented towards entertainment media, isn’t it?
Although I’m sure most gamers aren’t even aware that we’re now using three-dimensional sound here, too.
For example Dolby Atmos, which is available in many games. Or the Tempest 3D audio engine of the Playstation 5. Is 3D sound also available for VR glasses?

You’re right. If you want to reach consumers, you do it mostly through games. The advantage for listening in VR is that you don’t have a fixed viewing position. I mean, when you’re gaming on the computer, you’re always looking forward. But if you can hear if your opponent is coming from behind, that would be cool.

Gamers are one step ahead once again

Especially in first-person shooters, this is also really an asset. If you know that the opponent is coming from behind, you can react better. You intuitively have a better chance of survival and ideally get a better score. That alone is a nice application for 3D audio – if it weren’t for killer games.

But with VR, you still have that plus point of having that physical head turn. I’m a huge fan of dynamic head tracking. I went from VR to the whole 3D audio theme via head-tracked audio. All of a sudden, the audio dynamically rotates with it. When that’s mixed well, it’s also insanely fun.

What is head-tracking audio and how does it work in VR?

Then you no longer have these problems that you have with classic 3D headphones. That you can never actually tell the difference between front and back – or top and bottom. This is also called front-back confusion. As soon as you can turn your head a little bit, you immediately hear where all the objects are in the room. In combination with the visual, it has a blatant added value for games. The advantage with the head rotation applies to all interactive applications in the XR area. So beyond games also for the trainings I mentioned.

If you want to know more about it, head tracking is already available in several headphones.

Here is an example based on a 360° video. The female voice at the beginning is head-locked. This means that it is perceived as an in-the-head localization. It does not change with the video rotation. From 0:25 on, however, you can clearly hear the boy coming from one direction. If you change the image section with the mouse pointer on YouTube, you can hear how the spatial sound adapts to the voice:

Prime example Beat Saber: A rhythm game

It’s Okay, that’s very close to reality. But you didn’t deal with games either. You found your own subject area to deal with?

That’s exactly how I would put it. I think games are great, but I’m not a gamer myself. In my youth I have gambled enough. Today I don’t have the time. Except for one thing, which I would like to mention briefly is “Beat Saber”.

Maybe you’ve heard of it? There you hold a lightsaber in each of your two hands. There are virtual blocks flying towards you and you have to cut them with your sabers to the beat of the music. It’s so much fun! People put on the glasses and immediately understand how it works. VR can also be very intuitive. Here, you can just forget about the technology since nothing else is needed (like controllers). This game recently cracked over the 100 million mark in app purchases. As you can see, there’s serious money to be made here now too. I like to say:

“here someone paired the game “Guitar Hero” with “Fruit Ninja” and put it into VR – Beat Saber came out and it’s awesome.”

So, you can get into gaming. But that’s not my personal motivation. Because I’ve found that audio can be so much more than entertainment. I’m personally so hooked on experiences, like in the medical field. Where I can help people with sound. Whether it’s the patient you can guide through relaxation experiences. Or the doctor in a VR simulation can hear when he pierced the bone during surgery, for example. That’s where I can personally add value with sound.

VR Sound Design and Storytelling

The question that comes to my mind is what do you have to do to get the sound? Do you have to drill three times and record it with a microphone? Or how does the VR sound work?

It always totally depends on the application. I try to record as much as possible on the spot. Then everything sounds coherent, especially if I record everything with special microphones. For example, in the operating room, the sounds simply sound like the operating room. That’s super believable and better than trying to get some drill sounds out of a sound library.

A good sound designer can do it, but it usually takes longer. The trick is to design a good strategy in advance and save yourself work. I also have a lot of experience and I advise on such projects in advance.

VR Sound Effect Library

What exactly does library mean? – Do you have a database where you have a lot of sounds that you can pick out or that you can even buy? But it’s not optimal because the space doesn’t fit?

Exactly. Every sound person has his audio archive. A hodgepodge of random sounds, but they don’t necessarily always fit together the same way. That’s why I do as much as possible myself.

VR Storytelling and Emotions with Sound

But then storytelling finally comes into play. Or even emotions and story living, as it’s called. Is it just about the drill or is it also necessary to virtually simulate how stressful an operation can be? There are a few factors:

• the beeping of the ECG to show what the heartbeat is like
• the ever-changing beeping of the heartbeat responding to user input from the UX and UI
• Devices that may fail and also start beeping
• But more than beeping: assistants in the OR who are also communicating.

Such sounds can be simulated very realistically in VR. Clever use of music and sound effects can add to the emotional impact of the experience. This allows testing and practicing so that a doctor is not upset in stressful situations. Practicing worst-case scenarios in VR is a lot better than in real life.

All of this is just scratching the surface, so if you have any questions, feel free to email me.