Why (3D) sound in VR trainings so crucial for its success

Simulation applications in VR are becoming increasingly popular: from virtual earthquake simulations for school classes to medical training. The use of virtual training methods offers a wide range of possibilities to make training scenarios look authentic.

More and more use cases are emerging that actually save time and money. For example, a VR crane simulator may seem quite cost-intensive at first. But when you consider that in reality much more damage can be caused if the trainee makes a mistake, this is quickly put into perspective.

But the focus is usually only on usability. In addition to the visual presentation possibilities, the sound of course plays an important role again! We learn best for the real thing when the VR training is as realistic as possible. And that is not the case without three-dimensional sound.

Acoustic announcement of dangerous situations

Sound can be used presently to indicate dangerous situations more actively.

What is active sound design in VR?

In VR training for firefighters, the distant sound of hissing gas can indicate an impending gas explosion. In rescue operations on frozen lakes, the cracking ice suggests breaking. In trainings for police officers, which are simulations for tailing or rampages, the footsteps from behind can indicate imminent danger.

An experience that every driver has probably had is when oncoming traffic approaches from behind and – invisible to the eyes – closely overtakes our car. This kind of unexpected event triggers stress, which in turn causes emotional reactions. You can look around as much as you want in VR training and use the mirrors, but sound cannot believably depict the danger situation.

How can the right mood be created in VR training?

In many VR training applications (e.g. training firefighters or surgeons), it is extremely important for the effectiveness of the training that the people behave as they would in “real” training situations. A sterile 3D environment rarely does this justice.

A more sporting example is the mental training of professionals before a competition. Coaches always ask themselves why moves don’t work as well in real life as they did in training.

But you have to put yourself in the shoes of the athletes. When you go from a football training ground to a stadium, the pitch is the same, but the atmosphere is fundamentally different. It’s not the tens of thousands of spectators who might wave unfriendly gestures at you.

Much more frightening is simply the thunderous shouting that you don’t hear in training. What gives the fan goosebumps does not leave the professional footballer cold either. Immersion in the virtual world is therefore essential, with sound – you guessed it – playing a major role. So the right atmosphere is needed. And unfortunately, it is not enough to download a stereo atmosphere from the internet and import it into the VR application.

For VR training, it is definitely worth recording the sounds in their original form. Preferably directly in 3D audio. The users immediately hear the difference to sounds they know from their everyday life. Therefore, the virtual reality application must also sound credible and this is only possible through professional sound design.

Music and Voice-Over

In most VR trainings, original sound is almost dispensed with. The only audio elements that can be heard are usually music and perhaps a voice-over. Just because this is unfortunately common (better than no sound at all) doesn’t mean it’s the best approach.

How can music be used in VR?

I have already discussed in detail how music can be used in comparable scenarios. Therefore, I would only like to briefly break down the added value or danger of VR trainings.

In the vast majority of cases, music is used simply so that some sound can be heard. This does help to convey to people “there is no silence here”. So it doesn’t sound “broken”, if you like and as explained in this article.

Still, one has to keep in mind that especially with VR trainings, reality is depicted. But in reality, there is rarely music, as the VR application here falsely suggests.

So I am quite critical because the use of music in virtual realities needs to be fundamentally rethought. But there are certainly parallels to its use in films.

A few action drums can already help to trigger more stress in the user if it is desired for the situation. Even if these epic soundtracks are actually more familiar from blockbusters. As always, musical sensitivity is required to ensure that music supports VR trainings and does not hinder it.

How can a speaker be used?

More common is the use of speech to guide the user through VR trainings. In the production process, I am not at all angry if it is only text-to-speech at first. But this should not be found in the final applications.

Unless such speech synthesis is actually used in the real application. Otherwise, a professional speaker makes a much better impression. Here, too, authenticity is important in voice casting, so that users feel they are being met and the avatar can relate better to them.

But not only communication can be solved via speech. In most cases, the voice-over takes on the role of an instructor, i.e. a virtual trainer. This can be used intuitively to guide the user’s gaze in the desired direction.

Usually, only visual arrows are used here. These, too, are less likely to be found in reality.

However, it is rarely enough to simply record a speaker and integrate him or her into the VR application. It is important to decide whether the audio source is also spatialized, i.e. whether it becomes a 3D audio object.

Otherwise, it is called in-camera localization to convey to the user that the person is not part of the scene. Both have their advantages and disadvantages. Of course, I will be happy to help you find the optimal solution.

Sound can convey information

It should now be clear that the right atmosphere is essential for VR training. But not everyone knows that sound can also be used to convey information that goes beyond human communication. Sonification is the magic word.

This can not only be used as an indication of potential dangers but can also be used in a supportive way to convey information.

In the medical field, the continuous beeping of the ECG tells the doctor what the patient’s pulse is and how the situation may be changing. If the beeping becomes rapid: something is wrong. If it remains constant, there is no need to act, but we keep track of the situation thanks to sonification.

If you are interested in sound design in the medical field, you can read more here. After all, a mere “beep sound” does not exploit the full potential, as you can guess.

In the craft sector, VR trainings can be used to specifically convey what it sounds like when a drill runs at full power or encounters resistance and thus audibly loses revolutions. To see this in a VR application is almost impossible. To hear it – no problem.

This calls for quite a sophisticated game audio design that seamlessly strings together interactive sound assets.

Sound can tell without speaking

Sound is important for our perception of current situations. Firstly, of course, in an obvious way: a baby that starts to cry shows the parents that it needs attention. A gunshot or an explosion that can be heard at a distance simulates “danger”. Just like the siren, we know from our everyday lives, which instinctively makes drivers take caution.

Why is silence an error in the Matrix?

But sound can play a completely different role in VR. Sound can draw attention to things that are not visually apparent. This can also be much more subtle than the aforementioned dangerous situations. If sound is missing, we quickly notice: something is wrong.

Anyone who has children knows this: When the little rascals are playing in the next room, you have a constant noise. This gives us the feeling that we can leave our offspring unsupervised for a while. But sometimes it gets quiet … too quiet. Parents know immediately: something is wrong here!

Sound is not just a sense of hearing

So it’s not just about actively perceiving hearing. Rather, the ears help us to decipher what the mood is in a situation. You may know it when you enter a meeting room and the participants are noticeably restless. Either something is going on or a colleague has just told a joke. But we notice immediately when we enter the room that something is different – without our eyes being able to tell the difference so quickly.

This impression of an omnipresent sense that recognizes the situation correctly cannot be transmitted visually in VR applications. Professionally designed sound, however, can help with exactly that: pointing out a danger when it is subtly or obviously out of sight (as explained above).

Vibrations are closely related to sound

Closely related to sound are vibrations. These can also be implemented in trainings applications to better reflect realism in the field.

For example, there are devices such as SUBPAC that convert the sub-bass component into haptics in real-time. But there are also devices that are less plug-n-play, but give even more possibilities to work with audio triggers.

One application for this would be to transmit the vibrations of ship engines during VR trainings to show how it feels when ship engines are not running properly. This is often difficult to identify due to the droning of hearing, but with the support of vibrations, there is a way to train this detection.

Researchers have also developed footrests with vibrating platforms to create the sensation of walking while sitting by vibrations on the feet. This can also help make VR workouts in confined spaces more realistic and, accordingly, more effective.

Summary

VR trainings now offers wide-ranging opportunities to train people in a variety of areas and applications.

On the one hand, elaborate trainings can be made accessible to more people, dangerous situations such as earthquakes can be effectively shown and behavioral patterns can be practiced. It can also enable situations to be acted out several times, allowing mistakes to be made without the need for costly reconstruction of a set.

Sound plays a major role in all these applications: dangers can be announced before they are seen.

Sound can convey important information and make users feel as real as possible in the virtual world. This is particularly important to guarantee the effectiveness of training.

It is the only way to ensure that in an emergency the user is prepared for the situation. In this way, the user is not distracted by sounds that he or she is not familiar with from the VR trainings. VR worlds should come as close as possible to the real world in training scenarios, which means that all senses have high importance for VR experiences that are as realistic as possible. In real life, the auditory sense is often perceived rather subconsciously by many people.

However, the insights gained from this subconscious effect can be used to incorporate very deliberate aural effects in the virtual world. So how can your VR trainings become a complete success? I will help you with that, feel free to contact me!