How Medical Sound can accelerate the healing process with VR

Medical sound is not an established term, but it will be used in the following as a term for audio in the medical or health sector. Because yes, VR, spatial audio, and sound, in general, are not only entertainment media, as many may not know. This article will therefore give an impression of how (3D) audio is and can be used in the healthcare sector.

3D Audio is the new Stereo

Spatial audio offers a multitude of new possibilities from which the medical field can also benefit. But why is 3D audio better than stereo? I am one of the biggest enthusiasts but at the same time a critic. One should not immediately think 3D audio is always better than stereo. However, with surround sound, there are many advantages that will be beneficial especially in the medical field.

To summarize, 3D audio allows us to locate sound from any direction. Sound familiar? Hopefully, because that’s exactly how we hear in real life. So Spatial Audio allows us to have a natural or realistic virtual listening experience and therefore has a different effect than normal stereo. People often use the word “immersive” for this, meaning enveloping, you are “immersed” in a sound world. This can be used to create new types of medical sound use cases, as we will learn later.

It’s called Virtual Reality, not Visual Reality

Sure, when you hear the term virtual reality, you probably first think of VR goggles equipped with special displays to create a three-dimensional image. But many people forget that the visual is only one part of a convincing VR experience. Only with the right sound does such an experience become truly immersive.

How do we hear in (virtual) reality?

The following consideration should make us aware that sound is an essential part of an experience, whether in VR or real reality: Unlike our eyes, we cannot close our ears. This means that we are permanently exposed to auditory stimuli.

Furthermore, we can perceive sound from all directions, but our field of vision is limited to about 180°. There is also no absolute silence for us, if you pay attention you will quickly notice that there are always noises to be heard. And the fact that we quickly feel uncomfortable in anechoic rooms speaks for itself.

One could almost assume that many people don’t place such a high value on the sound of an experience because sound is always there anyway. But that also means, conversely, that sound is important – because, without, it sounds strange.

What does this mean for VR?

Audio complements the virtual environment, both as an enveloping atmosphere and as concrete movements, objects, actions, etc.. – In other words, everything that makes noise in reality. Of course, it also depends on the use of the VR application. It is not necessary to add sound to every detail for every VR use, and there is also artistic freedom.

Sound also acts as a guiding element according to the motto:

“You can overhear elements in VR, but you can’t miss them!”.

As already mentioned, we can perceive sound from all directions, i.e. the guiding function of audio can also or especially be useful where we see nothing, namely behind.

Gamers have known this for a long time. Especially in shooter games, enemies are often located by sound localization, even in non-VR games.

Or, for example, voiceovers of different protagonists can be better distinguished by spatialization. To this end, I did a Heatmap study to find out if 3D sound really makes a difference. Lo and behold – many people in the stereo group were irritated by the voice-over because the voice could not be assigned to a person in the room. So sound can be used as an intuitive storytelling element here.

Audio in the health and medical sector

So now let’s move on to medical sound. Audio in the health context can mean many things, from cognitive stimulation to concrete medical use cases. Let’s start with the basics of how our hearing works and what it means for health applications.

What is psychoacoustics?

Psychoacoustics deals with the correlation between our hearing perception and the physical properties of sound. In other words, research is conducted into what sound does in our brain. These are usually quite different things, what one would expect. Our brain sometimes hears things that are not there – or frequencies not, although they are there.

There are a few known effects that reveal that our brain does not always translate sound the way it hits our eardrum. And on the other hand, it is quite flexible in dealing with it.

One example would be the residual tone: We can hear the pitch of an instrument or voice correctly even if the fundamental is removed. This missing fundamental is called a residual tone. A concrete example of this would be a telephone conversation; the speakers built into it usually play no lower than 300Hz, but the fundamental of most male voices is well below that. Nevertheless, we can infer the correct vocal pitch of the person on the other end of the line. What is insanely difficult for artificial intelligence to determine via Deep Learning, our brain does casually.

Another psychoacoustic phenomenon is the cocktail party effect. This refers to our ability to selective hearing. In a comparatively noisy environment such as a party, we can focus our hearing so that we understand our conversation partner and block out the rest.

Also interesting are auditory illusions like the Shepard scale, also called Shepard tone, a scale that seems to go endlessly higher or lower. But as a matter of fact, these are temporally staggered sweeps that are tonally indistinguishable from each other.

Probably one of the most important properties of sound for us on a psychological level is that emotions can be triggered. First and foremost, music. This, of course, brings a lot of potential for medical sound, for example, in the treatment of trauma or even relaxation.

Where is audio/sound already being used in health today?

Medical sound is still in its infancy, but even today sound is already being used in a variety of ways. In the field of sonification, i.e. the acoustic representation of data, the beeping of an ECG machine is a well-known representative. The beeping reproduces the frequency of the heartbeat and in the event of cardiac arrest, a shrill continuous tone sounds – probably everyone has seen/heard this in a movie. Spoiler alert: there is still a lot of potential here, because the beeping serves its purpose, but is not exactly pleasant in the long run.

Also, the use of ultrasound has long been standard in hospitals. Granted, in that case, sound is used, but just not for auditory purposes. It is also called sonography and is used primarily for the examination of soft tissues. So it is less suitable for bones. But still crazy what sound is actually capable of if you hit the right frequency.

Music therapy is a completely different story. As mentioned earlier, music can evoke emotional responses. So music therapy is about harnessing these properties to restore or maintain patients’ mental but also physical health. Sound is a very personal medium here. Each person associates something with the sound of pattering rain other emotions. Also, our perception is culturally shaped and so for us, a gong may be more relaxing than in Asia.

Best Practice – Further Thought of Possible Applications of Medical Sound

What might the future of medical sound look like?

It is very likely that VR will play an increasingly important role in the medical field in the future. More and more research is currently being done, which is not easy in this area, as special hygiene must be maintained. In addition, we must not forget that we are working with sick people. With VR, there are not only new possibilities for better research into diseases but also new ways of conveying knowledge.

A prime example is the Stanford Virtual Heart, a medical VR “game” to illustrate heart defects. This VR application is already being used as a pilot project in 20 cardiology-related institutions to better explain heart defects. Heart defects are considered very complex, which is why knowledge transfer is not easy. The VR application, which is logically designed to be anatomically correct, allows some heart defects to be observed at close range. This is a great aid to understanding, especially for relatives.

As a user, you can even go inside the heart. But let’s get to the actual medical sound: In the case of heart defects, sound plays a decisive role; depending on the defect, the rhythm or timbre of the heartbeat changes. That’s why doctors need a trained ear. The Stanford Virtual Heart therefore also functions as a kind of virtual stethoscope, because the sound design was also created anatomically accurate. You can find an interesting interview with audio examples of the sound design here. Shoutout to my friends at Delta Soundworks!

This pilot project already hints at the potential of VR and medical sound. So sound designers will be faced with a whole new set of challenges in the future. As you can see from the Stanford Virtual Heart example, an artistic approach to sound is not enough in that case. For example, the question: “what does it actually sound like inside the heart?” You also have to deal with the matter from a medical point of view in order to create accurate sounds that can be used for training applications in VR.

Immersive EEG – How to turn brain waves into experiences?

Together with the company Brainboost, I was allowed to implement a special project with EEG and the brain activities measured with it. It is about audiovisual VR experiences generated from these activities. These primarily serve to combat stress. The user should develop an understanding of how his or her own brain works. What thought patterns trigger stress and how can I guide my brain into relaxation?

VR is very well suited for stress-reducing applications because the environment and thus potential distractions are blanked out. So you can focus entirely on the application. Plus, the audio gives immediate feedback on the data. A bit like the ECG beeper – only much more pleasant. The goal was to discharge the patients with a feel-good factor. So that on the one hand they had a learning experience, but also a pleasant session.

Of course, sound also plays a major role. With appropriate sounds and music, the calming effect is strengthened and immersive atmospheres do the rest. The emotional effect should not be ignored either. Ultimately, the project was about learning to control one’s own brain waves and transforming them into resulting VR experiences. If you want to learn more about it, you should check out my portfolio entry.

Medicine sound thought further – Where is sound still relevant?

The noise level in hospitals is increasingly becoming a problem. Not only is a loud environment disturbing, but according to some studies, it even has a negative impact on the recovery period or the well-being of patients. And it’s quite understandable that with so many different sources of noise, it’s impossible for patients to get any rest.

What would be a solution? Certain devices, such as the ECG, have to make noise, so one approach would not necessarily be noise reduction, but an ambient sound design. The devices could be tuned to each other so that harmonious sounds are created, for example. Sound design of beepers, etc. could also be improved. And yes, where it is possible, noise reduction is probably the best option.

You can see here that medical sound is also relevant where you probably wouldn’t have suspected it right away.

Conclusion

There is a famous quote by George Lucas that says that sound is 50% of the movie experience. In the case of VR, this value should probably be increased a bit more! You’ve seen how differently VR and medical sound can be used now, so you can only guess what’s going to happen in the future.

For me, it’s insanely fascinating that sound can not only be used for entertainment purposes but can literally help people feel better.

Spatial audio and VR are such flexible mediums, so it’s obvious that the possibilities are far from exhausted. Exploring these is one of my main tasks. So when you have ideas or are looking for someone to create the right sound, you know who to contact! 😉