Inside an orchestra VR experience with spatial audio

Sitting right in the middle of the orchestra, between strings, brass and timpani: visitors to the “Musikfest für alle” at the Gewandhaus Leipzig on 13 April 2025 could experience that with a VR headset. It was based on a 360° video of the youth symphony orchestra of the Musikschule Leipzig “Johann Sebastian Bach”, produced with diginetmedia. I mixed the sound for it in 360°.

Why a 360° concert needs more than the stereo recording

With a regular stereo recording something illogical happens in the headset: you turn your head, the picture turns with you, but the sound always stays the same. People are used to it, so it hardly stands out. But anyone sitting in the middle of the orchestra expects the brass to come from behind when it is sitting behind them.

The starting point wasn’t ideal. The recording was made exclusively through the hanging microphones of the broadcaster MDR: an AB main system and a Decca tree with outriggers, all suspended above the orchestra, nothing on stands. There was no dedicated room microphone for 360°.

How the microphones became a sound space

Using the music school’s microphone plan, I placed each individual track at the spot in the 360° picture where the microphone actually hung, and mixed the whole thing as ambisonics. That way the sound turns with you when you move your head in the headset.

The problem was crosstalk. On every microphone you also hear all the other instruments in the background, and that made the spatial effect more subtle than I would have liked. At the start the brass already came clearly from behind, and the percussion was easy to place. For a lay audience that would have been enough, for musicians not yet. The music school’s feedback said exactly that: still rather weak, but “impressive to stand virtually in the middle of the stage, people will love it!”

So I went further. AI-assisted channel separation made it possible to pull the instruments apart better, and localisation became clearer. There was also an offset of a few milliseconds between picture and sound. Syncing to the conductor was difficult, because he doesn’t produce an audible signal. It worked with the first timpani stroke.

Finally I added applause from the audience at the very beginning. The silence before the concert otherwise sounded more like noise, and the applause has a nice side effect: anyone who turns their head right at the start immediately hears that it comes from one direction and that the sound really is 3D.

What came of it

At the Musikfest, children and guests could experience the orchestra in VR from the musicians’ perspective. The video exists as a 360° version with ambisonics and additionally as a 7.1.4 mix in the Eclipsa Audio format for home cinema and headphones.

For me the project showed that you can build believable 360° sound even from a classic concert recording, as long as you know the microphone positions. A dedicated ambisonics microphone in the orchestra would have made it easier.

Frequently asked questions about an orchestra in VR

Why does a stereo recording sound wrong in a VR headset? Because it doesn’t turn with you. You look behind you but hear the same as when looking ahead. Ambisonics adapts the sound to the viewing direction.

Can you mix a 360° concert without a dedicated 360° microphone? Yes, if you know where the microphones hung. Each track is placed at its position in the picture. Crosstalk between the microphones does weaken the effect, though.

How do you sync picture and sound for an orchestra? Ideally on a clearly audible and visible event, here the first timpani stroke. The conductor doesn’t produce a signal you can lock onto.

Planning a concert in VR or 360°? Let’s talk.

Sound that turns with your gaze, from the microphone plan to the headset.