Humboldt 360° – Bilingual Spatial Sound for the Goethe-Institut

Alexander von Humboldt travelled with instruments. Barometer, sextant, thermometer, everything that could pin a foreign world down in numbers. Two hundred years later a film crew travelled the same route with a 360° camera, which at heart is the same idea: record a place so that other people can follow it without having been there.

For the Goethe-Institut this became “Humboldt 360°”, four episodes along Humboldt’s way through Colombia and Ecuador. I built the spatial sound version for it, in German and in Spanish.

Why a dubbed voice in 360° must not sit in the middle

The films were already cut when I came in. There was a Spanish original version and a German dubbed version, both for the institute’s YouTube channel. And that is exactly where the problem starts that makes this project technically interesting.

For 360° video on YouTube there is first order ambisonics, four channels describing the sound field around the head. What there is not is a second, head-locked stereo track. That was exactly the problem the client was stuck on: how do you get spatial sound and a head-locked track into one single video at the same time, when YouTube does not provide two separate slots for that? Facebook could do that at the time, YouTube could not, and in practice that means everything you want to be heard has to go into the sound field. There is no shelf for audio that is simply meant to hang in front of you.

For a dubbed version that is awkward. In an ordinary film you put the German voice where the original sat, and that is usually the centre. In 360° there is no centre, because the viewer turns their head. Fix the dubbed voice somewhere anyway and something unpleasant happens: the viewer hears someone speak, turns towards the place the voice comes from, and finds nobody there.

On top of that comes a second layer you have to separate cleanly. Some of the voices belong to the scene, because the person sits in the picture and speaks. Others are narration, a comment from outside that has nothing to do with the place. Film rarely makes this distinction consciously, but in immersive media it decides whether a scene holds together.

How four finished films were given a room

The work consisted of three layers, all of them aiming at the same thing.

The voices that belong to the scene I placed in the sound field where the speaking person actually stands, and in both language versions in the same spot. When Ernesto Mamanche, leader of the Muisca community in Sesquilé, sits to the right in front of his hut, his voice comes from the front right, no matter whether you are listening to the Spanish or the German version. That is not cosmetics. It is the difference between an interview and a conversation you turn towards.

Because the camera moves in several scenes, in the boat off the coast, on the way up the mountain, one fixed position was not enough. The voices had to travel with it, moving through the room via automation so that they match the person in the picture. That is the part of the work nobody notices when it is right.

The music came in as a stereo file and had to go into the sound field as well. Simply placing a stereo track in front would have been the comfortable route, but it would have set the same trap as a badly placed dubbed voice. So I carried it into the ambisonic field and spread it so that it holds the room rather than sticking to a wall.

The narrating voice I deliberately treated differently from the scene voices. It is not part of the place and should not sound like it, otherwise the audience goes looking for somebody who is not there. It stays wide and unlocated, and because everything else is locatable, the difference reads by itself.

With all of that you have to know what first order can do and what it cannot. Four channels describe the sound field roughly, not to the point, and higher orders soften that rather than solving it. A voice cannot be set to the centimetre this way, it gets a direction and a width. For a documentary that is plenty, because the aim is not to fool an ear but to give the audience a direction to turn in. Anyone expecting a voice in 360° to stand as sharply as in a cinema mix will be disappointed by the format, regardless of how well it was mixed.

Then there was the levelling. Across four episodes, two language versions and several deliveries the level was not consistent, and individual voices sat too hot. That sounds like a side issue, but with ambisonics it is not: whatever distorts in the source material distorts after decoding at every point in the room you put it.

What was delivered

In the end every episode went out in two languages and two forms. A first order ambisonic master for the 360° version on YouTube and a stereo version for everything else, so eight masters and eight stereo files in total, spread over three delivery rounds in June 2019.

The four episodes lead from the arrival on the Colombian coast via Bogotá and the Muisca community in Sesquilé to Quito and finally onto Chimborazo, the mountain where Humboldt had to turn back at over five thousand metres. The films appeared in 2019, the year of his two hundred and fiftieth birthday, in the Goethe-Institut’s digital culture magazine Humboldt for South America, funded by the German Federal Foreign Office. What a 360° documentary sounds like when the sound is planned in from the start rather than added afterwards is shown by Crossing Borders.

Direction, camera and editing were by Tim Hamelberg, production management by Christian Stein, the concept by Thomas Lilge, produced by Playersjourney UG. The speakers were Bettina Conradi and Christian Conradi, the translation came from Lina-Marie Büttner, the stitching from Matias Bertoia. I was responsible for the spatial sound version in both languages.

What I took away from this project holds for any bilingual 360° production. The dubbed version is not an afterthought you lay over the top at the end. It has to serve the same positions in the room as the original, otherwise the two versions fall apart into two different films, only one of which works. It was much the same with the German version of Emperor, where every voice file had to fit an existing timeline exactly.

Planning a multilingual 360° production?

If your film is going to run in several languages, how much extra work the second version costs is decided early. Let’s talk about it before the first mix is locked.