Inside Tumucumaque – Rainforest in 360° with Spatial Sound

A rainforest is not a picture with some sounds added to it. It is an acoustic space where nearly everything alive can be heard and almost none of it can be seen. Show the Amazon in 360° and then put a flat soundtrack underneath, and you strip the place of the one thing that defines it.

For the 360° film “Inside Tumucumaque” I did the spatial mix and the sound design. The film is part of DELTAS, a crossmedia project by Filmtank and the Interactive Media Foundation, and it takes you into the Tumucumaque National Park in northern Brazil, the largest tropical rainforest reserve in the world.

Why the jungle is harder on sound than on picture

The film has no plot in the usual sense. It has places: sunrise over the canopy, a boat ride through morning mist, the forest floor between roots, dusk on the river. In a linear film you would connect scenes like these with a camera move and some music. In 360° that does not work, because the viewer decides where to look.

That turns sound into the navigation system. It tells you what is happening behind you, how far away the bank is, whether that bird sits in the tree above you or across the river. The picture cannot do that job, because it only ever shows the section you happen to be facing.

There is also a practical problem: production audio from a 360° camera in the jungle is unusable. It is a carpet of noise with no direction, where everything sounds equally far away. What I received were individual field recordings plus narration in German and English. Out of those pieces, a space had to emerge that feels like standing in the middle of it.

The film was shot stereoscopically, so each eye gets its own image. Technically that changes nothing for the mix. It does say something about the ambition: if you take on twice the data and twice the effort, you want depth. Flat sound would have undone that in the first second.

How individual recordings became a forest you can walk into

I built the mix in ambisonics, using the 360pan suite up to third order. Every element sits at its own position on the sphere and stays there when the viewer turns their head. Water at the hull comes from below, the bird from above and behind, the rustle in the undergrowth from the direction you are not looking.

One decision held the film together for me: each scene got its own acoustic lead element instead of running the same jungle atmosphere everywhere. In the timelapse sequence it is movement sounds that push the clouds across the sky. In the boat scene it is water, creaking branches, and leaves passing close to the ear. On the forest floor it is the small things you only hear when nothing else is running.

Run everything at once and you get a wall of sound that reads as “nature documentary” and as nothing in particular. Spatial audio only works when something stands out from the rest. Christian Barth contributed the music and the transition sounds, and we kept both deliberately sparse so they would not take frequencies away from the animals.

The water recordings confirmed something I have seen in every nature project since: you cannot rebuild a river on a computer. You can approximate it with close mics and a spatializer, but it always sounds constructed, because the interplay of current, bank, and reflections is too complex to invent in a studio. The only approach that really works is going out and recording it. That is why the material from Brazil was so valuable, even though it arrived in separate pieces.

The spatial mix then had to travel three very different roads: a head-tracked version for the VR headset, a binaural stereo mix for screenings without one, and a version in TBE, the Facebook format of the day, for the VR cinema. Behind that sat a zoo of platforms that is hard to picture now: the Gear VR played through the Oculus Video Player, the Vive ran Vive Cinema, the cinema needed something else again, and every one of those chains had its own idea of how multichannel audio should sit inside a video file. That a format could simply disappear a few years later was a suspicion in 2018, not yet experience. Today I know it is exactly why you mix scene-based and not into a finished render.

What came out of it

The 360° film is one of three parts of DELTAS. The others are the five-part television series “Deltas of the World” and a walk-in VR installation that lets you slip into the role of five animals: vampire bat, black caiman, harpy eagle, Goliath birdeater tarantula, and poison dart frog. The installation ran at ZKM Karlsruhe from 6 April to 27 May 2018 as part of the exhibition series “The Art of Immersion” and then went on tour, including the Hamburger Bahnhof in Berlin, the Finnish nature centre Haltia, and festivals such as SXSW in Austin, FILE in São Paulo, and Raindance in London. It won a string of awards, among them Gold at the Art Directors Club and at the European Design Award 2019. It was created with Artificial Rome, guided scientifically by the Museum für Naturkunde Berlin and funded by Medienboard and MFG.

My part is the sound of the 360° film, the one that shows the real locations: the same places where the animated animals live later on. Both parts had to feel like they belong together, even though one was filmed and the other rendered.

The mobile 360° version never became a by-product. In the museums it runs alongside the room-scale installation, usually as an introduction or to bridge the waiting time before visitors get into the walk-in version. So a mix made in 2018 for headphones on an Oculus Go is still working in exhibitions in Berlin, Helsinki, and elsewhere.

Looking back, the project is mostly an example of how much work sits between “we have 360° footage” and “this feels like a place.” That work is invisible. You only notice it when it is missing, and then immediately. It was much the same on Crossing Borders, where documentary 360° material also only found its direction through spatial sound.

Images: © Interactive Media Foundation / Filmtank

Planning a 360° production for an exhibition, a trade fair, or a headset?

Nature film, museum installation, or VR documentary: sound decides whether your audience is in a place or just watching a video. Let’s talk early about which deliverables you actually need at the end.