Since the establishment of VR glasses as serious playback devices in the entertainment industry, head tracking technology has found a wide range of applications.
That the potential of 360° sound is far from exhausted can be seen in the latest developments in the field of spatial audio. With the introduction of head tracking enabled headphones and pods, the two big players Apple and Samsung are taking a giant step towards establishing 3D audio in the consumer market – out of its niche existence.
The following article will give an overview of the currently most important apps for the playback of spatial audio via handheld devices such as smartphones and tablets.
First of all, I think it’s important to lay out the basic way these applications work. Not just from a technological perspective, but from a conceptual one. This is to prevent the ‘wrong’ use of this technology and accompanying disappointments.
Also, the terminology surrounding immersive audio technology is currently being thrown around wildly. Apple talks about 3D spatial audio with dynamic head tracking. Samsung talks about a 360 audio function with an intelligent motion-tracking sensor. Bose AR even talked about Augmented Audio Reality (ARR) back then.
But everyone is ultimately talking about immersive, cinema-like surround sound that detects the direction of the headphones’ gaze. A spoiler directly related to the topic of movies: the manufacturers currently activate the function almost exclusively in combination with moving images. However, there are great applications that also work without pictures, for example 3D podcasts and audio plays. Or immersive 3d music with Dolby Atmos that I will talk about later.
For head tracking to work in terms of freedom of movement in a virtual sound scene, a reference point is needed. Additionally to the audio device, this framework is provided by the smartphone or tablet. The headphones, therefore, know from the position of the playback device from which direction the sounds should come.
But be careful, this does not mean that the sound only comes from the direction of the cell phone; as just mentioned, this only serves to calibrate the sound scene. This way you can make sure that, for example, sounds that should come from the left really do come from the left.
The crucial aspect for using said headphones is the software, i.e. the smartphone apps and the right operating system. Even if standard headphones are sufficient for a static, spatial 3D audio impression. For head tracking, it must be ensured that the corresponding apps also support spatial audio.
In the use case, these are mostly various streaming platforms. During my research, I had the feeling in some user reports that it is not always clear that there has to be compatibility here. To avoid such discrepancies for the loyal reader of this blog, the following should provide clarity about the hardware and software components. So that nothing stands in the way of a 3D audio experience.
Apple pioneered spatial audio with head tracking for handheld devices with the introduction of the AirPods Pro and AirPods Max. With Apple, surround sound formats like 5.1 and 7.1, as well as Dolby Atmos, are supported. However, this does not mean that every streaming provider that offers surround sound or Atmos is also compatible with Apple. The following video gives a short overview of the AirPods Pro. It gets interesting starting at 0:57.
Apart from the streaming services, the hardware and operating system must also be right. In terms of hardware, you need an iPhone 7 or newer, or an iPad. Also, the operating system iOS 14 or iPadOS 14 is required, but how the technical requirements look in detail you can read here.
It’s always a bit difficult to describe how the sound sounds. Regardless, here a short conclusion from my perspective as a spatial audio engineer:
|Air Video HD||The app that lets you stream your own videos from your Mac or MacBook to your phone or tablet supports Spatial Audio.|
|Apple TV App||The in-house streaming service offers a wide range of movies and series in Spatial Audio formats.|
|Discovery+||The streaming service for friends of non-fictional entertainment also offers Spatial Audio content in combination with AirPods.|
|Disney+||Disney Plus also offers spatial audio experiences in combination with AirPods, for example with the blockbuster Avengers: Endgame or the hit series The Mandalorian.|
|File Explorer||This data management app works like a cloud and thus allows data playback on different devices. For audiovisual content, Spatial Audio formats are supported, and was able to successfully create a 5.1 surround demo as a video|
|Foxtel Go||The Australian Internet TV service, which offers some live channels as well as some video-on-demand content, also works with Apple’s Spatial Audio feature.|
|HBO Max||The video-on-demand service provider HBO Max also uses Spatial Audio support.|
|Hulu||The now very popular streaming provider Hulu offers content in the Dolby Digital Plus 5.1 surround sound format supported by Apple.|
|Plex||Plex is a client-server media playback system that can access different content like video, audio, and photos. For the Spatial Audio feature to work here, "Old Video Player" must be enabled in the preferences.|
|Peacock||The streaming service Peacock also uses spatial audio content.|
|Paramount+||The recently renamed streaming service from CBS All Access to Paramount Plus also supports Apple’s Spatial Audio feature.|
|Vudu||The video-on-demand service provider offers content in Apple-supported spatial audio formats Dolby Digital Plus 5.1 Surround Sound and Dolby Atmos.|
|Spatial Scenes||The Spatial Scenes app offers a variety of spatial soundscapes. Designed for the iPad, the app also supports spatial audio playback via AirPods.|
|Tidal||The premium music streaming service Tidal offers high-resolution music in stereo as well as some tracks mixed in 3D audio (see Music Streaming Blog). Corresponding songs are also supported by Apple.|
|Netflix||Actually, Netflix doesn’t belong on this list (yet?). Earlier this year, the rumor emerged that Netflix and Apple were working on 3D audio support for the AirPods. However, this has since been revised again. Update 21/08: Now the time has come, 3D Audio is slowly being rolled out by Apple. Dolby Atmos already provided the basis as I saw coming, yeah!|
|M1 MNTR CTRL||Mach1 Spatial enables head-tracking for Apple wearables and Bose AR|
Shortly after Apple’s AirPods, Samsung sent the Galaxy Buds Pro into the race. They are using Dolby Head Tracking technology (TM). What is Spatial Audio for Apple is 360 Audio for Samsung – two terms for the same concept.
However, the devil is in the details. Currently, the Spatial Audio works automatically with every video. This means that even if the video only contains mono or stereo sound, the sound is artificially virtualized. Thus, you can hear the sound clearly directed from the cell phone. But it’s not an immersive, enveloping sound like the Apple Airpods manage. So it doesn’t benefit from multi-channel surround – or 3D sound. Which is a pitty that I feel like will be fixed in the future.
Of course, to take advantage of this technology in the first place, the hardware and software have to be right. 360 Audio works on Samsung Galaxy phones and tablets with the latest One UI 3.1 operating system. How to listen to the 360 Audio function can be found here.
The feature was initially announced exclusively for new models like the Galaxy S21. However, shortly after, the new operating system was also rolled out for my S10 and I can now also listen to media in 360 audio.
As mentioned, the Galaxy Buds do not enable 3D audio in the sense of 360° sound. The general advantage of this approach is that sound virtualization works with all sound sources. Thus, the function is activated as soon as audio or video content is consumed on Facebook, LinkedIn, YouTube, or wherever.
For example, I was on my phone listening to a podcast and scrolling through social media. As soon as a video popped up in my news feed, the 360 audio automatically activated. Then I heard the podcast suddenly switching from stereo to spatial audio. As well as the video that was then spatialized playing simultaneously. This is a good way to hear what the function does with media in the first place.
However, there is a good chance that this will be improved. Basically, the Galaxy Buds Pro have the same hardware like the Apple Airpods. The platforms also usually use Dolby Atmos or comparable multi-channel audio formats in the backend. So, we can be curious.
Although the focus of this article is on head tracking, a few more apps should be mentioned here that do not support head tracking, but can still provide immersive listening experiences.
This is the soft entry into immersive 3D audio, for all standard stereo headphones. The sound here is binaural, accessible for everybody.
This app offers guided meditations and mindfulness exercises. Also, there are sound worlds with 3D recordings of the "most beautiful places in the world". e.g. stream, rain, mountain stream, forests, oceans, and so on.
Unfortunately, the recordings are huge files with 45-500 minutes. So most of the time 800MB have to be downloaded first, which definitely gets in the way of relaxation.
Similar to Headspace, Synctuition is an app for relaxation and meditation. But here they didn’t just use dummy head recordings for the binaural sound. Music was also added and mixed into the space using spatialization tools.
So an entire sound experience was created and combined with binaural beats, which I will take a closer look at at some point. These audio experiences take 25 minutes and also a significant loading time. This could also be done better. But we are getting there.
Audible Reality offers the possibility to play stereo music, for example via Spotify, in a different spatial context. The music is played in a virtual space, which is called "vibes" in the app. The whole thing is reminiscent of the YouTube hype around 8D Audio. Similar sound colorations can be found here, such as bass-boosted music.
I think the sound personalization aspect is great, but unfortunately, the stereo suffers a lot here. Phasing artifacts can be heard. Sonarworks’ approach with Sound-ID is better thought out here.
Even though we’ve just ticked off the topic of head tracking, there’s a little oddball. The audio software manufacturer has its own app that can be combined with any headphones. So you can not only connect the hardware head tracker with the computer. Via Bluetooth, this is ready for use with the smartphone.
Finally, you can listen to music or your own audio in the app. Matching the Nx Virtual Mixing Room, the stereo sound now sounds as if you were sitting in Abbey Road Studio 3. However, multi-channel 5.1 surround sound has not been implemented here, as it is in the DAW plugin.
There are even dedicated headphones that have the head tracker built into the device. The Audeze Mobius 3D Headphones. So in principle, exactly what Apple did with the Airpods Max – only years later 😉
Here, too, a small special role for Sony 360 Reality Audio. Again, the idea is to immerse and directly dive into the sound. More precisely into music, because here artists are also in the foreground. This is produced with Sony 360RA ‘s own tools, the Creative Suite, as it is also used on TIDAL. There are also dedicated headphones here to further enhance the sound experience: The Sony 1000XM2, wireless headphones with noise canceling. Or the Sony WF-1000XM4 in-ears true wireless headphones.
Meanwhile, some music streaming platforms offer music in Dolby Atmos or 360 Reality Audio. Here, we would like to refer to this blog entry.
As you can see, there are now really many ways to enjoy spatial audio content. Fortunately, most of the already very popular streaming services recognize the potential of immersive listening experiences. Hopefully, head tracking compatible headphones will find favor. Because in combination with the streaming hype, this could be the biggest chance yet for 3D audio to finally find its rightful recognition in the market!
But the right technology also needs the right content. No matter whether Apple Airpods-Pro or Samsung Galaxy Buds-Pro. Up to now, existing material has often been "converted" into corresponding spatial audio formats, or upmixed.
But the true potential of spatial audio lies in creating content specifically to get the best out of the technology with creativity. And that’s exactly my job here at VRTonung, if you want to learn more about it or even already have ideas in mind, don’t hesitate to get in touch!