Augmented Audio: How Much Potential Does the AR App Have for Audio-Based On-The-Go Experiences?

Augmented Reality (AR) is an exciting technology that is becoming increasingly popular. Generally, AR apps are accessed through smartphones or tablets, allowing users to interact with digital content overlaid on the real world. We can already try out AR apps on modern smartphones and visually dive into augmented reality. The necessary technology in modern devices such as smartphones and tablets makes it possible to use AR apps without special hardware like AR glasses, since these devices provide powerful displays, cameras, and sensors. The camera is a key component for AR functionality, enabling the device to capture and superimpose virtual content in real time. In addition to hardware, appropriate software, a compatible platform, and specialized tools are essential for running and building AR apps. For the development and use of AR apps, certain technical fundamentals are also required, which both beginners and professionals should take into account. The creation of AR experiences has evolved significantly, with new AR creation tools and platforms making it easier for developers to innovate and expand the possibilities of AR technology. Developers play a crucial role in driving innovation and advancing the practical applications of AR.

To access AR apps, users typically need to install the app on their device, ensuring compatibility with iOS, Android, or iPhone. Many AR apps are also available on the web, providing easy access without installation. The phone serves as the primary interface for most AR experiences, bridging the physical and digital worlds.

This article is based on the master’s thesis by Carina Denz, which examined the potential and possible design spaces of AR audio applications in public spaces.

Introduction to Audio-Based Experiences

Audio-based experiences are rapidly transforming the way we connect with our environment, thanks to the rise of augmented reality and spatial audio. Today’s AR apps are not just about visuals—they’re about creating immersive experiences that blend sound, technology, and the real world. Whether you’re exploring interactive stories, playing games, or discovering new forms of entertainment, AR apps are making it possible to experience life in entirely new ways.

The role of AR apps in education

Augmented Reality apps open up completely new possibilities for education to convey knowledge and make complex topics tangible. By using modern AR technologies, learners can experience 3D models, videos, images as triggers or backgrounds for AR experiences, and interactive content directly in their own environment. Abstract knowledge becomes a vivid experience that sustainably supports the learning process.

Whether mathematics, biology, physics, or history – AR apps offer a wide range of functions to present learning content in an exciting and understandable way. With applications such as GeoGebra AR, mathematical concepts can be visualized as 3D models and viewed from different angles, allowing users to explore objects in detail. In biology, apps like Anatomy 4D make it possible to explore the human body in 3D and interactively discover individual organs or systems. Such Augmented Reality applications make it possible not only to read about topics but to experience them directly, often using a picture or pictures as visual aids to enhance understanding.

A major advantage of AR apps in education is the increase in learning motivation. Interactive and immersive content promotes engagement and active participation among learners. AR apps also help students brainstorm and visualize ideas before making decisions, supporting creative thinking. This often leads to a deeper understanding and better learning outcomes. In addition, AR apps make education more flexible: learning content can be conveyed independently of time and place – ideal for mobile learning with a smartphone or tablet. Many AR platforms allow teachers and students to upload images, videos, or 3D objects for use in the classroom, making it easy to customize educational experiences.

The range of AR apps for education is steadily growing. Alongside GeoGebra AR and Anatomy 4D, there are countless other Augmented Reality apps that cover different subjects and age groups. They offer a wide variety of functions, from visualizing complex 3D models and objects to interactive videos and quizzes that help reinforce what has been learned.

Overall, AR apps demonstrate the enormous potential and possibility in combining technology and education. They make learning more visual, varied, and accessible – and will play an increasingly important role in the educational sector in the future. Anyone who already relies on innovative Augmented Reality AR solutions today is actively shaping the education of tomorrow.

What increases the potential of AR audio apps?

The technologies of 3D audio and head tracking are an important factor when it comes to audio-based AR apps. By combining these technologies, a realistic soundscape can be created around the user. The sound also contains more information than classic stereo sound. Through spatial sound and the tracking of head movements, the user knows where a sound is coming from – and the AR audio app knows where we are listening.

This additional spatial information can also be used as an advantage, for example by working more with signal tones. Speech can in principle also be used, but it can become tiring when a high amount of information has to be expressed awkwardly through words.

“Your destination is on the right.”

Well great – which right?! A sound specifically designed for the user interface could express this information much more clearly. And this naturally opens up new design and interaction possibilities for audio-based apps that are used on the go. Modern AR audio apps offer numerous new functions and AR features, such as integration of environmental sounds, individual feature extensions, or the ability to deliberately shape the interaction between user and environment. The excitement generated by immersive audio experiences is driving more users and creators to explore these new possibilities. The incredible capabilities of current AR audio technology allow for highly advanced and compelling user experiences that were not possible before.

Another major advantage is the social acceptance of headphones in public spaces. When you walk through a city, you see people with small in-ear or large over-ear headphones everywhere. These modern earbuds are seen not only as technical devices, but also as accessories and lifestyle products – Apple is once again a good example here with its wireless AirPods. The use of these devices, with special features such as head tracking, adaptive noise cancellation, and personalized audio functions, opens up new possibilities for immersive audio experiences.

AR audio can also be inclusive, especially when hearing aids no longer stand out in everyday life. Due to this social acceptance, headphones are an excellent basis for AR audio apps. There are already the first products on the market that offer 3D audio and head tracking. These AR audio apps push the boundaries of what is possible in audio-based AR, enabling new forms of interaction and engagement.

You can find a headphone overview here: https://www.vrtonung.de/apps-spatial-audio-airpods-pro/

With AR audio apps, there is the potential to develop innovative solutions for a wide variety of application areas that are tailored to the individual needs of users. The interaction between user and environment plays a central role here, as it significantly influences the user experience and opens up new ways of communicating information. Looking into the future, there is hope for even greater advancements and broader adoption of AR audio technologies, further enhancing the possibilities for immersive and accessible experiences.

Spatial Audio in Interactive Stories

Spatial audio is at the heart of truly immersive experiences, giving users the ability to hear sounds as if they are coming from specific directions in 3D space. This technology is revolutionizing entertainment, games, and interactive stories by creating a powerful sense of presence and realism. Imagine being in a gaming environment where you can hear a character approaching from behind or sense the direction of an important event just by listening. Spatial audio allows users to feel like they are part of the story, not just observers. In interactive stories, this means users can explore virtual worlds, interact with characters, and experience the environment in a way that matches real life. The result is a deeper connection to the content, making every game, story, or entertainment experience more engaging and lifelike.

Challenges of AR audio

Audio-based AR apps also come with challenges. For example, AR audio apps can help users avoid a mistake by providing real-time audio feedback if they are about to perform an incorrect action. When they are used in public spaces such as a city, the following points should be considered. Even a bit of improvement in sound design can enhance the user experience, making interactions more engaging. Additionally, new challenges or features can pop up unexpectedly when deploying AR audio apps in public environments, requiring flexible solutions.

Group situations

When creating content for AR audio apps, you need to consider whether the target person is on their own or interacting with several people. Most apps like Spotify, podcasts, and similar services usually only consider a single individual. There shouldn’t only be AR audio experiences for individuals.

Whether with friends or strangers, when people experience something together, they usually want to talk about it. AR audio apps can help here by specifically supporting communication in groups and thereby improving the shared experience. For example, AR audio can enable you to share immersive soundscapes or interactive content with a friend or a group of friends, making the experience more social and collaborative.

For such group experiences, noise cancelling or similar modes should be used carefully. The other person’s voice needs to remain intelligible; otherwise, the headphones will be perceived as a disturbance. Features like noise awareness or transparent hearing could solve this problem. This function is already built into most premium AR-audio-capable headphones.

Designing virtual sounds

When designing UI sounds, it’s important to pay attention to clear prioritization. Users should be able to distinguish important signals from notifications. The different sounds should also be clearly distinguishable from one another. In addition, it is important that UI sounds are designed in such a way that humans can localize them well.

This presents sound designers with various challenges. There are still very few rules of thumb that I can use in my daily work. You need to clearly differentiate whether sounds are generated in an organic or synthetic way. Designers can draw inspiration from both organic and synthetic sources when creating 3D sounds, allowing for more expressive and contextually relevant soundscapes. A certain level of experience is required when creating such 3D sounds.

A maximum of three audio sources at the same time

People cannot listen to an unlimited number of audio sources simultaneously. At most, we can focus on two sources and listen actively. Spatial sound can make this easier. A third source of sound should only be used as background noise. And these virtual background sounds should only be used when real background noise is being filtered out via noise cancelling.

This should not be confused with the approach used by, for example, Dolby Atmos. In that case, the goal is to create an enveloping audio experience. In 3D music mixes, you can sometimes localize more than three instruments at once. The point there is that they work well together in context. For conveying information via AR audio apps, however, different rules apply.

High noise levels in busy locations

In inner cities or busy public places, there is usually a high noise level. Headphones without noise cancelling can only block loud environmental sounds physically, by sealing the ears. If noise cancelling is used, low-frequency sounds, which are common in urban environments, are primarily reduced.

However, these can be safety-relevant sounds such as approaching cars or warnings from other people. AR audio apps can help users stay aware of car sounds, even in noisy environments, by selectively allowing important vehicle noises to pass through. I remember that on my university campus, the first electric cars used to drive by almost silently. Even without wearing headphones, you had to be extremely careful not to miss what was happening around you.

When designing an AR audio app for use on the go, you should therefore consider how noise cancelling can be used in the most sensible way. Is it always needed, or only in specific parts of a city? In the future, smart noise cancelling could also be used, which activates itself only at important moments.

The diversity of environments

Cities and large public buildings offer a wide variety of environments. For sound design, it is relevant whether the listener is slowly walking through a park or quickly moving through a busy shopping street. These different environments demand differing levels of attention from the listener. This should be taken into account when designing the virtual soundscape. Ideally, an AR audio app should also be tested in different environments.

The example above with electric cars shows that the soundscape of our cities will change in the near future. Cars are becoming quieter and quieter. The question now is how to deal with this. A quieter urban environment is certainly a positive development. Nevertheless, there are use cases in which AR audio can be used to reintroduce artificial sounds.

You might know this effect from a restaurant: when there are only a few guests, the room feels empty. Only as more visitors arrive and the level of conversation increases do we notice how the environment seems to fill with life. Music can help bridge this silence and create a pleasant atmosphere. In a similar way, AR audio can be used independently of location to positively influence our mood.

Conclusion: Potential across different domains

There is definitely potential for audio-based AR apps in public spaces. Our sense of hearing, and particularly spatial hearing, is important for many interactions with our environment. Through 3D audio and head tracking in headphones, many exciting applications become possible. With these technologies, AR audio apps can provide a realistic soundscape with spatial information, opening up new possibilities. Past developments in AR audio, such as early hardware and trigger image techniques, have laid the groundwork for today’s innovations. An overview of the different topics and applications shows that AR audio apps can be used in areas such as navigation, education, tourism, marketing, and design in many different ways and serve as innovative solutions for a variety of requirements. These apps can be deployed in any country, making them accessible and relevant on a global scale.

There is also great potential for general information delivery in public places as well as in the smart city sector. Companies can deliberately integrate the use of AR audio apps as part of broader digital solutions, in order to optimize business processes, marketing, or training and create new user experiences. Features like auto sound adjustments or auto-play can further enhance user convenience and adaptability in different environments.

It becomes clear that there are many ways to shape the sound of the future with the right AR sound and specialized apps. In particular, the merging of digital and real reality through AR audio apps opens up new opportunities for innovative applications and supports companies and users in integrating immersive audio solutions into a wide range of subject areas. Curious what that might sound like? I’m happy to help answer those questions. So take part in shaping this future now and send me an inquiry.