An Interactive VR experience – Eddie and I

At the beginning stands a simple question. How can people understand each other when words aren’t enough? This is exactly where Eddie and I, an interactive VR experience, begins. The stage: a nocturnal forest. The main character: Ron, eight years old and deaf. The user takes on Eddie. A friendly monster. A companion and helper. Together, closeness emerges. Not through sound. But through hands, gaze, and space.

Eddie and I is an interactive VR experience designed specifically for deaf people. At the same time, the sound gently guides you through the scenes. For everyone who can hear it. Original music carries the dramaturgy. It explains nothing but supports moments. It also opens space for feeling. In addition, hand tracking makes learning active. Gestures are understood. Reactions come immediately. This creates a cycle of action and feedback.

Attitude and starting point

I like working where technology brings people together. Not as a gimmick. But as a bridge. This project starts exactly there. It is an interactive VR experience that centers deaf people and brings hearing people along. The goal is clear: to make empathy tangible. Not through explanatory texts. But through action.

The experience is called “Eddie and I.” Directed by Maya Shekel of Moosh Studio. A co-production with Reynard Films and Studio Geppetto. An interactive VR experience that lasts around 25 minutes.

It uses hand tracking to playfully teach sign language and close communication gaps. The trigger is personal: the inspiration was Maya’s niece, who was born deaf.

Classical learning via YouTube remains passive. There is hardly any feedback. Nothing reacts in real time. In this interactive VR experience, the hand becomes a tool. Every gesture receives a response. This creates learning by doing.

Technology must take a stand. Here, it does. The story is simple. A boy named Ron. Eight years old. Deaf. The night before his first camping trip. Fear mixes with curiosity. His mother tells him about Eddie. A friendly monster that gives children courage in their dreams. But the story isn’t enough. Ron falls asleep. Wakes up in a magical forest. From here on, the user embodies Eddie. They accompany Ron and learn with him. In doing so, they build trust.

Who Eddie and I, an interactive VR experience, is made for

The experience is especially compelling because it is designed specifically for deaf people. Hand tracking in VR headsets is the core. Hands are language and also interface.

At the same time, sound remains a gentle guide. The music follows the dramaturgy. It lays out a route. But it does not push itself to the front. This way, hearing people feel guided without excluding deaf people.

Thus, Eddie and I is an interactive VR experience that is inclusive. Studies show a need. Around 70 percent of hearing people have never spoken with a deaf person. Many are unsure. This format lowers the barrier. It creates a safe space. It invites you to try things out. Without shame. Without test pressure.

Target groups are schools, families, educational institutions, museums, and everyone who takes inclusion seriously. The structure is clear. Short chapters. Understandable tasks. Immediate feedback. But also low barriers. Large icons. Clear contrasts. Short hints. The interface languages are currently English and German. This keeps the entry threshold low.

Eddie and I is an interactive VR experience suitable for workshops, classes, and events. A short warm-up on sign language basics up front. Then the immersion. And a conversation at the end. What worked? Where did silence help? Which gesture felt strong? This turns a demo into a real encounter.

The journey in virtual reality

The interactive VR experience begins calmly. Arriving. Breathing. Orienting. Then small tasks. Build a campfire. Look at the stars. Skip stones. Even overcome carnivorous plants. Everything is graspable. Everything via hand tracking.

Compared to classic controllers, hand tracking enables a more natural and intuitive interaction with the virtual world. The decision to use hand tracking instead of controllers was made to increase immersion and depth and to allow people to act directly with their own hands.

The 3D representation of the hands and their precise position in virtual space ensure that users can grasp, move, and manipulate virtual objects. Various actions like help, try, or look are triggered by specific gestures.

Hand movements are captured by sensors and cameras integrated into the devices used. This opens up diverse interaction possibilities and increases immersion. At certain points in the game, special challenges or interactions are offered that demand users’ attention and skill.

Ron responds through gaze, facial expressions, and movement. This way, the user learns the signs in context. No vocabulary list. But relationship.

In one chapter, there is complete silence and that is intentional. Both characters must cope without sound. This is precisely where the interactive VR experience shows its strength.

It makes communication physical. It compels attention. No button-mashing. No menu jungle. Instead: closeness, space, timing. The duration of about 25 minutes is tight. Suitable for online play. Festival-ready. Classroom-ready. The pacing stays brisk. Pauses are set. Successes come quickly. Mistakes are allowed. Feedback stays friendly.

Those who hear follow music and sound. Those who do not hear follow light, haptics, and level design. This makes the interactive VR experience feel fair. No one is excluded. Everyone is addressed. That is the dramaturgical frame. The role of sound design and the creative use of 3D audio create a dense atmosphere and support orientation in space. And that is exactly the message: we can understand each other. Even without words.

How it’s built with hand tracking

Interactive audio is demanding. You don’t deliver one piece of music and leave. You deliver many assets and different elements. Loops that carry. One-shots that set events. Parameters that flow. States that transition cleanly. The targeted use of 3D audio and sound design provides immersive depth and supports spatial orientation in the game.

I worked in Unity because the project was set up there. FMOD was connected to precisely control the interactions. Gestures trigger one-shots and set off actions. Parameters modulate intensity, tempo, and layering.

The forest ambience was designed in 7.1.4. That is, with four height channels. This creates a real roof overhead. Leaves and insects audibly move around you. The space feels real and conveys a special depth.

Localization remains stable. For me, that’s a step up in quality compared to Ambisonics with HRTF. Especially in height imaging. Fewer comb filters. More precision. Thus sound leads without being dominant.

The interactive VR experience uses this as quiet navigation. A crack front left. Or a chirp behind and then a breath of air above. The user intuitively senses where to go next. At the same time, the experience remains robust. Drop-outs are avoided. Transitions are crossfaded. Loops are seamless. System load is monitored.

Hand tracking provides stable pose recognition and enables precise capture of hand movements via sensors and camera. Evaluation of the tracking methods shows high reliability and accuracy in determining the position of the hands in virtual space.

The 3D representation of the hands makes it possible to grasp, recognize, and manipulate virtual objects with the hands, thereby triggering various actions. Different devices and VR devices such as headsets and special hand-tracking devices are supported, opening the possibility to interact on various platforms like Oculus Quest or Steam.

Integrating hand tracking into the system was implemented by the developers with a lot of work, combining elements such as audio, haptics, and visual feedback. Certain places and individual points in the code and design presented particular challenges to ensure a seamless and immersive user experience.

Who is working on it

“Eddie and I” is teamwork across several countries. Creative lead is Maya Shekel, founder of Moosh Studio in Tel Aviv and New York. Partners are Reynard Films and Studio Geppetto. Studio Geppetto is responsible for environment design.

A Quill workflow in VR was used. This created a sketchbook aesthetic that fits Ron’s imagination. Character design came from James A. Castillo.

Moosh Studio handled story and art direction: story lead Nitay Dagan, art direction Maor Sharvit. Animation was led by Liron Topaz. Development worked with ASL interpreters and deaf consultants.

The project was supported by Medienboard, FFF Bayern, CNC, AOR Film Foundation, Gisher Film Foundation, La Ville de Paris, Meta, and Unity for Humanity. In 2023, the team won at the NewImages XR Development Market. Invitations followed to the Tribeca Creators Market and the Biennale Gap Financing Market. The premiere will take place in 2025 in the Venice Immersive Competition. The project is not a tech showcase. It is an offer of connection. That is exactly what the audience should feel.

Where audio games are effective and what comes next

Use cases are diverse. Schools. Museums. Libraries. Media centers. Festivals. An interactive VR experience also makes sense at home.

A workshop flow is easy to build. Short warm-up on signs. Then the experience. Afterwards, conversation prompts. What surprised you? How did silence feel? Which gesture sticks with you? This turns entertainment into a learning experience. And learning into empathy.

Next steps are clear. Sharpen hand tracking further. Especially transitions between poses. Modularize ambience. More micro loops. Finer states. Softer crossfades. Consider additional languages. Subtitles as a low-barrier optional add-on.

All without losing the core idea. Eddie and I is primarily intended for deaf people. It remains quiet when silence is appropriate and uses music when music carries. It also guides without dominating. And it shows that connection is possible. Even without a shared spoken language.

If you have ideas for similar projects, you are welcome to contact me. I look forward to working with you.