• Son Luu

Magic Windows Final Project Ideation

Updated: Nov 19, 2020

Class: Magic Windows

Instructor: Rui Pereira

My childhood is very much tied to hanging out with friends on the city streets, buying cheap toys and snacks from the street vendors. The sounds of street vendors could be a singing voice or interesting ways of them advertising the items they are selling. Street vendors can be considered a big part of a city's culture and lifestyle in many countries, especially in non-Western cultures. However, over time, with advanced automation, along with reorganization and planning of urban spaces to accommodate new social infrastructural and economic challenges, a big part of this culture no longer exist in many big cities. Many street vendors' sounds are even pre-recorded and played-back nowadays. In addition to these changes, more and more vending machines have been occupying commercial buildings where people no long have the need to leave the building for the convenience of purchasing smaller products.

At times, when buying snacks from a vending machine, especially after years of living in Western culture where street vendors may not be part of the culture, I miss the sounds of the street vendors, especially the ones who even know your name by heart because you used to be a regular customer.

So, for the final project, I'd like to create an audio AR experience for customers of vending machines. As a customer enters the AR experience on their phone at different vending machines at different locations, while purchasing an item from those machines, they would hear the sound of a street vendor selling similar item(s) to them. The transaction in the real world remains to be just between the human and the machine, but the street vendor sound will be augmented to such reality to bring an interesting, nostalgic feeling of human interaction that either doesn't exist anymore or is rare to find in modern days and cultures, and probably become even rarer in the future.

The plan is to develop the experience in AR Foundation with spatial audio.

Technical Exploring Journey

Trying with Vuforia

The strategy is to place an object upon image target recognition. And then, have a sound file attached to that image, to be triggered for playback.

To do this, I need to add Audio listener to the AR camera, and add Audio Source to the Image Target:

Vuforia requires sound file playback to be coded inside Default Trackable Event Handler of Image Target.

So, next step is to get into Edit Script inside Default Trackable Event Handler.

Create 2 public variables to allow trigger of sound playback:

Additional codes to enable sound to be played when image target recognized and stop when image is not recognized or disappear. The added codes are marked "added" in the below script.

The image tracking to trigger sound worked, but the spatial feature of the sound didn't.

I also tried to build and run the application on my phone but for some reason, the app could not develop.

At this point, I moved on to testing with AR Foundation Image Tracking and I also came up with a new inspiration for my project at this point....

Trying with AR Foundation

With AR Foundation, I used the Unity's Image Tracking AR Foundation sample.

Similarly to Vuforia, I added Audio Listener to the AR Camera and Audio source to a 3D object to be augmented based on an image recognition.

I wrote some code for image tracking...:

And, it actually worked for the image tracking. But, I didn't have time to figure out the audio component which was key to what I was trying to create....

I will continue to work on it, but I created this video to share what my vision for the output would be like...