Well-FormedRSS Feed

Design Technology at the Intersection of Art and Science

Augmented Listening

Stop for a second and listen. Close your eyes, use your ears, and just listen.

Whether you are in a quiet office environment or out on a busy street, you'll be amazed by how many sounds there are around you. Most of us do not pay attention to the ambient sounds that surround us. Our brains filter them out and we don't listen. Yet the sounds we miss can be very enjoyable.

Designed Sounds

Today, what we hear in our daily lives is often designed sound- music and sound effects carefully crafted for games, devices, and products. For example, mission-critical products, such as heart rate monitors used during medical surgery or a plane’s flight deck controls, use distinctive alarming sounds that are designed to be easy to perceive and raise a sense of urgency or danger.

In interfaces for everyday tasks, sound is used to create engaging and beautiful experiences. Sounds can generate a special feeling or underline brand identity while simultaneously providing cues that a command has been received by the system. Most smart phones today come with subtle sounds that indicate the pressing of a touch screen’s virtual buttons. Since there is no way to feel if a virtual button has been pressed, the sounds reinforce the action for the user. Another example can be found in industrial design, where the latest electric cars are being designed with artificial motor sounds. The sounds alert pedestrians to the car as well as reinforce the sense of driving a powerful vehicle. These examples underline the overall trend of sound being used to create an aesthetic experience rather than serving as purely a functional aid to improve interaction.

Blurring the Border Between Listening and Composition

While systems and products are becoming more enjoyable and pleasant to listen to, they are usually not intentionally designed for sound interaction. The emergence of accessible music software on computers and mobile devices is changing this. These programs allow for easy modification of sound by the average user and blur the border between listening and sound creation. The small form and limited complexity of mobile interfaces has forced music software designers to reduce the complexity of their products, resulting in music software that is widely used by average mobile phone users.

Music apps are often top sellers. Popular applications allow people to become mobile DJs, to transform sounds, and to design ringtones.

I was interested in exploring the blur between sound creation and listening when my friend and colleague Matteo Penzo put me in contact with Matteo Milani from the U.S.O. Project sound art group. The ideas and compositions of the U.S.O. Project revolve around the use of noise and ambient sound as a foundation for sound installations and music composition. Together we wanted to create a mobile experience that would support active listening to the everyday sounds that surround us, making the listener a part of a personal sound installation. Instead of creating a tool for recording and transforming sound, we wanted to start from the sounds themselves. Our goal was to reinforce the sounds of the listener’s environment while blending them with more musical sounds. Together the sounds would form a unique experience that could be enjoyed by anybody that has an interest in sound and art.  

Early Experiments

We started with a small prototype app for iOS using simple sound algorithms to blend U.S.O. music with live recording from the iPhone microphone. The prototype was tested with real use cases that included listening to the app while taking a long walk as well as while sitting at the computer in the office. We added many parameters for the user to be able to tweak and play with the sound transformation.The parameters were mapped to on-screen sliders and buttons and to sensors like the accelerometer.

While doing the informal tests we found that the users were struggling to understand the relationship between the parameters and the sound output. Also, in most cases they would end up spending time experimenting with the parameters to discover how they work. The visual interface and controls were clearly distracting, taking attention away from the app’s original goal of reinforcing ambient sounds for the listener.  

Following these early experiments, we decided to take a drastically different approach.We limited the visual interface as much as possible and provided a set of sound themes in the app for the listener to select. This worked much better. All of a sudden the users would pick up the app and, once started, would tuck it away in a pocket while listening to the sounds. Each theme takes sounds from the microphone and blends them with sounds composed by U.S.O. Project. The sounds are blended using sound algorithms, unique to each theme. Each algorithm is carefully calibrated to replicate the work and skill that goes into producing a great listening experience.

Lis10er

The result is Lis10er (pronounced Listener), an augmented sound installation app. Sounds are blended from the listener’s surroundings, creating dynamic music that changes while maintaining its identity. Lis10er provides users with a creative way of listening to their environment and a unique experience with every listen. 

Tue Haste Andersen is a senior software architect based in frog’s Milan studio. Tue is a human computer interaction and computer music expert, with research ranging from DJ work practices to the use of sound and music in common interaction tasks. He is also the founder and original author of the popular open source DJ software, Mixxx.