Spatial mixing of effects

I got to perform a piece with the new group at Yale called [A]² (pronounced “A squared”) during their debut last December. The group explores creative possibilities of the human voice using live effects and processing through Ableton Live. Although the concept of “voice as instrument” has been explored in the past, the group does this all live. The director of the group, Jacob Reske ’14, discussed the idea with me last year and we came up with the name [A]² to stand for “Augmented Acapella” or “Ableton and Acapella”. After leaving the group for a semester, I had a chance to join in again on their debut concert and feature some of the HCI work I have been doing.

In this video, the Leap Motion was used to modulate effects during a live performance. The singers were sampled in a preceding performance and sliced into an Ableton Drum Rack. The Leap Motion was interfaced with Ableton through Masaykuki Akamatsu’s leapmotion.aka object and a max patch I developed. Different effects such as reverb, bit-crusher, low-pass and delays were placed at different vertices of an imaginary 3 dimensional cube above the Leap Motion (See figure). During performance, I can dynamically mix multiple effects by  moving one hand while the other hand is free to trigger samples. Moving to one edge of the cube activated the associated effect. As I move to different vertices, I can organically mix audio effects in an intuitive and expressive manner.

Although modulating effects using motion is not a new concept, the Leap Motion allows a non invasive method of doing so with unprecedented accuracy. I found that I was able to  play effects in an expressive manner not possible with other interfaces. For example, the Kinect is unable to track  small and subtle hand movements as it is designed for larger body gestures. Multitouch interfaces like the iPad allow control of multiple effects, but there are physiological limitations that prevent each finger from being truly independent. The “audio effects cube” provides an intuitive interface to mix between different effects in a live performance setting.

The inspiration for this project came from my experience from DJing and live music performance. DJ’s and live electronic musicians usually control effects using knobs, sliders or at best some kind of multitouch interface. I found knobs and sliders to be severely limiting in expressivity since I could only control two parameters. Although software like Ableton Live and Traktor Pro allow “chaining” of multiple effects to one “macro” knob, this approach did not allow fine tuning of each effect. Using the “Effects Cube” allows me to modulate each effect individually. I have tried various iPad apps that allow control of multiple effects through various interfaces such as vertices of a shape or bouncing balls on an XY pad. Although more intuitive than knobs and sliders, I found these to be limited by my hand’s physiological capabilities. For example, my thumb can only go so far away from my index finger. The Effects Cube does not suffer this problem. Later, I plan to introduce additional micro variations by accounting for finger movement while modulating an effect.

The performance also features MIT Media Lab’s Makey Makey. I wired a system that was concealed underneath the cardboard and also edited the code to work with Ableton Live. The chopsticks have nothing to do with the tracking and was employed only for comical effect. However, the Leap Motion does have a special “pencil” mode designed to detect long slender objects. This was not used in the performance.