MirrorFugue is a project by PhD candidate Xiao Xiao and Prof. Hiroshi Ishii in the Tangible Media Group at the MIT Media Lab. The system consists of a player-piano with a projector displaying the upper half of a recorded player, creating the illusion of a pianist playing from the reflection.
The pianist’s movement including body, facial expressions and fingerings are displayed onto the keys and piano surface. MirrorFugue embeds these embodied expressions – lost in simple audio recordings – into a format users can physically interact with.
I worked with Xiao Xiao on extendingMirrorFugue into an integrated piano pedagogy, performance and composition environment. A teacher is able to record specific exercises encouraging the learner to pay attention to a phrase’s expressive arc. During improvisation, a spur of inspiration on the keyboard can be saved and played back in MirrorFugue. Composers can integrate and “tag” musical sketches on staff paper with piano realizations in MirrorFugue.
The above video features original music by myself and Xiao Xiao. Sheet music available upon request.
My experience working with Xiao Xiao was baptism by fire into the world of HCI. I still remember when Xiao Xiao told me to “prototype the interaction.” I was puzzled by such a bizarre statement, “prototype an interaction?” Having been trained in the physical sciences, my concept of “prototyping” meant testing a mechanical connection before it is machined, making a scale model for stress analysis, but not “interaction”…who prototypes an interaction? Very quickly, I learned just how intricate and detailed user interaction and experience was. Should I move this little box a few centimeters to the right or the left? I finally had a glimpse of the quantum leap in TMG’s vision of material computation and Radical Atoms. JamSheets and TRANSFORM for example, “offload” the computation to changes in the material’s shape.
My experience with Prof. Ishii in TRANSFORM and Xiao Xiao in extending MirrorFugue exposed me to design and interaction considerations further downstream from my academic specialization in Physics and Music. In turn, they changed the way I approach upstream disciplines like science and engineering. Shifting across this spectrum and exploring how each stage should inform another was one of most stimulating and exciting parts of being in TMG! I am now very interested in the ways HCI and Things to Think with can fundamentally change our interaction with scientific inquiry. A chemical version of Topobo or Senspectra could feature components that are aware of possible reactions and repel and attract accordingly. A physicist interested in nanoparticle synthesis, but lacking domain knowledge, can interact with chemical reactions with quicker results than reading the equivalent theory. Scientific theory embedded in material computation could even revolutionize STEM education from K to 12 and into college.
{ lh.hantrakul } | Research Blog 