M3: "Wings", for Solo Clarinet and Automated Accompaniment Video Animation

Chris Raphael: Christopher Raphael received his PhD in Applied Mathematics at Brown University in 1991, where his studies focused on statistical pattern recognition. He held a postdoctoral appointment in the Statistics department at Stanford, was a research scientist in Arabic character recognition at Bolt Beranek and Newman, and held a faculty position in the Department of Mathematics and Statistics at the University of Massachusetts, Amherst. He came to Indiana University in 2004 where he leads the Music Informatics group in the School of Informatics and Computing and Engineering. His music research includes musical accompaniment systems, algorithmic musical analysis, modelling of musical expression, and optical music recognition. His work in music informatics fuses long-standing interests in music, statistics, and computation.

Nikki Pet: Nikki Pet is a masters student at the Yale School of Music, studying with David Shifrin. Prior to studying at Yale, Nikki received a B.A. in computer science from Columbia University, where she was also a member of the Juilliard Exchange Program, studying with Alan Kay. Nikki also produces multimedia performances dedicated to engaging audiences unfamiliar with classical music. Live and digital productions include Stravinsky’s L’Histoire du Soldat, Prokofiev’s Peter and the Wolf (arr. wind quintet), and Bach’s Second Violin Partita (arr. clarinet and oboe duo). Many of these performances can be found on Nikki’s YouTube channel: youtube.com/c/claripet

Kaitlin Pet: Kaitlin Pet is an Informatics PhD student in the Intelligent and Interactive Systems track at Indiana University SICE school of Informatics, Computer Science and Engineering. Her research focuses on many aspects of music and artificial intelligence, including music visualization, score-following based flexible and live media, and improving the technology around remote chamber and orchestral playing. She holds a BA in Biology and Computer Science from Columbia University and a GPD from the Hartt School in oboe performance, where she concentrated on oboe chamber performance.

"Wings" is a work for solo clarinet written by world-renowned classical composer Joan Tower in 1986. Our clarinetist and animator, Nikki Pet, approached Joan Tower to augment her work with animation that emphasizes musical nuance through visual media. We sought to take this video animation one step further by transforming it into a live concert experience using our new system for “visual accompaniment”.

Our visual accompaniment system is based on the Informatics Philharmonic automated accompaniment system created by Chris Raphael (described in detail in a previous publication The Informatics Philharmonic works by simultaneously performing 1) online score following, determining solo note onset times as the performance evolves, and 2) future note onset scheduling, or predicting when a soloist will place future notes before they occur. This predictive ability allows the Informatics Philharmonic to stretch an audio accompaniment track so it “meets” the soloist at its next note onset. Our current work extends this idea from audio accompaniment to “video accompaniment”, where a pre-made video is stretched in real time to match a live player. In our current iteration of the software, we modified the existing Informatics Philharmonic application to send out predicted times of future soloist notes as they are calculated. These times are received by a MaxMSP program that controls the video frame playback rate of Nikki Pet’s Wings animation. A test run of Wings recorded by Nikki Pet in her home can be seen here