PRiSM Musical Audio Gesture Recognition (AGR)

PRiSM Musical Audio Gesture Recognition (AGR, or previously Musical Gesture Recognition) is a machine-listening interface designed to recognise and classify real-time audio input.

PRiSM AGR research emerged out of an artistic residency of composer Professor George Lewis (Edwin H. Case Professor of American Music, Columbia University), whose RNCM PRiSM commission, Forager, saw him work closely with RNCM staff and student performers in 2022. The research involved developing and releasing a fully standalone, open-source and open-access AGR toolkit, which allows for preparing and thus creating custom datasets of any musical audio gestures (e.g. an upward glissando) and real-time recognition of these gestures based on live audio input. The toolkit utilises machine-learning techniques to classify and interpret musical audio gestures, enabling applications in interactive music performance, composition, and more.

Following Lewis’ residency in 2022, which resulted in a prototype AGR unit being added to the existing Voyager framework, the development of a fully self-contained AGR system was driven by PRiSM-associated artists collaborating closely with Dr Hongshuo Fan (PRiSM Research Software Engineer 2023-24) and the International Contemporary Ensemble (ICE) based in New York. The finalised software tool was launched officially through a headline premiere concert on 16 May 2024 at the acclaimed Roulette Intermedium in Brooklyn (and livestreamed to an international audience). It was also featured in Lewis’ The Reincarnation of Blind Tom (2024) – a double concerto for human Soloist, AI-pianist and orchestra – for SWR Symphonieorchester and Donaueschinger Musiktage that was premiered in Donaueschingen, Germany in October 2024.