Audiovisual Rock

Interactive Media

(work in progress)

Music, media, and design. I signed up for an open mic during the Covid pandemic, which was also technically my first solo performance. However, due to the imposed lockdowns, the performance was meant to be online. Instead of performing over a webcam I thought I could mix some of the interaction design tools I was learning at UCL and carry on with my performance in a different way, screensharing my performance over TouchDesigner instead; thus, I declared I was participating in an open mic with an Audiovisual Performance. The event carried on, and the performance was a beautiful mess, but it planted the seed for interdisciplinary experimentation I wanted to pursue. After working on capturing real-time MIDI input from an analog guitar, I now have a working prototype for a true rock style, audiovisual performance.

This virtual version of myself was documented through a Microsoft Kinect. Video shows a point cloud version of myself reacting to melody and chords played on guitar.

The first steps:

The intial version I displayed at the open mic was not sound reactive. Video displays a soundless motion test before performance together with Touch OSC interface I developed to control visuals. The problem with having a remote controller to control visuals is free hands are required, which I do not have when playing guitar, or at least the presence of 2 people. Given I was performing alone, no one could modify the values on the remote… leading me to a better idea. To be truly audiovisual sound must match visual, and as an interactive musician that’s exactly what should happen. Volume and individual notes should be independent triggers.

I have now begun to bring musical values to TouchDesigner, but a lot of MIDI fine tuning is required before more interesting stuff starts to happen.

And learn how to play better of course…

Touch OSC controller for first “audiovisual rock” performance. In current state, interface forces me to stop playing to change visuals, or rely on a second person to modify values.

Visual test for second virtual performance scenario. Fiery silhouette recorded through Microsoft Kinect.

Updates coming soon!