Using Video-Game Technologies in New Interfaces for Musical Expression

Adriana Sá (2025) Using Video-Game Technologies in New Interfaces for Musical Expression. In Teaching Sound: Aesthetics & Praxis, CILECT - International Association of Film and Television Schools.

Among the artists and researchers who have explored music with 3D environments, many have made explicit references to gaming while exploring participatory interaction, video-game player paradigms, and allusive iconography. I have also been using 3D technologies since 2007, but in a quite different way. My 3D software operates based on the sound from an acoustic instrument, and the image is intended to work like a reactive stage scene.

Whatever motivates a practitioner, it is useful to think of how interaction designs and audio-visual relationships drive experience. I will begin this paper with a short description of how 3D engines render sound, and will then proceed with an overview of the diversity of approaches to music in 3D environments. Following this, I will analyse convergences and divergences between the notion of flow in music and gaming. Then, I will draw from perception science and film theory to analyse how audio-visual relationships and the dynamics of sound and image drive attention. Armed with these tools, it is possible to glean how any work exploring music with 3D environments conducts perception.