This is an Interactive Music System (IMS) that uses Extended Reality (XR) and Spatial Audio technologies for a musical multi-track looping session. Every track is represented by an “Agent,” a quasi-autonomous entity embodied as a sound source in space visualized as a virtual sphere.

I have two versions, both of which implement the XR component in Unity. The first was developed for the Microsoft HoloLens and includes a motion capture system, a spatial audio system, and a physical MIDI controller. A video of this version is shown below.

The second version uses the Meta Quest 3 headset, and considering that this device allows hand-tracking, the motion capture from the previous version was no longer required, which allows for greater portability. A video showing different users interacting with this version is presented below.

For additional details you can check this article: https://www.mn.uio.no/ifi/english/people/aca/pedroplu/xr-human-swarm-interactive-music-system.html

Code in Unity for the HoloLens: https://github.com/pedro-lucas-bravo/mr_agent_visualizer

Code in Max for the Controller and Sound Synthesis: https://github.com/pedro-lucas-bravo/agent_music_improviser

Related Publications