VR concert experience

SoundScopePad provides an active music listening experience using AI, virtual reality (VR), and spatial acoustics. Users can find the performer that they want to hear and focus on them by emphasizing their performance.

RIKEN’s SoundScopePad provides an active music listening experience using AI, virtual reality (VR), and spatial acoustics. Users can find the performer that they want to hear and focus on them by emphasizing their performance.

VR concert experience

Performers stand around you and start playing, and you can see three at a time in front of you on your iPad screen. If you point your iPad to the left, the performer on the left will appear, and if you point it to the right, the performer on the right will appear.

Spatial acoustic experience

If you hold your iPad in front of you, you will see the application tracking your face on the display. By turning your head left and right, you can follow the sound of the performers on their respective sides. Likewise, if you turn around with your iPad in hand, you will be able to face the performers behind you.

Active music listening experience

This app provides an active music listening experience by enabling you to focus on certain performers that you want to hear. If you turn to face the performer of your choice, it will sound as if they are directly in front of you. By moving your left or right palm closer or further from your ear, you can adjust the range of the performance that you want to focus on. You can highlight one part or multiple parts of the performance such as the horn section and the rhythm section. You can check the direction of the performer and the direction you are facing on the position display.

Development history

The Music Information Intelligence Team, part of the RIKEN Center for Advanced Intelligence Project, is exploring potential new ways of creating and listening to music using AI. When listening to classical music, for example, you may want to hear the first violin a little more clearly, or suppress the sound of the trombone sound, or only hear the flute. SoundScopePad is an app that enables you to specify the sections you want to listen to.

The app is available to the public to gauge the overall interest in such a way of listening to music. We ask that users the app who are prompted to fill out a questionnaire to please submit their responses. We plan to add songs in the future. If you are interested in the app and technology, please contact us at the support URL.

References

[LINK] [SoundScopePad] Masatoshi Hamanaka: “Sound Scope Pad”, Proceedings of the 19th Sound and Music Computing Conference (SMC2022), pp.657-658, June 2022.