Video exploration using AR

MusicScopePad is a tool for quickly finding a desired video by playing multiple videos at the same time in AR space. You can explore videos using AI, augmented reality (AR), and spatial sound.

RIKEN’s MusicScopePad is a tool that allows you to quickly find a video you want to watch by playing multiple videos at the same time in AR space. When you launch the app, multiple videos will be presented around you, depending on the orientation of your iPad. A video can be selected by searching for the video you want and pausing in that direction to listen. Video exploration can be an enjoyable experience using AI, augmented reality (AR), and spatial sound.

Put on your earphones or headphones and press the play button in MusicScopePad. Then, ten videos will start playing around you. You can see five at a time on the iPad screen, and if you move the iPad left and right around you, you can see the videos to the right, left, and behind the screen.

If you want to emphasize the sound of a particular video, put your hand to your ear and stop to listen. The video in front of you will be emphasized so that can hear it clearly. If you look at this video long enough, it will be selected, and playback will start in a larger size. If you want to stop playing and search again, hold your palm up to the iPad.

Development history

The Music Information Intelligence Team, part of the RIKEN Center for Advanced Intelligence Project, is exploring potential new ways of creating and listening to music using AI. When listening to classical music, for example, you may want to hear the first violin a little more clearly, or suppress the sound of the trombone sound, or only hear the flute. SoundScopePad is an app that enables you to specify the sections you want to listen to.The app is available to the public to gauge the overall interest in such a way of listening to music. We ask that users the app who are prompted to fill out a questionnaire to please submit their responses. We plan to add songs in the future. If you are interested in the app and technology, please contact us at the support URL.