Touch, Sound, and Space: Exploring Immersive Music Interaction through AI-Generated Environments

Wanfang Xu; Jifan Yang; Fengwen Zhang; Yu Lu; Lijie Yao; Le  Liu; Lingyun Yu

doi:10.1145/3769534.3769589

Touch, Sound, and Space: Exploring Immersive Music Interaction through AI-Generated Environments

Wanfang Xu, Jifan Yang, Fengwen Zhang, Yu Lu, Lijie Yao, Le Liu, Lingyun Yu^*

^*Corresponding author for this work

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

Abstract

We introduce an interactive music system powered by AI-generated content (AIGC) that enables users to engage with music through multimodal interactions involving touch, sound, and spatial immersion.
Motivated by the desire to enhance engagement and emotional connection with music, our system enables users to co-create and interact with musical content. Users upload a song and a descriptive text prompt, from which the system generates 3D visuals. During playback, users can embed their own audio inputs and trigger responsive visual effects such as color-driven point clouds using tangible controls.
To explore how spatial scale and embodiment shape user experience, we implement the system across three increasing spatial scale and embodiment: (1) a handheld AR music box, (2) a table-sized stage box, and (3) a fully immersive VR environment. Through a user study, we investigate how different levels of immersion and interaction influence user engagement, emotional response, and sense of presence. Our findings demonstrate the potential of combining AIGC with embodied interaction to enrich creative expression and enhance immersive musical experiences.

Original language	English
Title of host publication	2025 International Symposium on Visual Information Communication and Interaction (VINCI)
Pages	1-8
Number of pages	8
DOIs	https://doi.org/10.1145/3769534.3769589
Publication status	Published - Dec 2025

Keywords

Immersive Creation
Music Generation
Visual Design

Access to Document

10.1145/3769534.3769589

TDF: Empowering Students, Not Replacing Thinking: A Framework for Critical and Responsible AI Use in Programming Education
Yao, L. (PI), Leach, M. (Team member), Dong, D. (Team member), Bu, Q. (Team member) & Sun, J. (Team member)
1/09/25 → 31/08/28
Project: Internal Research Project
A First Design Space of Visualization in Motion
Yao, L. (PI)
1/01/25 → 31/12/27
Project: Internal Research Project

Cite this

@inproceedings{83f53dcf946549808baf2e3471f2fd18,

title = "Touch, Sound, and Space: Exploring Immersive Music Interaction through AI-Generated Environments",

abstract = "We introduce an interactive music system powered by AI-generated content (AIGC) that enables users to engage with music through multimodal interactions involving touch, sound, and spatial immersion. Motivated by the desire to enhance engagement and emotional connection with music, our system enables users to co-create and interact with musical content. Users upload a song and a descriptive text prompt, from which the system generates 3D visuals. During playback, users can embed their own audio inputs and trigger responsive visual effects such as color-driven point clouds using tangible controls. To explore how spatial scale and embodiment shape user experience, we implement the system across three increasing spatial scale and embodiment: (1) a handheld AR music box, (2) a table-sized stage box, and (3) a fully immersive VR environment. Through a user study, we investigate how different levels of immersion and interaction influence user engagement, emotional response, and sense of presence. Our findings demonstrate the potential of combining AIGC with embodied interaction to enrich creative expression and enhance immersive musical experiences.",

keywords = "Immersive Creation, Music Generation, Visual Design",

author = "Wanfang Xu and Jifan Yang and Fengwen Zhang and Yu Lu and Lijie Yao and Le Liu and Lingyun Yu",

year = "2025",

month = dec,

doi = "10.1145/3769534.3769589",

language = "English",

pages = "1--8",

booktitle = "2025 International Symposium on Visual Information Communication and Interaction (VINCI)",

}

TY - GEN

T1 - Touch, Sound, and Space: Exploring Immersive Music Interaction through AI-Generated Environments

AU - Xu, Wanfang

AU - Yang, Jifan

AU - Zhang, Fengwen

AU - Lu, Yu

AU - Yao, Lijie

AU - Liu, Le

AU - Yu, Lingyun

PY - 2025/12

Y1 - 2025/12

N2 - We introduce an interactive music system powered by AI-generated content (AIGC) that enables users to engage with music through multimodal interactions involving touch, sound, and spatial immersion. Motivated by the desire to enhance engagement and emotional connection with music, our system enables users to co-create and interact with musical content. Users upload a song and a descriptive text prompt, from which the system generates 3D visuals. During playback, users can embed their own audio inputs and trigger responsive visual effects such as color-driven point clouds using tangible controls. To explore how spatial scale and embodiment shape user experience, we implement the system across three increasing spatial scale and embodiment: (1) a handheld AR music box, (2) a table-sized stage box, and (3) a fully immersive VR environment. Through a user study, we investigate how different levels of immersion and interaction influence user engagement, emotional response, and sense of presence. Our findings demonstrate the potential of combining AIGC with embodied interaction to enrich creative expression and enhance immersive musical experiences.

AB - We introduce an interactive music system powered by AI-generated content (AIGC) that enables users to engage with music through multimodal interactions involving touch, sound, and spatial immersion. Motivated by the desire to enhance engagement and emotional connection with music, our system enables users to co-create and interact with musical content. Users upload a song and a descriptive text prompt, from which the system generates 3D visuals. During playback, users can embed their own audio inputs and trigger responsive visual effects such as color-driven point clouds using tangible controls. To explore how spatial scale and embodiment shape user experience, we implement the system across three increasing spatial scale and embodiment: (1) a handheld AR music box, (2) a table-sized stage box, and (3) a fully immersive VR environment. Through a user study, we investigate how different levels of immersion and interaction influence user engagement, emotional response, and sense of presence. Our findings demonstrate the potential of combining AIGC with embodied interaction to enrich creative expression and enhance immersive musical experiences.

KW - Immersive Creation

KW - Music Generation

KW - Visual Design

U2 - 10.1145/3769534.3769589

DO - 10.1145/3769534.3769589

M3 - Conference Proceeding

SP - 1

EP - 8

BT - 2025 International Symposium on Visual Information Communication and Interaction (VINCI)

ER -

Touch, Sound, and Space: Exploring Immersive Music Interaction through AI-Generated Environments

Abstract

Keywords

Access to Document

Fingerprint

Projects

TDF: Empowering Students, Not Replacing Thinking: A Framework for Critical and Responsible AI Use in Programming Education

A First Design Space of Visualization in Motion

Cite this