MEinVR: Multimodal Interaction Paradigms in Immersive Exploration

Immersive environments have become increasingly popular for visu-alizing and exploring large-scale, complex scientific datasets because of their inherent features: immersion, engagement, awareness, etc. Virtual Reality has brought rich opportunities for supporting a wide variety of novel interaction techniques, such as tactile and tangible interactions, gesture interactions, voice commands, etc. Multimodal interaction refers to that users are equipped with multiple modes for interacting with data. However, it is still important to determine how these techniques can be used and combined as a more natural interaction metaphor. In this paper, we aim to explore interaction techniques combining VR controller with voice input for a novel multimodal experience. We present MEin VR, a multimodal interaction technique that enables users to manipulate 3D molecular data in the virtual environment. Users can use VR controller to specify location or region of interest, and use voice command to express the tasks that they intend to perform on the data. This combination can serve as an intuitive means for users to perform complex data exploration tasks in immersive settings. We believe that our work can help inform the design of multimodal interaction techniques that incorporate multiple inputs for 3D data exploration in immersive environments.