Tangible and Mid-Air Interactions in Hand-Held Augmented Reality for Upper Limb Rehabilitation: An Evaluation of User Experience and Motor Performance

Hand-held augmented reality (AR) offers accessible, interactive rehabilitation options for patients with upper limb motor deficits. Incorporating hand-involved interactions (e.g., tangible and mid-air interactions) into hand-held AR provides patients with intuitive manners to perform rehabilitation exercises mimicking real-world activities. Previous work has shown the importance of user experience and motor performance in rehabilitation systems, but little was known in the literature regarding the impact of hand-involved interactions in hand-held AR on user experience and motor performance in rehabilitation exercises. Hence, this study aims to evaluate user experience and motor performance when using three types of hand-involved interactions in hand-held AR rehabilitation: (1) tangible cube (i.e., a space-multiplexed tangible interaction with a physical cube acting as a real proxy to manipulate a virtual object in the same form); (2) tangible controller (i.e., a time-multiplexed tangible interaction with a physical controller applied to manipulate a virtual object); and (3) hand motion (i.e., a form of mid-air interaction to move a virtual object with hands). Based on the findings from self-report, electroencephalography (EEG), and performance measures, this study reveals the advantages of the tangible cube over the tangible controller, both superior to the hand motion in hand-held AR rehabilitation regarding user experience and motor performance. This study offers new understanding of the advantages and disadvantages of various interaction techniques in hand-held AR rehabilitation, emphasizing crucial design considerations for these systems, with a focus on user experience and motor performance in upper limb rehabilitation.