Beyond The Controllers: Hand Tracking and Gesture Recognition for an Immersive Virtual Skiing Experience

Over the past 30 years, virtual reality has evolved beyond entertainment into industrial, medical, and sports applications, allowing realistic training and performance analysis in controlled environments. This study presents the development of a virtual reality-based skiing simulation. The environment consists of the Horres and Beauvoir slopes, within the Mount Jaffreau ski area (Ambin Group) in Bardonecchia, Turin. The project aims to push the boundaries of conventional skiing simulations, focusing on enhancing the user’s kinesthetic perception of skiing. Our approach integrates advanced motion-tracking technologies to improve user interaction. This application leverages both hand tracking and gesture recognition, allowing users to engage with the virtual environment naturally and intuitively. These techniques enable a more immersive and responsive skiing experience, where users can simulate essential movements such as shifting their balance, using ski poles, and adjusting their posture dynamically without the need for external controllers. As a key design principle, motion-based interaction was prioritized, ensuring that the simulation captures the nuances of skiing mechanics. This is particularly crucial for sports and training applications, as it provides athletes and recreational skiers with a more accurate representation of real-world skiing dynamics. A user study was conducted involving 17 participants with varying levels of expertise in skiing and VR. Data were collected through semi-structured interviews and the NASA-TLX questionnaire, focusing on several experiential dimensions including ski perception, physical fatigue, interaction usability, and enjoyment. The findings suggest that gesture-based interaction significantly enhances user engagement and realism, particularly among experienced skiers. However, limitations such as the absence of haptic feedback and environmental resistance were also noted. The outcomes of this research underscore the potential of controller-free interaction in VR simulations and contribute to ongoing discussions within the fields of human-computer interaction and immersive system design. Future developments aim to further improve the realism and applicability of virtual reality skiing simulations by incorporating several advanced features. For example, it may include haptic feedback integration to provide users with a tactile sense of snow resistance or ski edge control, biomechanical tracking to analyze and refine skiing techniques, customization of the landscape and weather conditions, and AI-driven adaptive difficulty.