Gesture Recognition in Virtual reality


  • Meng Wu Jiangsu Second Normal University, People's Republic of China



gesture recognition, machine leaning algorithms, sensor technologies, VR


Virtual Reality (VR) provides users with a sensory experience that is close to reality, creating a sense of interaction. It is widely used, and the gesture recognition in VR also has a great effect. Gesture recognition enriches VR using experience and promotes a more direct and natural interaction. Gesture recognition usually employs sensors to collect data from users and machine leaning algorithms to interpret and respond to human activities. Complex gestures need more complex algorithms and more rigorous operations. The reason is that complex gestures mean larger quantity of data. If data is larger, the harder to get robust and effective datasets. Then, features can also become difficult to extract, contributing to misrecognition or unrecognizable. Though machine leaning algorithms are widely used in gesture recognition, there are still some important challenges need to be addressed, like lack of standardization and limitations of availability of diverse and large datasets. However, VR, gesture recognition and machine leaning algorithms all have excellent prospect, because they are in line with the development of the Times and show the progress of science and technology. This paper not only focuses on their advantages but also does not ignore their shortcomings, and looks at them comprehensively.


Angad, B., Khaldoun, A. M. M., Hussam, A. H., Shayan, Q., & Patrick, O. (2023). The Integration of Artificial Intelligence Into Patient Care: A Case of Atrial Fibrillation Caught by a Smartwatch. Cureus, 15(3), e35941-e35941.

Berk, C., Bahadir, A. M., Tolga, K. C., & Ufuk, C. (2024). Gaze-directed and saliency-guided approaches of stereo camera control in interactive virtual reality. Computers & Graphics, 118, 23-32.

Brian, C., Johnathan, H., Chris, H., Anthony, C., Amir, I., Emmanuel, B., Jo, S. A., Taylor, S., Wayne, S., Michael, T., Michael, B., & Ivona, C. (2024). Assessing the utility of high spectral resolution lidar for measuring particulate backscatter in the ocean and evaluating satellite ocean color retrievals. Remote Sensing of Environment, 300.

Caglar, Y. (2023). Point and Select: Effects of Multimodal Feedback on Text Entry Performance in Virtual Reality. International Journal of Human–Computer Interaction, 39(19), 3815-3829.

Cantone, A. A., Esposito, M., Perillo, F. P., Romano, M., Sebillo, M., & Vitiello, G. (2023). Enhancing Elderly Health Monitoring: Achieving Autonomous and Secure Living through the Integration of Artificial Intelligence, Autonomous Robots, and Sensors. Electronics, 12(18).

Caon, A. J., Sícoli, S. J. C., Vanessa, L. G., Gomes, d. S. E., Guilherme, F. L., Elena, d. S. S. E., Mendes, d. O. I., & Cipresso, P. P. H. (2023). High-resolution optical remote sensing geomorphological mapping of coral reef: Supporting conservation and management of marine protected áreas. Journal of Sea Research, 196.

Constanza, M., Fernando, A., Ignacio, V., & Julián, G. (2020). Developing an Innovative Medical Training Simulation Device for Peripheral Venous Access: A User-Centered Design Approach. Healthcare (Basel, Switzerland), 8(4).

Cun-jiang, Y., Guo-bao, Z., Cheng-wei, Y., xiao-ying, D., & Cheng-shuo, L. (2022). An Improved Gesture Recognition Model Based on Mini-Xception. Journal of Physics: Conference Series, 2400(1).

D.A.Sanaguano-Moreno, J.F.Lucio-Naranjo, R.A.Tenenbaum, & G.B.Sampaio-Regattieri. (2024). Real-time impulse response: a methodology based on Machine Learning approaches for a rapid impulse response generation for real-time Acoustic Virtual Reality systems. Intelligent Systems with Applications, 21.

Dongyun, K., Hoyoung, Y., & Youngjoo, L. (2021). Ultralow-Latency Successive Cancellation Polar Decoding Architecture Using Tree-Level Parallelism. IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS, 29(6), 1083-1094.

Elster, L., Staab, J. P., & Peters, S. (2023). Making Automotive Radar Sensor Validation Measurements Comparable. Applied Sciences, 13(20).

Eva, C., & Helder, A. (2022). An Experimental Assessment of Depth Estimation in Transparent and Translucent Scenes for Intel RealSense D415, SR305 and L515. Sensors, 22(19), 7378-7378.

Fan, Z., Manman, P., Yuanyuan, S., & Qiang, W. (2024). Hierarchical features extraction and data reorganization for code search. The Journal of Systems & Software, 208.

Guoliang, C., & Lin, W. (2022). A Novel Automatic Tracking Method of Moving Image Sequence Marker Points Uses Kinect and Wireless Network Technology. Wireless Communications and Mobile Computing, 2022.

Hongjiao, G., & Fan, L. (2024). The application of virtual reality technology in the teaching of clarinet music art under the mobile wireless network learning environment. Entertainment Computing, 49.

Huashi, Z., Zhichao, W., Yubin, H., Qiujia, F., Shouyu, L., Guang, M., Wenchao, L., & Qun, Y. (2023). Combination optimization method of grid sections based on deep reinforcement learning with accelerated convergence speed. Frontiers in Energy Research, 11.

HyungIl, K., Boram, Y., Young, O. S., & Woontack, W. (2023). Visualizing Hand Force With Wearable Muscle Sensing for Enhanced Mixed Reality Remote Collaboration. IEEE transactions on visualization and computer graphics, PP.

Jella, P. (2023). From E-Commerce to Virtual Commerce: The Exciting Opportunities of Virtual Shopping. NIM Marketing Intelligence Review, 15(2), 10-17.

Jestine, P., Yeling, J., & Mesut, A. (2023). Virtual reality technology for workplace training: The case for developing cultural intelligence. International Journal of Cross Cultural Management, 23(3), 557-583.

Jing, Q., Yinuo, Z., Weizhong, T., Wenming, C., Yu, Z., & Lingguo, B. (2023). Developing a virtual reality healthcare product based on data-driven concepts: A case study. Advanced Engineering Informatics, 57.

Jonathan, W. K. (2023). Distance Perception in Virtual Reality: A Meta-Analysis of the Effect of Head-Mounted Display Characteristics. IEEE transactions on visualization and computer graphics, 29(12), 4978-4989.

Josefine, S., Tonia, M., Danny, S., Maximilian, T., Jazan, O., Maciej, P., & Christian, H. (2022). A multimodal user interface for touchless control of robotic ultrasound. International journal of computer assisted radiology and surgery, 18(8), 1429-1436.

Jung-Wook, H., Jeongkyun, P., Rae-Hong, P., & Hyung-Min, P. (2023). Audio-visual speech recognition based on joint training with audio-visual speech enhancement for robust speech recognition. Applied Acoustics, 211.

Kamil, K., Thomas, C., & Marek, S. (2023). Visual programming simulator for producing realistic labeled point clouds from digital infrastructure models. Automation in Construction, 156.

Liang, L. (2023). Motion detail feature extraction of sports injury based on three-dimensional sensor tracking. Physical Communication, 61.

Lingjun, Z., Bo, L., Lijie, D., & Guanghui, G. (2023). Tough, recyclable and biocompatible carrageenan-modified polyvinyl alcohol ionic hydrogel with physical cross-linked for multimodal sensing. International Journal of Biological Macromolecules, 253(P4).

Lurdes, C. M. d., & Soumodip, S. (2024). A systematic review of virtual reality in tourism and hospitality: The known and the paths to follow. International Journal of Hospitality Management, 116.

Mahajan, R., & Padha, D. (2018). A Review On Various Techniques To Recognize Gesture Based Facial Expressions. Research Cell: An International Journal of Engineering Sciences, 27(1), 78-82.

Marc, L. W., K., M. J., & Madhumita, D. (2024). Augmented and virtual reality in hotels: Impact on tourist satisfaction and intention to stay and return. International Journal of Hospitality Management, 116.

Nikola, H., Tomislav, M., Ivan, U., & Stanko, Š. (2024). Use it early: The effect of immersion on spatial and design space aspects in team-based mechanical design reviews. Advanced Engineering Informatics, 59.

Panella, M., & Altilio, R. (2019). A Smartphone-Based Application Using Machine Learning for Gesture Recognition: Using Feature Extraction and Template Matching via Hu Image Moments to Recognize Gestures. IEEE Consumer Electronics Magazine, 8(1), 25-29.

Patricia, L., Diego, I., Alejandro, M., Fernando, O. J., Pilar, P. E., & Rocío, B. (2022). Machine Learning-Based Processing of Multispectral and RGB UAV Imagery for the Multitemporal Monitoring of Vineyard Water Status. Agronomy, 12(9), 2122-2122.

Patrick, S. (2023). Unbalanced visual cues do not affect search precision at the nest in desert ants (Cataglyphis nodus). Learning & behavior.

Perry, A. . B. S. (2023). Book Review: Rituals for Virtual Meetings: Creative Ways to Engage People and Strengthen Relationships. Adult Learning, 34(4), 260-261.

Proffitt, R., Ma, M., & Skubic, M. (2023). Development and Testing of a Daily Activity Recognition System for Post-Stroke Rehabilitation. Sensors, 23(18).

Salihbegovic, F. (2020). The Encounter with the Real: What Can Complicite’s Theatre Performance 'The Encounter' Teach Us about the Future of VR Narratives? Body, Space & Technology, 19(1), 125-152.

Seda, A., Xiaolong, L., Qiyuan, W., Paige, M., YueHin, L., Jed, J., Joey, H., Laura, O., Narutoshi, H., & Axel, K. (2023). Virtual Planning and Patient-Specific Graft Design for Aortic Repairs. Cardiovascular engineering and technology.

Serena, S., Marco, A., Milena, M., & Marco, R. G. (2023). Experimental validation and uncertainty analysis of an innovative IoT infrared sensor for wall thermal transmittance measurement. Measurement Science and Technology, 34(12).

Shengcai, D., Le, W., Aiping, L., & Xun, C. (2023). Alignment-Enhanced Interactive Fusion Model for Complete and Incomplete Multimodal Hand Gesture Recognition. IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society, PP.

Shinji, S., Miho, A., Toru, S., Yuta, T., Jun, H., Taishi, W., Yuki, M., Sukchan, K., Saeri, K., Yuki, S., Kensuke, S., Yoshiaki, A., & Shigenori, K. (2023). Evaluation of electrical activity at the carpal tunnel area in response to median nerve elbow stimulation using magnetoneurography. Clinical Neurophysiology, 156, 89-97.

Shui’er, H., Sujin, K., & Hyun, J. J. (2023). The effect of visual rivalry in peripheral head-mounted displays on mobility. Scientific Reports, 13(1), 20199-20199.

Stanley, M., Krisha, G. V., Christine, C., Matteo, Z., Rezia, M., & Damien, C. (2022). Improving Haptic Response for Contextual Human Robot Interaction. Sensors, 22(5), 2040-2040.

Takeshi, K., Xiao, M., Santhosh, S., Thije, B., Gerard, d. H., Bart, P., Razvan, P., Daniel, T., Roy, V., B., A. H., Moreno, H. L., Florian, d. R., Itai, L., Fujito, Y., J., J. R. A., A., M. E., Jisk, K. A., & H., G. G. (2023). A touchless user interface based on a near-infrared-sensitive transparent optical imager. Nature Electronics, 6(6), 451-461.

Tamanna, N., & Waqar, N. (2022). A research protocol of an observational study on efficacy of microsoft kinect azure in evaluation of static posture in normal healthy population. Journal of Datta Meghe Institute of Medical Sciences University, 17(1), 30-33.

Tanmay, D. S. (2023). Augmented reality applications and the future library. Library Hi Tech News, 40(9), 7-11.

Tarkov, M. S., & Chiglintsev, E. A. (2012). Reducing the dimensionality of the data in the problem of diagnosing thyroid disease. Optical memory & neural networks, 21(2), 119-125.

Tianchen, X., Xiaohua, R., Jiale, Y., Bin, S., & Enhua, W. (2023). Efficient Binocular Rendering of Volumetric Density Fields With Coupled Adaptive Cube-Map Ray Marching for Virtual Reality. IEEE transactions on visualization and computer graphics, PP.

Un, L. S., Jinwook, K., & Jeongmi, L. (2023). Effects of Reward Schedule and Avatar Visibility on Joint Agency during VR Collaboration. IEEE transactions on visualization and computer graphics, PP.

Viet, L., & Luca, C. (2022). Practical Approach to Digitally Simulate Nonsynoptic Wind Velocity Profiles and Its Implications on the Response of Monopole Towers. Journal of Structural Engineering, 148(1).

Wang, J. (2021). A Review of Deep Learning on Medical Image Analysis. Mobile Networks & Applications, 26(1), 351-380.

Wang, S. (2015). Pathological Brain Detection by a Novel Image Feature—Fractional Fourier Entropy. Entropy, 17(12), 8278-8296.

Wang, S. (2021). Advances in data preprocessing for biomedical data fusion: an overview of the methods, challenges, and prospects. Information Fusion, 76, 376-421.

Wang, S.-H. (2021a). COVID-19 classification by CCSHNet with deep fusion using transfer learning and discriminant correlation analysis. Information Fusion, 68, 131-148.

Wang, S.-H. (2021b). Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network. Information Fusion, 67, 208-229.

William, E., Dominique, B., & Mohsen, Z. (2023). Will visual cues help alleviating motion sickness in automated cars? A review article. Ergonomics, 31-34.

Yongyi, L., Ghaffar, A., & Abdul, R. A. (2023). Advances in geothermal energy prospectivity mapping research based on machine learning in the age of big data. Sustainable Energy Technologies and Assessments, 60.

Zhang, Y. (2015). Pathological brain detection in MRI scanning by wavelet packet Tsallis entropy and fuzzy support vector machine. SpringerPlus, 4(1), Article 716.

Zhang, Y. (2016). Smart detection on abnormal breasts in digital mammography based on contrast-limited adaptive histogram equalization and chaotic adaptive real-coded biogeography-based optimization. Simulation, 92(9), 873-885.

Zhang, Y. (2017). Pathological brain detection in MRI scanning via Hu moment invariants and machine learning. Journal of Experimental & Theoretical Artificial Intelligence, 29(2), 299-312.

Zhang, Y.-D. (2016). Computer-aided diagnosis of abnormal breasts in mammogram images by weighted-type fractional Fourier transform [Article]. Advances in Mechanical Engineering, 8(2), Article 11.

Zhang, Y.-D. (2018). Twelve-layer deep convolutional neural network with stochastic pooling for tea category classification on GPU platform. Multimedia Tools and Applications, 77(17), 22821-22839.

Zhanming, C., Huawei, T., & Huiyue, W. (2023). User-Defined Foot Gestures for Eyes-Free Interaction in Smart Shower Rooms. International Journal of Human–Computer Interaction, 39(20), 4139-4161.




How to Cite

Wu, M. (2023). Gesture Recognition in Virtual reality. Psychomachina, 1(1), 1–10.