Gesture Recognition in Virtual reality

Authors

  • Meng Wu Jiangsu Second Normal University, People's Republic of China

DOI:

https://doi.org/10.59388/pm00336

Keywords:

gesture recognition, machine leaning algorithms, sensor technologies, VR

Abstract

Virtual Reality (VR) provides users with a sensory experience that is close to reality, creating a sense of interaction. It is widely used, and the gesture recognition in VR also has a great effect. Gesture recognition enriches VR using experience and promotes a more direct and natural interaction. Gesture recognition usually employs sensors to collect data from users and machine leaning algorithms to interpret and respond to human activities. Complex gestures need more complex algorithms and more rigorous operations. The reason is that complex gestures mean larger quantity of data. If data is larger, the harder to get robust and effective datasets. Then, features can also become difficult to extract, contributing to misrecognition or unrecognizable. Though machine leaning algorithms are widely used in gesture recognition, there are still some important challenges need to be addressed, like lack of standardization and limitations of availability of diverse and large datasets. However, VR, gesture recognition and machine leaning algorithms all have excellent prospect, because they are in line with the development of the Times and show the progress of science and technology. This paper not only focuses on their advantages but also does not ignore their shortcomings, and looks at them comprehensively.

References

Angad, B., Khaldoun, A. M. M., Hussam, A. H., Shayan, Q., & Patrick, O. (2023). The Integration of Artificial Intelligence Into Patient Care: A Case of Atrial Fibrillation Caught by a Smartwatch. Cureus, 15(3), e35941-e35941. https://doi.org/10.7759/cureus.35941

Berk, C., Bahadir, A. M., Tolga, K. C., & Ufuk, C. (2024). Gaze-directed and saliency-guided approaches of stereo camera control in interactive virtual reality. Computers & Graphics, 118, 23-32. https://doi.org/10.1016/j.cag.2023.10.012

Brian, C., Johnathan, H., Chris, H., Anthony, C., Amir, I., Emmanuel, B., Jo, S. A., Taylor, S., Wayne, S., Michael, T., Michael, B., & Ivona, C. (2024). Assessing the utility of high spectral resolution lidar for measuring particulate backscatter in the ocean and evaluating satellite ocean color retrievals. Remote Sensing of Environment, 300. https://doi.org/10.1016/J.RSE.2023.113898

Caglar, Y. (2023). Point and Select: Effects of Multimodal Feedback on Text Entry Performance in Virtual Reality. International Journal of Human–Computer Interaction, 39(19), 3815-3829. https://doi.org/10.1080/10447318.2022.2107330

Cantone, A. A., Esposito, M., Perillo, F. P., Romano, M., Sebillo, M., & Vitiello, G. (2023). Enhancing Elderly Health Monitoring: Achieving Autonomous and Secure Living through the Integration of Artificial Intelligence, Autonomous Robots, and Sensors. Electronics, 12(18). https://doi.org/10.3390/ELECTRONICS12183918

Caon, A. J., Sícoli, S. J. C., Vanessa, L. G., Gomes, d. S. E., Guilherme, F. L., Elena, d. S. S. E., Mendes, d. O. I., & Cipresso, P. P. H. (2023). High-resolution optical remote sensing geomorphological mapping of coral reef: Supporting conservation and management of marine protected áreas. Journal of Sea Research, 196. https://doi.org/10.1016/J.SEARES.2023.102453

Constanza, M., Fernando, A., Ignacio, V., & Julián, G. (2020). Developing an Innovative Medical Training Simulation Device for Peripheral Venous Access: A User-Centered Design Approach. Healthcare (Basel, Switzerland), 8(4). https://doi.org/10.3390/HEALTHCARE8040420

Cun-jiang, Y., Guo-bao, Z., Cheng-wei, Y., xiao-ying, D., & Cheng-shuo, L. (2022). An Improved Gesture Recognition Model Based on Mini-Xception. Journal of Physics: Conference Series, 2400(1). https://doi.org/10.1088/1742-6596/2400/1/012020

D.A.Sanaguano-Moreno, J.F.Lucio-Naranjo, R.A.Tenenbaum, & G.B.Sampaio-Regattieri. (2024). Real-time impulse response: a methodology based on Machine Learning approaches for a rapid impulse response generation for real-time Acoustic Virtual Reality systems. Intelligent Systems with Applications, 21. https://doi.org/10.1016/J.ISWA.2023.200306

Dongyun, K., Hoyoung, Y., & Youngjoo, L. (2021). Ultralow-Latency Successive Cancellation Polar Decoding Architecture Using Tree-Level Parallelism. IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS, 29(6), 1083-1094. https://doi.org/10.1109/TVLSI.2021.3068965

Elster, L., Staab, J. P., & Peters, S. (2023). Making Automotive Radar Sensor Validation Measurements Comparable. Applied Sciences, 13(20). https://doi.org/10.3390/APP132011405

Eva, C., & Helder, A. (2022). An Experimental Assessment of Depth Estimation in Transparent and Translucent Scenes for Intel RealSense D415, SR305 and L515. Sensors, 22(19), 7378-7378. https://doi.org/10.3390/s22197378

Fan, Z., Manman, P., Yuanyuan, S., & Qiang, W. (2024). Hierarchical features extraction and data reorganization for code search. The Journal of Systems & Software, 208. https://doi.org/10.1016/J.JSS.2023.111896

Guoliang, C., & Lin, W. (2022). A Novel Automatic Tracking Method of Moving Image Sequence Marker Points Uses Kinect and Wireless Network Technology. Wireless Communications and Mobile Computing, 2022. https://doi.org/10.1155/2022/7033711

Hongjiao, G., & Fan, L. (2024). The application of virtual reality technology in the teaching of clarinet music art under the mobile wireless network learning environment. Entertainment Computing, 49. https://doi.org/10.1016/J.ENTCOM.2023.100619

Huashi, Z., Zhichao, W., Yubin, H., Qiujia, F., Shouyu, L., Guang, M., Wenchao, L., & Qun, Y. (2023). Combination optimization method of grid sections based on deep reinforcement learning with accelerated convergence speed. Frontiers in Energy Research, 11. https://doi.org/10.3389/FENRG.2023.1269854

HyungIl, K., Boram, Y., Young, O. S., & Woontack, W. (2023). Visualizing Hand Force With Wearable Muscle Sensing for Enhanced Mixed Reality Remote Collaboration. IEEE transactions on visualization and computer graphics, PP. https://doi.org/10.1109/TVCG.2023.3320210

Jella, P. (2023). From E-Commerce to Virtual Commerce: The Exciting Opportunities of Virtual Shopping. NIM Marketing Intelligence Review, 15(2), 10-17. https://doi.org/10.2478/nimmir-2023-0011

Jestine, P., Yeling, J., & Mesut, A. (2023). Virtual reality technology for workplace training: The case for developing cultural intelligence. International Journal of Cross Cultural Management, 23(3), 557-583. https://doi.org/10.1177/14705958231208297

Jing, Q., Yinuo, Z., Weizhong, T., Wenming, C., Yu, Z., & Lingguo, B. (2023). Developing a virtual reality healthcare product based on data-driven concepts: A case study. Advanced Engineering Informatics, 57. https://doi.org/10.1016/J.AEI.2023.102118

Jonathan, W. K. (2023). Distance Perception in Virtual Reality: A Meta-Analysis of the Effect of Head-Mounted Display Characteristics. IEEE transactions on visualization and computer graphics, 29(12), 4978-4989. https://doi.org/10.1109/TVCG.2022.3196606

Josefine, S., Tonia, M., Danny, S., Maximilian, T., Jazan, O., Maciej, P., & Christian, H. (2022). A multimodal user interface for touchless control of robotic ultrasound. International journal of computer assisted radiology and surgery, 18(8), 1429-1436. https://doi.org/10.1007/s11548-022-02810-0

Jung-Wook, H., Jeongkyun, P., Rae-Hong, P., & Hyung-Min, P. (2023). Audio-visual speech recognition based on joint training with audio-visual speech enhancement for robust speech recognition. Applied Acoustics, 211. https://doi.org/10.1016/J.APACOUST.2023.109478

Kamil, K., Thomas, C., & Marek, S. (2023). Visual programming simulator for producing realistic labeled point clouds from digital infrastructure models. Automation in Construction, 156. https://doi.org/10.1016/J.AUTCON.2023.105126

Liang, L. (2023). Motion detail feature extraction of sports injury based on three-dimensional sensor tracking. Physical Communication, 61. https://doi.org/10.1016/J.PHYCOM.2023.102210

Lingjun, Z., Bo, L., Lijie, D., & Guanghui, G. (2023). Tough, recyclable and biocompatible carrageenan-modified polyvinyl alcohol ionic hydrogel with physical cross-linked for multimodal sensing. International Journal of Biological Macromolecules, 253(P4). https://doi.org/10.1016/J.IJBIOMAC.2023.126954

Lurdes, C. M. d., & Soumodip, S. (2024). A systematic review of virtual reality in tourism and hospitality: The known and the paths to follow. International Journal of Hospitality Management, 116. https://doi.org/10.1016/J.IJHM.2023.103623

Mahajan, R., & Padha, D. (2018). A Review On Various Techniques To Recognize Gesture Based Facial Expressions. Research Cell: An International Journal of Engineering Sciences, 27(1), 78-82.

Marc, L. W., K., M. J., & Madhumita, D. (2024). Augmented and virtual reality in hotels: Impact on tourist satisfaction and intention to stay and return. International Journal of Hospitality Management, 116. https://doi.org/10.1016/J.IJHM.2023.103631

Nikola, H., Tomislav, M., Ivan, U., & Stanko, Š. (2024). Use it early: The effect of immersion on spatial and design space aspects in team-based mechanical design reviews. Advanced Engineering Informatics, 59. https://doi.org/10.1016/J.AEI.2023.102270

Panella, M., & Altilio, R. (2019). A Smartphone-Based Application Using Machine Learning for Gesture Recognition: Using Feature Extraction and Template Matching via Hu Image Moments to Recognize Gestures. IEEE Consumer Electronics Magazine, 8(1), 25-29. https://doi.org/10.1109/MCE.2018.2868109

Patricia, L., Diego, I., Alejandro, M., Fernando, O. J., Pilar, P. E., & Rocío, B. (2022). Machine Learning-Based Processing of Multispectral and RGB UAV Imagery for the Multitemporal Monitoring of Vineyard Water Status. Agronomy, 12(9), 2122-2122. https://doi.org/10.3390/agronomy12092122

Patrick, S. (2023). Unbalanced visual cues do not affect search precision at the nest in desert ants (Cataglyphis nodus). Learning & behavior. https://doi.org/10.3758/S13420-023-00613-0

Perry, A. . B. S. (2023). Book Review: Rituals for Virtual Meetings: Creative Ways to Engage People and Strengthen Relationships. Adult Learning, 34(4), 260-261. https://doi.org/10.1177/10451595221099569

Proffitt, R., Ma, M., & Skubic, M. (2023). Development and Testing of a Daily Activity Recognition System for Post-Stroke Rehabilitation. Sensors, 23(18). https://doi.org/10.3390/S23187872

Salihbegovic, F. (2020). The Encounter with the Real: What Can Complicite’s Theatre Performance 'The Encounter' Teach Us about the Future of VR Narratives? Body, Space & Technology, 19(1), 125-152. https://doi.org/10.16995/bst.336

Seda, A., Xiaolong, L., Qiyuan, W., Paige, M., YueHin, L., Jed, J., Joey, H., Laura, O., Narutoshi, H., & Axel, K. (2023). Virtual Planning and Patient-Specific Graft Design for Aortic Repairs. Cardiovascular engineering and technology. https://doi.org/10.1007/S13239-023-00701-2

Serena, S., Marco, A., Milena, M., & Marco, R. G. (2023). Experimental validation and uncertainty analysis of an innovative IoT infrared sensor for wall thermal transmittance measurement. Measurement Science and Technology, 34(12). https://doi.org/10.1088/1361-6501/ACF064

Shengcai, D., Le, W., Aiping, L., & Xun, C. (2023). Alignment-Enhanced Interactive Fusion Model for Complete and Incomplete Multimodal Hand Gesture Recognition. IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society, PP. https://doi.org/10.1109/TNSRE.2023.3335101

Shinji, S., Miho, A., Toru, S., Yuta, T., Jun, H., Taishi, W., Yuki, M., Sukchan, K., Saeri, K., Yuki, S., Kensuke, S., Yoshiaki, A., & Shigenori, K. (2023). Evaluation of electrical activity at the carpal tunnel area in response to median nerve elbow stimulation using magnetoneurography. Clinical Neurophysiology, 156, 89-97. https://doi.org/10.1016/j.clinph.2023.09.016

Shui’er, H., Sujin, K., & Hyun, J. J. (2023). The effect of visual rivalry in peripheral head-mounted displays on mobility. Scientific Reports, 13(1), 20199-20199. https://doi.org/10.1038/s41598-023-47427-8

Stanley, M., Krisha, G. V., Christine, C., Matteo, Z., Rezia, M., & Damien, C. (2022). Improving Haptic Response for Contextual Human Robot Interaction. Sensors, 22(5), 2040-2040. https://doi.org/10.3390/s22052040

Takeshi, K., Xiao, M., Santhosh, S., Thije, B., Gerard, d. H., Bart, P., Razvan, P., Daniel, T., Roy, V., B., A. H., Moreno, H. L., Florian, d. R., Itai, L., Fujito, Y., J., J. R. A., A., M. E., Jisk, K. A., & H., G. G. (2023). A touchless user interface based on a near-infrared-sensitive transparent optical imager. Nature Electronics, 6(6), 451-461. https://doi.org/10.1038/s41928-023-00970-8

Tamanna, N., & Waqar, N. (2022). A research protocol of an observational study on efficacy of microsoft kinect azure in evaluation of static posture in normal healthy population. Journal of Datta Meghe Institute of Medical Sciences University, 17(1), 30-33. https://doi.org/10.4103/jdmimsu.jdmimsu_176_21

Tanmay, D. S. (2023). Augmented reality applications and the future library. Library Hi Tech News, 40(9), 7-11.

Tarkov, M. S., & Chiglintsev, E. A. (2012). Reducing the dimensionality of the data in the problem of diagnosing thyroid disease. Optical memory & neural networks, 21(2), 119-125. https://doi.org/10.1108/LHTN-07-2023-0129

Tianchen, X., Xiaohua, R., Jiale, Y., Bin, S., & Enhua, W. (2023). Efficient Binocular Rendering of Volumetric Density Fields With Coupled Adaptive Cube-Map Ray Marching for Virtual Reality. IEEE transactions on visualization and computer graphics, PP. https://doi.org/10.1109/TVCG.2023.3322416

Un, L. S., Jinwook, K., & Jeongmi, L. (2023). Effects of Reward Schedule and Avatar Visibility on Joint Agency during VR Collaboration. IEEE transactions on visualization and computer graphics, PP. https://doi.org/10.1109/TVCG.2023.3320221

Viet, L., & Luca, C. (2022). Practical Approach to Digitally Simulate Nonsynoptic Wind Velocity Profiles and Its Implications on the Response of Monopole Towers. Journal of Structural Engineering, 148(1). https://doi.org/10.1061/(ASCE)ST.1943-541X.0003228

Wang, J. (2021). A Review of Deep Learning on Medical Image Analysis. Mobile Networks & Applications, 26(1), 351-380. https://doi.org/10.1007/s11036-020-01672-7

Wang, S. (2015). Pathological Brain Detection by a Novel Image Feature—Fractional Fourier Entropy. Entropy, 17(12), 8278-8296. https://www.mdpi.com/1099-4300/17/12/7877

Wang, S. (2021). Advances in data preprocessing for biomedical data fusion: an overview of the methods, challenges, and prospects. Information Fusion, 76, 376-421. https://doi.org/10.1016/j.inffus.2021.07.001

Wang, S.-H. (2021a). COVID-19 classification by CCSHNet with deep fusion using transfer learning and discriminant correlation analysis. Information Fusion, 68, 131-148. https://doi.org/10.1016/j.inffus.2020.11.005

Wang, S.-H. (2021b). Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network. Information Fusion, 67, 208-229. https://doi.org/10.1016/j.inffus.2020.10.004

William, E., Dominique, B., & Mohsen, Z. (2023). Will visual cues help alleviating motion sickness in automated cars? A review article. Ergonomics, 31-34.

Yongyi, L., Ghaffar, A., & Abdul, R. A. (2023). Advances in geothermal energy prospectivity mapping research based on machine learning in the age of big data. Sustainable Energy Technologies and Assessments, 60. https://doi.org/10.1016/J.SETA.2023.103550

Zhang, Y. (2015). Pathological brain detection in MRI scanning by wavelet packet Tsallis entropy and fuzzy support vector machine. SpringerPlus, 4(1), Article 716. http://www.springerplus.com/content/4/1/716

Zhang, Y. (2016). Smart detection on abnormal breasts in digital mammography based on contrast-limited adaptive histogram equalization and chaotic adaptive real-coded biogeography-based optimization. Simulation, 92(9), 873-885. https://doi.org/10.1177/0037549716667834

Zhang, Y. (2017). Pathological brain detection in MRI scanning via Hu moment invariants and machine learning. Journal of Experimental & Theoretical Artificial Intelligence, 29(2), 299-312. https://doi.org/10.1080/0952813X.2015.1132274

Zhang, Y.-D. (2016). Computer-aided diagnosis of abnormal breasts in mammogram images by weighted-type fractional Fourier transform [Article]. Advances in Mechanical Engineering, 8(2), Article 11. https://doi.org/10.1177/1687814016634243

Zhang, Y.-D. (2018). Twelve-layer deep convolutional neural network with stochastic pooling for tea category classification on GPU platform. Multimedia Tools and Applications, 77(17), 22821-22839. https://doi.org/10.1007/s11042-018-5765-3

Zhanming, C., Huawei, T., & Huiyue, W. (2023). User-Defined Foot Gestures for Eyes-Free Interaction in Smart Shower Rooms. International Journal of Human–Computer Interaction, 39(20), 4139-4161. https://doi.org/10.1080/10447318.2022.2109260

Downloads

Published

2023-12-13

How to Cite

Wu, M. (2023). Gesture Recognition in Virtual reality. Psychomachina, 1(1), 1–10. https://doi.org/10.59388/pm00336

Issue

Section

Articles