Taekwondo motion image recognition model based on hybrid neural network algorithm for wearable sensor of Internet of Things

Scientific Reports volume 13, Article number: 13097 (2023) Cite this article

287 Accesses

3 Altmetric

Metrics details

Conventional IoT wearable sensor Taekwondo motion image recognition model mainly uses Anchor fixed proportion whole body target anchor frame to extract recognition features, which is vulnerable to dynamic noise, resulting in low displacement recognition rate of motion image. Therefore, a new IoT wearable sensor Taekwondo motion image recognition model needs to be designed based on hybrid neural network algorithm. That is, the wearable sensor Taekwondo motion image features are extracted, and the hybrid neural network algorithm is used to generate the optimization model of the wearable sensor Taekwondo motion image recognition of the Internet of Things, so as to achieve effective recognition of Taekwondo motion images. The experimental results show that the designed wearable sensor of the Internet of Things based on the hybrid neural network algorithm has a high recognition rate of the motion image displacement of the Taekwondo motion image recognition model, which proves that the designed Taekwondo motion image recognition model has good recognition effect, reliability, and certain application value, and has made certain contributions to optimizing the Taekwondo movement.

Taekwondo is an official Olympic event1, which evolved from Hualangdao in North Korea, and later developed into a martial art popular in Asia for a long time. In the process of Taekwondo, athletes often use both hands and feet to fight effectively. Early Taekwondo and coaches mainly judged athletes’ movements with the naked eye2, which was easily affected by subjective factors, leading to inaccurate final evaluation results. In the context of informatization, the evaluation of Taekwondo competitions has also been gradually upgraded, and advanced processing tools such as computers are used for evaluation3. However, due to the influence of complex action features of Taekwondo, its moving image recognition is difficult, and it needs to be completed through an effective moving image recognition model.

Moving image recognition is an advanced computer perception technology, which can combine the interaction state between human and computer to complete the recognition, thus generating an effective moving image recognition model4. To improve the recognition effect of moving images, it is necessary to capture human behavior perception data and set reasonable recognition parameters5. At present, many scholars at home and abroad are studying the problem of motion recognition perception, and put forward a variety of motion image perception recognition assumptions. However, due to the lack of relevant experience6,7, the recognition effect of most existing motion image recognition models is general.

Early in the process of human motion image recognition, a special camera was mainly used. This camera can take effective moving image sequences to identify human motion8. At this time, the camera is also called perception camera.With the progress of computer vision technology, in order to obtain the human motion image data9,10 from all angles, more and more perceptual cameras are used, and the total number of motion image sequences taken by cameras is increasing, so the recognition effect is relatively improved. However, research shows that the recognition limitations of the above recognition methods are large, and they are vulnerable to light, perception camera location, occlusion and other factors, resulting in high unique recognition deviation11. In addition, their recognition privacy is relatively intrusive, and they are not suitable for use in some scenes. In order to solve the above problems, this paper constructs a new wearable sensor Taekwondo motion image recognition model based on the hybrid neural network algorithm.

Wearable sensors are sensor devices that can be worn on the body to collect data. Several types of wearable sensors that can be used for feature extraction include:

Accelerometers—Wearable accelerometers typically measure the acceleration and direction of body movement, which can be used to measure motion characteristics such as step count, walking speed, and activity intensity.

Gyroscopes—Gyroscopes can measure rotational movement of the body, such as bending, rolling, and spinning, and can be used to detect motion or posture changes.

Heart rate sensors—Heart rate sensors can measure changes in heart rate and the relationship between heart rate and other exercise or activity.

Temperature sensors—Temperature sensors can measure skin surface temperature changes, which can be used to detect changes in body temperature and other physiological features.

EMG sensors—Electromyography (EMG) sensors can measure the electrical signals generated when muscles contract, which can be used to detect muscle fatigue and activity level.

Body pressure sensors—Body pressure sensors can measure the pressure distribution of various parts of the body, which can be used to detect changes in body position and posture.

Optical sensors—Optical sensors can measure the reflection light intensity on the skin surface, which can be used to detect physiological features such as skin color, blood flow, and oxygenation.

The above sensors can be used to collect motion, physiological, and environmental data, and perform feature extraction. By processing and analyzing these data, various features such as human posture, movement behavior, and physiological conditions can be extracted and recognized. This article uses wearable sensors to extract Taekwondo motion image features.

Before moving image recognition, image pre-processing is required, which may be interfered by many factors during moving image transmission, resulting in recognition noise12. Therefore, image pre-processing is required before extracting moving image features to reduce noise. That is to use the average calculation method to divide a processing range13, screen effective change points, conduct noise removal processing, solve the problem of image blur, reduce image noise, and increase image clarity14,15. In addition, the computer can also be used to decompose the motion steps in the moving image, divide the motion details to achieve feature differentiation16, extract the comprehensive features of the image, and image feature differentiation \(y_{ij}\) as shown in (1) below.

In formula (1), \(W_{st}\), \(x_{i - s + 1}\) represents the coordinate element of the moving image, S and T represent the size of the recognition filter respectively. After feature extraction of moving image, because the dimension of feature image cannot be determined, it is necessary to select improved neural network parameters, carry out dimension reduction processing, connect sampling layers17, reduce the complexity of dimension reduction calculation, shorten the dimension reduction range, and improve the calculation accuracy. At this time, we can use mathematical methods to describe the Taekwondo moving image18, and use formula (2) to calculate the corresponding gray value of pixels \(V_{gray}\)

In formula (2), \(V_{red}\), \(V_{green}\), \(V_{blue}\) they represent different color brightness values of moving images, \(sa,sb,sc\) they represent the total content of pixels in the moving image19. At this time, the gray value of pixels in the moving image recognition can be used \(f(x_{m} ,y_{m} )\) in order to improve the effectiveness of extracted features20, it is necessary to calculate the original weighted average details of the moving image \(g(x,y)\), as shown in (3) below.

In formula (3), \(w(i,j)\) represents the filtered moving image, M represents the number of pixels in the neighborhood, and i and j represent the change threshold21 of the moving image. After the above steps are completed, the comprehensive features of the image can be extracted \(f(X)\), as shown in (4) below.

In formula (4), \(\alpha\) represents the iteration threshold set by the identification, \(g(x)\) represents the feature recognition index of the moving image. Using the above steps can improve the recognition accuracy of the moving image and reduce the impact of external interference on the final recognition result.

In order to solve the problem that the anchor frame of anchor fixed proportion whole body target is affected by dynamic noise when extracting recognition features, which leads to the low recognition rate of motion image displacement22,23, this paper generates an optimization model for Taekwondo motion image recognition of Internet of Things wearable sensor based on hybrid neural network algorithm. The method designed in this paper selects BP and LSTM hybrid neural networks for motion image recognition and judgment24, as shown in Fig. 1.

BP and LSTM hybrid neural network structure.

At this time, the standard mean value of each joint point measured by IMU needs to be calculated \(\sigma\), as shown in (5) below.

In formula (5), \(b\) represents the number of joint points, \(a_{1}\) represents the acceleration value of the joint point, \(a\) represents the average acceleration value, \(n\) representing the number of sequence frames, combined with the above standard mean25, we can judge the motion stillness category of the moving image. At this time, the generated moving image recognition process is shown in Fig. 2.

Moving image recognition process.

It can be seen from Fig. 2 that training samples can be input in combination with the above moving image recognition process, and the input value of hidden layer neurons26 can be calculated. At this time, the optimal model for Taekwondo moving image recognition of wearable sensors of the Internet of Things is built based on the hybrid neural network algorithm \(E(w)\) as shown in (6) below27.

In formula (6), \(y_{n}\) represents the hidden neuron connection weight, \(y\) represents the input neuron connection weight28. Using the above built IoT wearable sensor Taekwondo motion image recognition optimization model can effectively obtain the motion recognition weight, output effective image sequence recognition results29,30, and improve the reliability of motion image recognition.

In order to verify the recognition effect of the designed wearable sensor Taekwondo motion image recognition model based on the hybrid neural network algorithm, this paper built an experimental platform, and compared it with the conventional wearable sensor Taekwondo motion image recognition model, and carried out experiments, as follows.

Combined with the experimental requirements, this paper selects the Solid Works 3D virtual simulation platform as the experimental platform. The experimental platform is equipped with mainstream CAD analysis software, with good comprehensive performance. In the process of Taekwondo movement, the movement modes and basic movement angles of human joints are different for different movements. Therefore, the movement trend of human joints can be predicted according to the movement transformation trend of human joints to achieve coordinated motion control. Therefore, this experiment conducted motion estimation in combination with the MMG signal of human motion images, and generated the basic experimental process, as shown in Fig. 3.

Basic flow of experiment.

It can be seen from Fig. 3 that during the experiment, the MMG signal of the image can be continuously detected, and the action simulation can be carried out. With the relevant motion angles as a reference, effective control algorithms can be used to correct, so as to collect Taekwondo motion images that meet the experimental requirements, as shown in Fig. 4.

Taekwondo motion images.

At this time, the joint angle discretization parameters under different motion modes are shown in Table 1.

It can be seen from Table 1 that moving image recognition data can be selected in combination with the above joint discretization parameters. In order to ensure the reliability of recognition, image data needs to be preprocessed, that is, select the PC association platform, adjust the main frequency of moving image processing to 2.5 GHz, and then the corresponding internal memory is 5 GB. Input the above obtained joint angle discretization parameters, calculate the discrete state frequency of different signal segments. After the above steps are completed, the joint angle basic signal graph can be generated to classify the experimental data. At this time, the segment slope and the number of association rules of each signal segment are shown in Table 2.

It can be seen from Table 1 that after the above experimental parameters are set, the experimental hardware can be connected. This paper selects the RFID Internet of Things wearable sensor as the experimental sensor. At this time, the schematic diagram of the experimental hardware device connection is shown in Fig. 5.

Connection diagram of experimental hardware device.

It can be seen from Fig. 5 that the core of the above experimental hardware is Impinj R2000 RFID reader writer, which is configured with a power supply of 9000 mA and several identification antennas. The transmission power of the identification antenna is 0–30 dbm, and the accuracy can be adjusted. During the experiment, it is necessary to ensure that the reading and writing frequency band of the core reader writer is within the specified range. In order to improve the effectiveness of recognition results, this experiment selects US FCC 47 CFG as the support, and sets the ETSI EN 302,208 image recognition standard. During the experiment, it is necessary to always ensure that the reader is in the Inventory working mode, and try to improve the read–write range of the reader. After the above hardware devices are connected, the experimental indicators can be selected and the recognition rate formula after moving image displacement can be designed \(D\), as shown in (7) below.

In formula (7), \(W\) represents the position of the base image after displacement, \(W_{0}\) represents the position of the identified image before displacement, \(R\) image preset displacement, the higher the recognition rate after moving image displacement, the better the recognition effect of the moving image recognition model. On the contrary, the lower the recognition rate after moving image displacement, the poorer the recognition effect of the moving image recognition model. Before the experiment process, it is also necessary to connect the MEMS inertial sensor to ensure that it meets the requirements of actual moving image recognition. The specifications and parameters of the sensor are shown in Table 3.

Table 3 shows that the parameters of the above inertial sensors meet the experimental requirements. In addition to the above preparations, it is also necessary to set the relevant parameters of the hybrid neural network and prepare the experimental data set.

The hybrid neural network is a model that combines convolutional neural networks and fully connected neural networks. In the process of image recognition, the following parameters are usually set for the hybrid neural network:

Input layer size this refers to the size of the input image, which usually has three dimensions of length, width, and channel number, such as 224 × 224 × 3.

Convolutional layer parameters these include specifying the size of the convolution kernel, the number of convolution kernels, the step size, and the padding mode. These parameters determine the size of the output of the convolutional layer and the number of feature maps.

Pooling layer parameters these include specifying the size of the pooling kernel and the step size. These parameters determine the size of the output of the pooling layer and the number of feature maps.

Fully connected layer parameters these include specifying the number of neurons in the fully connected layer and activation function.

In the recognition process of the hybrid neural network, the first step is to input an image, which then undergoes a series of processing by convolutional layers and pooling layers. The feature maps are continuously reduced in size while extracting different features of the image. Multiple feature maps are then merged into a single vector, and classification is performed through the fully connected layer, ultimately resulting in the classification result of the image.

The Taekwondo motion image recognition data set used in this article is as follows:

KTH-TIPS2-B This data set contains 384 images of 6 different Taekwondo movements (front kick, back kick, turn kick, side kick, high side kick and low side kick). Each movement was performed by two different actors, and each actor performed it 16 times.

KyungHee TaeKwonDo Dataset This data set contains 1010 images of 10 different Taekwondo movements (push hands, wrist palm strike, front kick, knee kick, backward kick, back kick, side kick, single leg jump kick, consecutive kick and bottom kick). Each movement was performed by different actors.

NTU RGB + D Dataset This data set contains 56 different actions, including 11 different Taekwondo movements. Each movement was performed by 40 different actors, and was captured under RGB and depth sensors.

After all experimental preparation devices have been connected, subsequent motion image recognition experiments can be conducted.

In combination with the above experimental preparations, we can carry out subsequent experiments on the wearable sensor of the Internet of Things for Taekwondo motion image recognition. That is, in the built experimental platform, the wearable sensor Taekwondo motion image recognition model based on the hybrid neural network algorithm designed in this paper and the conventional wearable sensor Taekwondo motion image recognition model are respectively used for motion image recognition, and the public formula (1) is used to record the motion image displacement recognition rate of the two methods in different motion modes. The experimental results are shown in Table 4.

Table 4 shows that the displacement recognition rate of the motion image of the wearable sensor Taekwondo motion image recognition model designed in this paper based on the hybrid neural network algorithm is high in different motion modes, while the displacement recognition rate of the motion image of the conventional wearable sensor Taekwondo motion image recognition model is relatively low. It proves that the wearable sensor Taekwondo motion image recognition model designed in this paper has good recognition performance, effectiveness and certain application value.

By comparing the contrast, clarity, and recognition time of the denoised images using other algorithms, the advantages of the proposed algorithm in identifying image features were verified.

Contrast refers to the measurement of the different brightness levels between the brightest white and darkest black areas in an image. The greater the difference range, the greater the contrast, and vice versa. This value has no standard definition, therefore, in this study, it is set to 35 based on human comfort level.

Clarity is the average gradient of the image, which can sensitively reflect the ability of the image to express small contrasts. Information entropy represents the size of the information contained in the image. With the increase of the information entropy value, the information value contained in the feature image also increases, resulting in higher clarity.

Recognition time reflects the recognition efficiency of each model, and this value is recorded by the computer.

The enhancement effect comparison results between the method proposed in this paper and traditional methods are shown in Table 5.

From Table 5, it can be concluded that the larger the average value during image recognition, the better the ability to improve image brightness, and the accuracy of the proposed method is 95.39%, the highest among compared algorithms. This indicates that the proposed algorithm has a better ability to enhance image brightness, ensuring higher contrast and information entropy values. This shows that while the image recognition effect is good, it also contains more information. In terms of algorithm running time, the proposed algorithm also has the shortest time. The experimental results above prove that the proposed algorithm has obvious advantages in identifying image features.

Taekwondo is a common sport, which has many trainers in various countries. The motion pattern of Taekwondo is complex and difficult to identify effectively, so it needs to be comprehensively evaluated through the sensor photography. In recent years, computer vision technology has developed more and more rapidly in China, and some researchers have applied it to the recognition of Taekwondo motion images. The conventional recognition model of Taekwondo motion image has poor recognition effect and does not meet the current recognition requirements. Therefore, this paper designs a new wearable sensor Taekwondo motion image recognition model based on the hybrid neural network algorithm. The experimental results show that the wearable sensor Taekwondo motion image recognition model has good recognition performance, reliability and certain application value, and has made certain contributions to optimizing Taekwondo sports skills.

At present, hybrid neural network algorithms have achieved certain results, but there is still room for improvement. Future research will use many state-of-the-art deep learning models such as Alexnet, Googlenet, and Xception to classify images and improve algorithm performance.

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Hung, M. S. & Chiu, Y. T. Application of image recognition for plant virus detection. Microsc. Microanal. 27(S1), 2274–2276 (2021).

Article Google Scholar

Toldinas, J. et al. A novel approach for network intrusion detection using multistage deep learning image recognition. Electronics 10(15), 1854 (2021).

Article Google Scholar

Soriano, R. P. et al. New image recognition technique for intuitive understanding in class of the dynamic response of high-rise buildings. Sustainability 13(7), 3695 (2021).

Article Google Scholar

Yong, K. C. et al. Double-sided infrared display using an opaque substrate based on infrared image recognition mechanism. AIP Adv. 11(2), 025136 (2021).

Article ADS Google Scholar

Wang, C. et al. A field-programmable gate array system for sonar image recognition based on convolutional neural network: Proceedings of the Institution of Mechanical Engineers, Part I. J. Syst. Control Eng. 235(10), 1808–1818 (2021).

Google Scholar

Lin, W., Adetomi, A. & Arslan, T. Low-power ultra-small edge AI accelerators for image recognition with convolution neural networks: Analysis and future directions. Electronics 10(17), 2048 (2021).

Article Google Scholar

Deng, X. et al. Interpretable multi-modal image registration network based on disentangled convolutional sparse coding. IEEE Trans. Image Process. 32, 1078–1091 (2023).

Article ADS Google Scholar

Sun, X. et al. A method of mining truck loading volume detection based on deep learning and image recognition. Sensors 21(2), 635 (2021).

Article ADS PubMed PubMed Central Google Scholar

Chen, Y. & Qu, R. Study on infringement identification of art works based on CNN image recognition technology. J. Phys. Conf. Ser. 1802(3), 032084 (2021).

Article Google Scholar

Zhang, J. et al. ROSEFusion: Random optimization for online dense reconstruction under fast camera motion. ACM Trans. Graph. 40(4), 1–17 (2021).

Google Scholar

Bai, H., Zhang, H. & Wang, Q. Dual discriminative auto-encoder network for zero shot image recognition. J. Intell. Fuzzy Syst. 40(12), 1–12 (2021).

Google Scholar

Guo, W. Intelligent detection device of pavement disease based on image recognition technology. J. Phys. Conf. Ser. 1884(1), 012032 (2021).

Article Google Scholar

Xin, M. & Wang, Y. Image recognition of crop diseases and insect pests based on deep learning. Wirel. Commun. Mob. Comput. 2021(10), 1–15 (2021).

Article Google Scholar

Chen, Y. Research on convolutional neural network image recognition algorithm based on computer big data. J. Phys. Conf. Ser. 1744(2), 022096 (2021).

Article Google Scholar

Wang, Y. et al. High-order interaction learning for image captioning. IEEE Trans. Circuits Syst. Video Technol. 32(7), 4417–4430 (2022).

Article Google Scholar

Chen, Y. Research on image recognition of convolutional neural network under different computer data set capacities. J. Phys. Conf. Ser. 1744(4), 042091 (2021).

Article Google Scholar

Tian, L., Xu, H. & Zheng, X. Research on fingerprint image recognition based on convolution neural network. Int. J. Biometr. 13(1), 64 (2021).

Article Google Scholar

Sun, C. et al. Image recognition technology in texture identification of marine sediment sonar image. Complexity 2021(2), 1–8 (2021).

ADS Google Scholar

Daradkeh, Y. I. et al. Development of effective methods for structural image recognition using the principles of data granulation and apparatus of fuzzy logic. IEEE Access 9(99), 13417–13428 (2021).

Article Google Scholar

Hu, J. et al. Design of an intelligent vibration screening system for armyworm pupae based on image recognition. Comput. Electron. Agric. 187(6), 106189 (2021).

Article Google Scholar

Wang, Y. et al. Image recognition model based on deep learning for remaining oil recognition from visualization experiment. Fuel 291(3), 120216 (2021).

Article CAS Google Scholar

Zhang, Y. et al. Robust text image recognition via adversarial sequence-to-sequence domain adaptation. IEEE Trans. Image Process. 99, 1 (2021).

ADS CAS Google Scholar

Xiong, S., Li, B. & Zhu, S. DCGNN: A single-stage 3D object detection network based on density clustering and graph neural network. Complex Intell. Syst. https://doi.org/10.1007/s40747-022-00926-z (2022).

Article Google Scholar

Andriyanov, N. A., Dementiev, V. E. & Kargashin, Y. D. Analysis of the impact of visual attacks on the characteristics of neural networks in image recognition. Procedia Comput. Sci. 186(12), 495–502 (2021).

Article Google Scholar

Cao, P. & Zhang, S. Research on image recognition of Wushu action based on remote sensing image and embedded system. Microprocess. Microsyst. 82(3), 103841 (2021).

Article Google Scholar

Liu, H. et al. Learning scale-consistent attention part network for fine-grained image recognition. IEEE Trans. Multimedia 99, 1 (2021).

Google Scholar

Dan, H. C., Bai, G. W. & Zhu, Z. H. Application of deep learning-based image recognition technology to asphalt–aggregate mixtures: Methodology. Constr. Build. Mater. 297(12), 123770 (2021).

Article Google Scholar

Liu, G., Zhao, L. & Fang, X. PDA: Proxy-based domain adaptation for few-shot image recognition. Image Vis. Comput. 110(3), 104164 (2021).

Article Google Scholar

Guo, X. & Chen, J. Research on the construction of smart library simulation Platform system. Comput. Simul. 39(6), 5 (2022).

Google Scholar

Cheng, D. et al. Light-guided and cross-fusion U-Net for anti-illumination image super-resolution. IEEE Trans. Circuits Syst. Video Technol. 32(12), 8436–8449 (2022).

Article Google Scholar

Download references

Physical Education Institute, Yongin University, Yongin, 17092, South Korea

Xiaotong Lu

You can also search for this author in PubMed Google Scholar

The author confirms being the sole contributor of this work and has approved it for publication.

Correspondence to Xiaotong Lu.

The author declares no competing interests.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

Lu, X. Taekwondo motion image recognition model based on hybrid neural network algorithm for wearable sensor of Internet of Things. Sci Rep 13, 13097 (2023). https://doi.org/10.1038/s41598-023-40169-7

Download citation

Received: 20 April 2023

Accepted: 06 August 2023

Published: 11 August 2023

DOI: https://doi.org/10.1038/s41598-023-40169-7

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.