JS library for real-time monitoring hand movement can realize many interesting functions

Time:2021-9-15

[introduction]:Handtrack.jsIt is a prototype library that can directly realize real-time hand motion tracking and detection in the browser. It is an open-source model trained by tensorflow and does not need users’ own training. With it, you only need a few lines of code to detect the hand movement in the picture.

GitHub home page

https://github.com/victordibi…

JS library for real-time monitoring hand movement can realize many interesting functions

1. Introduction

Handtrack.js, based onTensorFlowThe object detection API training model is built to realize real-time monitoring of hand movement through the camera. Its characteristics mainly include:

  • It allows developers to quickly create gesture interaction prototypes using trained hand detection models;
  • By providing users with useful functions, it allows users to detect the hand in the image without any ml experience, and does not need to train the model themselves;
  • Users do not need to export any diagrams or saved models, and can directly reference them in web applications handtrack.jsThe library then invokes the method it provides.

JS library for real-time monitoring hand movement can realize many interesting functions

2. Application scenario

If you are interested in gesture based interactive experience,Handtrack.jsWould be very useful. Users can immediately get a gesture based interactive experience without using any additional sensors or hardware.

JS library for real-time monitoring hand movement can realize many interesting functions

Some related application scenarios:

  • Mapping mouse movement to hand movement to achieve the purpose of control;
  • When the hand overlaps with other objects, it can represent some meaningful interactive signals (such as touching or selecting objects);
  • Human hand motion can be used as a scene of some activity recognition agents (for example, automatically tracking the actions of chess players in video or image), or simply calculating how many people are in the image or video frame;
  • Create demos that anyone can easily run or experience with minimal setup.

JS library for real-time monitoring hand movement can realize many interesting functions
JS library for real-time monitoring hand movement can realize many interesting functions

3. Method of use

You can go directly toscriptInclude the URL address of the library in the tag, or use the build tool to import it from NPM.

3.1 using script tags

Handtrack.jsThe minimized JS file is currently hosted on jsdelivr, a free open source CDN that allows you to include any NPM package in your web application.

<script> </script>

<img id="img"/> 
<canvas id="canvas" class="border"></canvas>

Put the abovescriptAdd label tohtmlPage, you can usehandTrackVariable referencehandtrack.js, as follows:

<script>
  const img = document.getElementById('img'); 
  const canvas = document.getElementById('canvas');
  const context = canvas.getContext('2d');
  
  // Load the model.
  handTrack.load().then(model => {
    model.detect(img).then(predictions => {
      console.log('Predictions: ', predictions); 
    });
  });
</script> 

The code snippet above will print out throughimgLabel the prediction frame of the incoming image. If you change the video or submit the image frame through the camera, you can “track” the hand that appears in each frame.

3.2 using NPM

You can install handtrack.js as an NPM package using the following command:

npm install --save handtrackjs

You can then import and use its examples in your web application:

import * as handTrack from 'handtrackjs';

const img = document.getElementById('img');

// Load the model.
handTrack.load().then(model => {
  // detect objects in the image.
  console.log("model loaded")
  model.detect(img).then(predictions => {
    console.log('Predictions: ', predictions); 
  });
});

3.3 API of handtrack.js

Handtrack.jsTwo main methods are provided,load()Methods anddetect() Methods are used to load the hand detection model and obtain the prediction results.

load()Method: receive an optional model parameter and return a model object. The user is allowed to control the performance of the model through the optional model parameter:

const modelParams = {
  flipHorizontal: true,   // flip e.g for video 
  imageScaleFactor: 0.7,  // reduce input image size for gains in speed.
  maxNumBoxes: 20,        // maximum number of boxes to detect
  iouThreshold: 0.5,      // ioU threshold for non-max suppression
  scoreThreshold: 0.79,    // confidence threshold for predictions.
}

handTrack.load(modelParams).then(model => {

});

detect() method: receive an input source parameter (which can be an img, video or canvas object) and return the frame prediction result of the hand position in the image:

An array of borders with class names and confidence.

model.detect(img).then(predictions => { 
        
});

The format of prediction results is as follows:

[{
  bbox: [x, y, width, height],
  class: "hand",
  score: 0.8380282521247864
}, {
  bbox: [x, y, width, height],
  class: "hand",
  score: 0.74644153267145157
}]

Handtrack.js also provides other methods:

  • model.getFPS(): get FPS, i.e. detection times per second;
  • model.renderPredictions(predictions, canvas, context, mediasource): draws a border (and source image) on the specified canvas. amongpredictionsyes detect()Method.canvasYes, renderingpredictionsofhtml canvasObject,contextIs a canvas 2D context object,mediasourceYespredictionsReference to the input frame (IMG, video, canvas, etc.) used in (render it first and draw a border on it).
  • model.getModelParameters(): return model parameters;
  • model.setModelParameters(modelParams): update model parameters;
  • dispose(): delete model instance;
  • startVideo(video): starts a camera video stream on a given video element. Return apromise, which can be used to verify whether the user has provided video permission;
  • stopVideo(video): stop video streaming;

4. Next step

  • The high computational cost is mainly due to the need for neural network operation in predicting the boundary box, which is a point that needs to be improved and optimized in the follow-up;
  • Cross frame tracking ID: assign an ID to each object when it enters a frame and keep tracking;
  • Add some discrete poses: for example, not just hands, but detecting open hands, fists).

5. References

Open source outpostShare popular, interesting and practical open source projects on a daily basis. Participate in maintaining the open source technology resource library of 100000 + star, including python, Java, C / C + +, go, JS, CSS, node.js, PHP,. Net, etc.

Recommended Today

Beautify your code VB (VBS) code formatting implementation code

However, vb.net does have many new functions that VB6 does not have. The automatic typesetting of code is one, which is the function we want to realize today – VB code formatting.Let’s look at the effect: Before formatting: Copy codeThe code is as follows: For i = 0 To WebBrowser1.Document.All.length – 1 If WebBrowser1.Document.All(i).tagName = […]