Multi screen interaction – H5 intermediate advanced



With the popularity of smart hardware, mobile phones, tablets, PCs and even roadside electronic billboards, modern browsers are everywhere. It’s not enough to weave our own world in the browser, but H5 gives the browser too many new features, waiting for us to use. This article introduces how to use the compass API of mobile browser to draw a 3D box model in real time in PC browser.

This cool way of playing is called “multi screen interaction”. It’s like using a mobile phone as a game controller and a PC monitor as a TV, but it’s all implemented in a browser.

First on the renderings
Multi screen interaction - H5 intermediate advanced

(the tester is the HTC thunderbolt 2 + Chrome browser with the Xiaomi system and the screen cracked.)

Please stamp the source code here:…

Local test process:

  1. On the PC, use the command node index.js to automatically open the project home page. (please shut down adsafe, if there is a virtual machine, please disable the virtual network card)

  2. Create a room and automatically enter room.

  3. Scan the QR code anywhere in the “room” with your mobile phone.

  4. Make sure the phone and PC can ping each other

Adsafe is a good advertising software, but it will block the IP access of this computer, which may cause the front page of the project to be closed, so please close it temporarily
This program will automatically obtain the local IP address. If there is a virtual network card, the IP address may not be obtained correctly

Client (browser)

1. Mobile browser

The rotational position of an object in space can be expressed by a direction vector (x, y, z) and a rotation angle (angle). CSS3transformOfrotate3d(x,y,z,angle)Four arguments to this function.

In order to draw the rotation of a stereo model conveniently in the browser, the key point is to use the H5 new feature of mobile browser to obtain the data of mobile rotation state, and then convert it into these four parameters.

1.1 gravity induction API

devicemotionEquipment movement as the name suggests
In fact, there are not only gravity induction data, but also moving acceleration and swing angle.
But this interface tends toMoment in motionThe data show that, in addition to the acceleration of gravity, other data (moving acceleration, swing angle) are basically 0.

window.addEventListener('devicemotion', deviceMotionHandler, true);
function deviceMotionHandler(evt){
       document.body.innerHTML = 
          "X-axis acceleration:" + EVT. Accelerationincludinggravity. X + "< br >
        +"Y-axis acceleration:" + EVT. Accelerationincludinggravity. Y + "< br >
        +"Z-axis acceleration:" + EVT. Accelerationincludinggravity. Z + "< br >
    if(evt.rotationRate ){
       document.body.innerHTML +=  
         "X-axis torsion:" + EVT. Rotationrate. Beta + "< br > 
       +"Y-axis twist:" + EVT. Rotatingrate. Gamma + "< br >
       +"Z-axis twist:" + EVT. Rotatingrate. Alpha + "< br >

(for the old model of Meizu, the built-in browser of Android 4.4.4 does not fully support this API, please install QQ browser separately.)
Run the above code in the mobile browser, and shake it slightly, you will see the printing data jump.
Get the data, then start to observe the law.
The screen of the mobile phone is facing up and placed horizontally. The acceleration of z-axis gravity is 9.8, y, X is 0.
The screen of the mobile phone is facing down, placed horizontally and still, the acceleration of z-axis gravity is – 9.8, y, X is 0.
The microphone of the mobile phone is placed vertically and still, with the acceleration of Y gravity of 9.8, X and Z of 0.
The microphone of the mobile phone is placed vertically and still, with the acceleration of Y gravity of – 9.8, X and Z of 0.
The right side of the mobile phone is facing up, and it is placed vertically and still. The acceleration of X gravity is 9.8, y and Z are 0.
The left side of the mobile phone is facing down, and it is placed vertically and still. The acceleration of X gravity is – 9.8, y, Z is 0.

The spatial coordinates of the mobile phone are as follows:
Multi screen interaction - H5 intermediate advanced
The arrows point in the positive direction of the coordinates.

When the mobile phone starts to tilt, the acceleration components of X, y and Z axes have values, and the absolute values are less than 9.8. According to the value of the component, the tilt state of the mobile phone in three-dimensional space can be calculated, but the calculation process is complex, and when the mobile phone moves, the value of the acceleration of gravity does not accurately express the current tilt. Generally, this data is not used to calculate the tilt of mobile phones in three-dimensional space.

When the mobile phone is placed horizontally, dial the mobile phone to rotate slowly, and the data of gravity acceleration does not change.
So,The gravity sensing API can only obtain the current tilt state of the device, but not the rotation direction of the device。 Some simple functions, such as shaking, shaking, can be implemented with this interface.

Using gravity induction API, you can easily use high school mathematicsAnti trigonometric function, to realize the rotation of XY 2D plane, the effect is as follows:

Multi screen interaction - H5 intermediate advanced

The code is as follows:

function deviceMotionHandler(evt){
    var angle = 
            0 - evt.accelerationIncludingGravity.x ,        
        ).toFixed(2) / Math.PI * 180 ; 

This angle can be directly applied to the CSS properties of DOMtransform:rotate(angle deg)Up.

1.2 compass API

window.addEventListener('deviceorientation', deviceOrientationHandler, true);
function deviceMotionHandler(evt){
     document.body.innerHTML = 
          "Z-axis rotation (compass direction) alpha:" + event. Alpha + "< br >"
        +"Y-axis rotation Gamma:" + event. Gamma + "< br >"
        +"X-axis rotation beta:" + event.beta


Here comes the point,deviceorientationIt can well represent the state of the object in space, the direction of rotation, the angle of inclination, whether it is still or moving or accelerating.

Here we have to go withdevicemotionOfevt.rotationRateTo make a distinction, although there are alpha, gamma, betadevicemotionIt describes the angle value changed by rotation. Only when the object’s angle changes can there be data. When it is still, it will become 0deviceorientationDescribes the angle value at rest.

The units of these three values are deg. how to convert them to CSS3transform:rotate3d(x,y,z,angle)The four parameters of are troublesome for front-end dogs without any 3D knowledge.

Now let’s introduce a concept:Quaternion

The quaternion is a high-order complex q = [w, x, y, Z].
The basic mathematical equation of quaternion is:
q = cos (a/2) + i(x sin(a/2)) + j(y Sin (A / 2)) + K (Z * sin (A / 2)), where a is the rotation angle and X, y, Z is the rotation axis.
Quaternions represent a complete rotation.
Quaternions can be obtained from the rotation angles (alpha, beta, gamma) of each axis.
Quaternions convert the axis of rotation (x, y, z) and the angle of rotation (angle).

As a preliminary test, this paper does not discuss the specific definition of Quaternion in depth. The difficulty is to obtain quaternion [w, x, y, Z].
Fortunately, the official method to convert the rotation angle (alpha, beta, gamma) into quaternion is provided…
Search within this pagegetQuaternion

In addition, I wrote a quaternion function getacquaternion according to the mathematical formula
The code is as follows:

var degtorad = Math.PI / 180;
Function getquaternion (alpha, beta, gamma) {// official quaternion method

  var _x = beta  ? beta  * degtorad : 0; // beta value
  var _y = gamma ? gamma * degtorad : 0; // gamma value
  var _z = alpha ? alpha * degtorad : 0; // alpha value

  var cX = Math.cos( _x/2 );
  var cY = Math.cos( _y/2 );
  var cZ = Math.cos( _z/2 );
  var sX = Math.sin( _x/2 );
  var sY = Math.sin( _y/2 );
  var sZ = Math.sin( _z/2 );

  var w = cX * cY * cZ - sX * sY * sZ;
  var x = sX * cY * cZ - cX * sY * sZ;
  var y = cX * sY * cZ + sX * cY * sZ;
  var z = cX * cY * sZ + sX * sY * cZ;

  return [ w, x, y, z ];


Function getaquaternion (_w, _x, _y, _z) {// my quaternion rotation axis and rotation angle method

  var rotate = 2 * Math.acos(_w)/degtorad ;

  var x = _x / Math.sin(degtorad * rotate/2) || 0;
  var y = _y / Math.sin(degtorad * rotate/2) || 0;
  var z = _z / Math.sin(degtorad * rotate/2) || 0;

  return {x:x,y:y,z:z,rotate:rotate};


Function devicemotionhandler (EVT) {// deviceorientation event handler
  var qu = getQuaternion(evt.alpha,evt.beta,evt.gamma);
  var rotate3d = getAcQuaternion(qu[0],qu[1],qu[2],qu[3]);
  //Rotate3d's parameters are already available. You can handle them. I sent it to the server, gave it to the PC, and displayed the rotation on the PC

1.3 calibration

Here’s a 3D concept, camera location. Our PC monitor is a camera. Can only passively show the scene from a certain angle. Normally, the plane of the mobile phone should be parallel to the plane of the display and perpendicular to the ground plane. For example, the camera is facing the front of the mobile phone.
If the mobile phone is not perpendicular to the ground plane when it is calibrated, the position of the camera is not necessarily in the front. At this time, the display is not horizontal synchronous.
As shown in the figure below, when calibrating, the phone screen is facing up. At this time, the camera is on the ceiling, and the image you see is the top view.

Multi screen interaction - H5 intermediate advanced

In the same way, when calibrating, the screen of the mobile phone is facing down. At this time, the camera is on the ground, shooting upward, and the image you see is the elevation view.

In summary:When calibrating, the camera will shoot the screen where the mobile screen is facing, and it will not move.

1.4 compatibility

The compatibility test of demo is not ideal
Well tested and fluent on IOS platform.

On the Android platform, except for the Chrome browser, there will be various problems, mainly manifested in the inaccurate compass data.
Chrome doesn’t have a sweep function, because it’s not popular abroad. So it’s very painful on Android platform. I need to install another one to check, so I can have a complete experience.
(if there is a problem of inaccurate rotation, you can try to calibrate the compass. It’s about drawing 8 with your mobile phone. There are many ways for Baidu)

If the code has compatible writing method or other compatible problems, please let me know. You can email me (overtree) on coding. Thank you very much.

2. PC browser

The function of PC browser is to display room information and create rooms.

Display room, creation time, number of participants, click enter.
Create a room, and enter the room automatically after success.

In the room, receive the message from the mobile terminal forwarded by the server and make corresponding actions, including online, calibration, rotation and offline.

When online, arrange to sit (hide QR code, display model)
During calibration, reset the display angle of the model.
When you rotate, you rotate.
Display QR code again when offline (display QR code, hide model)

2.1 initialization, establishing WS connection

It’s about the things in the room. So here’s just what happens when you enter the room.
First, the room parameters should be correct, at least with room number.

Room Routing:
Roomnumber is a 16 bit random string.
Seat Routing:

var uri = win.location.pathname.split('/'),roomNumber;

function initUrlData(){
  if(uri.length>=3 && uri[1] == "room"){
    roomNumber = uri[2];
    Document.title = "virtual room" + roomnumber + "number"
    return 1;
    window.location.href = "/index";
    return 0;

function initWebSocket(){
   Var wsuri = "WS: //" + window. Location. Hostname + ": <% = config. Wsport% >" + "/ WS / room"; // a placeholder for EJS is used here, so that the correct port can be used in time when the server changes the websocket port.
   var websocket = new WebSocket(wsUri); 
   websocket.onopen = function(evt) { 
   }; // after the link is established, send a message indicating which room it is in

   websocket.onclose = function(evt) { 


   websocket.onmessage = function(evt) { 
       Parsemessage (EVT. Data) // parse data
   websocket.onerror = function(evt) { 

   //After binding these processing functions, websocket starts to establish links instead of new ones

$(".room-place .qrcode").each(function(index,item){
        "size": 200,
        "color": "#3a3",
        "text": window.location.origin + "/room/" + roomNumber + "/" + (index+1)
    //Here we use jQuery plug-in, jQuery QRcode to initialize QR code according to seat route

2.2 pure CSS3 stereo model

As an ordinary front-end person, if you want to draw a 3D model, the most familiar way is to use CSS3.
(skip this section if you are using three.js)
However, if you want to draw a hexahedron quickly, you need to think about it. After all, this skill is rarely used.

Draw a box

<section class="container">
    <div id="box" >
      < figure class = "front" >
      < figure class = "back" >
      < figure class = "right" > < span > right < / span > < figure >
      < figure class = "left" > < span > left < / span > < figure >
      < figure class = "top" > < span > Top < / span > < figure >
      < figure class = "bottom" > < span > bottom < / span > < figure >
        Margin: 0; / * no increase will skew*/
    .container {
      width: 300px;
      height: 200px;
      position: relative;
      Perspective: 1200px; / * camera distance. If the setting is small, the cube display will be deformed*/
    #box figure {
      display: block;
      position: absolute;
      border: 2px solid black;
      line-height: 200px;
      font-size: 40px;
      text-align: center;
      font-weight: bold;
      color: white;
      Box sizing: border box; / * because there is a 2px wide border, if it is not set to this value, then the width and height of each face must be set less than 4 pixels to align*/
    #box {
      width: 100%;
      height: 100%;
      position: absolute;
      Transform style: preserve-3d; / * this is very important. The default is flat*/

    #box .front,
    #box .back {
      width: 300px;   
      height: 200px;
    #box .right,
    #box .left {
      width: 100px;
      height: 200px;
      Left: 100px; / * adjustment*/
    #box .top,
    #box .bottom {
      width: 300px;
      height: 100px;
      Top: 50px; / * adjustment*/

     /*Give each face a translucent color*/
     #box .front  { background: hsla( 000, 100%, 50%, 0.7 ); }
     #box .back   { background: hsla( 160, 100%, 50%, 0.7 ); }
     #box .right  { background: hsla( 120, 100%, 50%, 0.7 ); }
     #box .left   { background: hsla( 180, 100%, 50%, 0.7 ); }
     #box .top    { background: hsla( 240, 100%, 50%, 0.7 ); }
     #box .bottom { background: hsla( 300, 100%, 50%, 0.7 ); }

     #Box. Front {/ * the distance multiplied by 2 is the distance from the front to the back*/
         transform: translateZ( 50px );
     #Box. Back {/ * front face rotates 180 degrees along the X axis to do the back*/
         transform: rotateX( -180deg ) translateZ(  50px );
     #Box. Right {/ * the distance multiplied by 2 is the distance between the left and right sides*/
         transform: rotateY(   90deg ) translateZ( 150px );
     #Box. Left {/ * front face rotates 90 degrees along the Y axis to make the side*/
         transform: rotateY(  -90deg ) translateZ( 150px );
     #Box. Top {/ * the distance multiplied by 2 is the height of the box*/
         transform: rotateX(   90deg ) translateZ( 100px );
     #Box. Bottom {/ * front face rotates 90 degrees along the X axis to make the bottom face*/
         transform: rotateX(  -90deg ) translateZ( 100px );

What do you want to make complaints about such CSS?

Such a stylesheet is a time of slash and burn

If you use sass, you only need to write a box and multiple nesting once.

The effect is as follows:
Multi screen interaction - H5 intermediate advanced

If we use webgl to draw, import some ready-made 3D models, whether objects or people, can play with the palm 360 degrees without dead angle.
(if there is a model of Mr. Cang, I’m still a little excited. The VR feeling comes from –)

The next step is to wait for the rotation information from the mobile terminal, x, y, Z, angle, so that the box can rotate.

+(- parsefloat (content. X)) + "," // negate
+ (+parseFloat(content.y))+","
+(- parsefloat (content. Z)) + "," // negate
+ content.rotate +"deg)");

If it is not reversed, the rotation is wrong. I have tried many times to negate different coordinates, and finally came to the conclusion that this negate method is the only one that shows normal combination.

I can’t understand these two inversions. I guess it’s because the coordinates of X, y, Z in CSS and the coordinates of X, y, Z in physical equipment are different. After all, the display is flat, and its definition of X, y, Z cannot be consistent with that of the cell phone sensor.

2.3 calibration

The PC calibration is much easier. Put a layer of div.adjust on the box.
When receiving the calibration information x, y, Z, angle from the mobile terminal, set the rotation of div.adjust of the jacket to x, y, Z, – angle.

+ (-parseFloat(content.x))+","  
+ (+parseFloat(content.y))+","
+ (-parseFloat(content.z))+","  
+(- parsefloat (content. Rotate)) + "DEG"); // negate

Of course, this adjust style contains at least the following styles

  position: absolute;

2.4 compatibility

PC compatibility is much better, as long as the modern H5 browser basically has no compatibility problem.


1. Data structure

This service only saves temporary data and forwards messages.
Temporary data: for example, the websocket connection handle and room information at each end. I put them under the global global object, just like shared memory, which is easy to access and fast.

global.ShareMem = {
       "12345678": {// the room number is used as the key for easy searching
         Player: [{socket: connection, place: place}], // cell phone array: connection handle, seat number
         Projector: [], // array on PC side
         Maxplayer: 2, // maximum number of seats
         Type: "DDD" // room type


If you are the God of nodejs, or are using the nodejs framework such as koajs and express, please skip this section. Because I wrote webserver with the original nodejs once. Although it’s not good to build wheels repeatedly, it’s good to review the basic knowledge of webserver. This section is suitable for beginners.
Including knowledge points: header parsing, static file searching, gzip, file hash calculation, status code.

2.1 directory structure

    /Funmap.js / * HTTP function set*/
    /Funmap.js / * websocket function set*/
/Util / * tool directory, get local IP, open default browser*/
    /Common / * public resource directory*/
    /M / * mobile HTML, JS, CSS, etc*/
    /P / * PC HTML, JS, CSS, etc*/
/Index.js / * entry file*/
/Config.js / * configuration file, port number, WS maximum packet size, etc*/
/Socketserver.js / * websocket handler*/

2.2 webServer

The basic rule is as follows: build a static server, read and return the static resources normally, and render the HTML file with EJS and then return.

Because of EJS, the HTML file has not been modified, but the rendered content has been modified. For example, the port of WS has been changed, but the HTML file has not been modified. So it can’t be usedLast-ModifiedTo determine whether the file is up-to-date or not, you need to use theEtag

Etag needs to calculate the hash value according to the content, generally MD5.

Before returning content, gzip compression is required to save bandwidth. 90KB of jquery.min.js can be gzip to 30KB. Compression is the king.

Because the mobile terminal and the PC terminal execute completely different codes, it is necessary to judge the code transmitted from the clientuser-agentInclude or notMobileString to distinguish whether the client is a PC or a mobile phone, so as to return the correct resources.

Distinguish static files from rest requests by simple conventions

if (libPath.extname(pathName) == "") {
      //If the path does not have an extension 
        Pathname + = "/"; // access the root directory 
      }Else if (params [1] = = "API") {// access begins with API
        Parseapi (params, req, RES); // function
        return ;
        pathName = params[1]+".html";

I have made a simple framework here. Add a JS file in the API directory or socket API directory, a JS file corresponds to a processing function, and then aggregate it into a map in funmap.js, which is convenient to find the function, and easy to isolate and modify the function name.

var funMap = {
module.exports = funMap;

When the client accesses, it can access the desired service through / API / [functionname].

3 webSocketServer

Nodejs itself does not provide a module of websockerserver, so you need to install another one.

When NPM is installed, a WS module will be installed, and require (“WS”) can be used. The usage is similar to that of HTTP modulecreateServer({options},MainHandlerFunction)To create a service, WS has only a few more parameters.

Mainlyport, be careful not to duplicate the webserver port.
One moremaxPayloadIt is the maximum size of a single WS packet, in bytes. You can estimate the packet size when the project transmits data. The default is 65535, or 64KB. Generally, websocket is used for packet transmission, not too large. I set 1024, 1KB.

Main processing functionMainHandlerFunction, a parameter will be passed in when a client connects inconnection, the content of this object is very rich. You can print it out and study it slowly without reading the manual.
The way to establish a successful connection is toconnectionbindingmessagemethod.

Because wssocket access can carry URL, we can use URL to isolate different function functions instead of parsing message body.

var connectHandler = function(connection){
  // :4002/api/Function1 
  var URIarray = connection.upgradeReq.url.split("/")
    connection.send("{err:Function Not Found!!}");

3.1 news, broadcast, live

Whenever WS is connected, there is a file descriptor like ID to distinguish each different connection.
connection._ultron.idIt’s useful to distinguish your connection with others.

//Message format
function msgPack(){
  return JSON.stringify({
    "who":arguments[0],      // Mobile , PC
    "Place": arguments [1], // seat
    "dowhat":arguments[2],   // "connect","ready","message","lost"
    "Content": arguments [3] | "" // content

//Broadcast by room, broadcast all roles in the room
function boradCast(room,msg,ignore){
      // console.log("ignore!!!")
      //Ignore yourself and don't send it to yourself
      // console.log("ignore!!!")
      //Ignore yourself and don't send it to yourself

In order to check whether the client is offline, add the live protection mechanism manually when establishing the connection. The method is simple:
When sending an empty message to the client, lastkeeplife is 1. As long as the client returns any message, lastkeeplife is updated to 0. If there is no reply within 5 seconds, it will be judged as offline.
If the client drops the connection, close the connection and remove it from the connection pool. And broadcast the offline message to other roles in the room.

var keeplifeHandler = setInterval(function(){
    if(lastkeeplife == 0){
      lastkeeplife = 0;
      console.log("keep live error! "+ e +"\n\n");

      If (keepalifehandler) {// close the keep alive loop
      var room = global.ShareMem.rooms[roomid];
      //Remove connection handle from connection pool
      if(platform === PC){
              if(item.socket === connection){
                  return false;
              if(item.socket === connection){
                  return false;
      //Send offline message
      boradCast( room, msgPack(platform,place,"lost") , connection );

If the IOS device locks the screen, it will send a disconnection message to the server, while Android will not. To disconnect, you must wait until the default 120 second timeout to turn off.
Ws initialization did not provide the configuration of initialization timeout. By modification
WS. _server. Timeout = 1000; / / 1 second timeout
It will not take effect. Here comes the problem. How can I modify it to set the timeout?

At present, we can only use the above emergency methods to disconnect the offline equipment in time.


Multi screen interaction is not a new thing. I was inspired by a project called “lightsaber out of sheath” in chrome lab to make this demo. Because the mobile phone and PC need to turn over the wall at the same time during the experience, resulting in poor experience, and then I want to do one myself. When I do it, I feel cool, magical and excited.
There are still many things that can be expanded and improved in the future. I hope that they can eventually become a mature product, rather than just stop at demo.

Related reading

  • No flash required for image clipping – HTML5 intermediate advanced

  • Five tips to improve the application performance of node.js

  • Browser storage and use

Author information
The author comes from Li Pu, Suyun, leapcloud team member: Wang Shishi [original]
First address:

Wang Shishi, a new front-end person, has been working in front-end for two years. Once worked in AMI as a low-level software developer. Like to analyze H5 code, pursue simple CSS, build exquisite dynamic effect, before doing front end, these are hobbies. Currently working in maxleap UX group, responsible for the development and maintenance of maxwon. Now I’m keen on real time webapp.

Welcome to the wechat subscription number: maxleap ﹣ yidongyanfa