Front end screenshot implementation

Time:2021-10-25

preface

Demand background:

  • Goal: turn the page content into pictures and share them
  • Persistence: no server-side storage is required
  • Client capability: independent of client capability

Use pupeteer to do screenshot service

advantage:

  • The front end is easy to use, just call the interface
  • The sharing page can be separated from the content of the page displayed to users, and there is no need to deal with problems such as style

Disadvantages:

  • New service & maintenance service
  • Long interface response time will affect the experience

conclusion

Research found that it can be usedhtml2canvasThe pure front-end does the function of converting pages into pictures.

text

How the front end produces a picture, the first thing you can think of is to use canvas to draw the area, and then turn it into a picture. However, manual canvas drawing has the following problems:

  • Ctx.drawimage cannot be used to directly take an overall screenshot of the shared area:ctx.drawImageFor picture elementsImagesvgElement, video elementVideoCanvasElementsImageBitmapData, etc. can be drawn, but for general othersdivOr listliElement, which cannot be drawn
  • callcanvasDrawing: layout calculation is required, and drawing is also very cumbersome, resulting in a large amount of development
  • Some style issues / device compatibility issues need to be resolved

For the above reasons, we intend to use open source NPM to implement itDOM to pictureFunction.
The following libraries are used more frequently:

Analysis of Library advantages and disadvantages

html-to-image

/* ES6 */
import * as htmlToImage from 'html-to-image';
import { toPng, toJpeg, toBlob, toPixelData, toSvg } from 'html-to-image';

Supported configurations:

filter / backgroundColor / width, height / style / cacheBust / inagePlaceholder /  pixelRatio / fontEmbedCss

Summary:

  • Advantages: small warehouse 183KB; It is relatively simple to use and generally does not need to be configured
  • Disadvantages: there are many compatibility problems in history; The content picture Src in the screenshot area does not recognize Base64; Clarity problem (it seems ok now)

html2canvas

Supported configurations:https://html2canvas.hertzen.c…

Summary:

  • Advantages: 23K + start quantity, powerful function and many solutions; No processing is required for clarity
  • Disadvantages: reduction degree problem; Have configuration burden; 2.96 MB

about   foreignObject:

Front end screenshot implementation

html-to-imageBecause it’s used in implementation  foreignObjectTherefore, the browser must support this property. After looking at the support level, chrome Safari supports very well except ie. So don’t worry about not using HTML to image.

Problems that cannot be solved by html2canvas [avoid during development]:

There are two important issues to pay attention to:

  • Picture problem: support img label picture, background picture, not border picture
  • Font problem: fonts defined with fontface are not supported

Explanation in html2canvas:

Front end screenshot implementation

summary

  • Conservative solution: use html2canvas directly because of its powerful function and good compatibility
  • Ultimate pursuit: you can give priority to HTML to image because it is very small & you can call it foolishly without worrying about Configuration & high degree of restoration. Use html2canvas for the bottom of the pocket.

Scenario 1: the screenshot content cannot directly reuse the page content

There is a share button at the bottom of the main page. After clicking, you can share a picture to the circle of friends. At the bottom of the picture, there is an applet QR code with applet path parameters,The content of the picture is inconsistent with the content of the page displayed to the user

conceptual design

Because the picture content is dynamic and composed of data, we need to draw it manually. The simple way to think of isCreate the poster content into a div and hide the Div。 After that, the div can be operated as a whole.

Several element invisibility schemes
display: none  ❎
visibility: hidden    [black screen]
margin-left / nargin-top: -1000px    [black screen]
position:absolute;  Left / top: - 1000px [black screen]
z-index: -1000 ✅

In addition to invisible elements, in order to achieve the hiding effect without affecting the normal display of the page, you also need to deal withThe shared element content does not occupy the position of the document stream

Specific code

html:

<div id="app">
  <div class="wrapper">
    ...
    <div id="share" class="share-screen">
      ...
    </div>
  </div>
</div>

// style
.wrapper{
  position:relative;
}
.share-screen{
    width: 100%;
    z-index: -100; //  User invisible
    position: absolute;
 //Positioning
    top: 0;
    left:0;
    background-color: #fff;
}

js:

//Get pictures of content in the sharing area
  private getShareImg() {
    const node = document.getElementById('share');

    return html2canvas(node, {
      useCORS: true,
      x: 0,
      y: 0,
      logging: false,
    }).then(canvas => {
      return this.getPosterUrl(canvas);
    }).catch();
  }

  //Convert to a picture Base64
  public getPosterUrl(canvas): string {
    return canvas.toDataURL('image/jpeg');
    // return canvas.toDataURL('image/jpeg').replace(/data:image\/\w+;base64,/, '');
  }

Question reference

Picture cross domain problem

useCORSSet totrue

Picture definition problem

After testing,html2canvas v1.0.01-rc7After that, there is no problem of clarity

v1.0.01-rc5Previous reference:

/*
*Image cross domain and screenshot blur processing
* https://www.cnblogs.com/padding1015/p/9225517.html 
*/
      Let sharecontent = domobj, // wrapped (native) DOM object requiring screenshots
          width = shareContent.clientWidth,//shareContent.offsetWidth; // Get DOM width
          height = shareContent.clientHeight,//shareContent.offsetHeight; // Get DOM height
          Canvas = document.createelement ("canvas"), // create a canvas node
          scale = 2; // Define any magnification and support decimals
      canvas.width = width * scale; // Define canvas width * scale
      canvas.height = height * scale; // Define canvas height * scale
      canvas.style.width = shareContent.clientWidth * scale + "px";
      canvas.style.height = shareContent.clientHeight * scale + "px";
      canvas.getContext("2d").scale(scale, scale); // Get context and set scale
      let opts = {
          Scale: scale, // added scale parameter
          Canvas: canvas, // custom canvas
          Logging: false, // the log switch is used to view the internal execution process of html2canvas
          Width: width, // DOM original width
          height: height,
          Usecors: true // [important] enable cross domain configuration
      };
html2canvas(shareContent,opts).then()
The content area of the screenshot is outside the visual area, and the HTML 2canvas screenshot is blank

You need to determine your screenshot area. Screenshot area style settings:

// html
<div id="app">
  <div class="wrapper">
    ...
    <div id="share" class="share-screen">
      ...
    </div>
  </div>
</div>

// style
.wrapper{
  position:relative;
}
.share-screen{
    width: 100%;
    z-index: -100; //  User invisible
    position: absolute;
 //Positioning
    top: 0;
    left:0;
    background-color: #fff;
}

Html2canvas configuration:

html2canvas(node, {
      useCORS: true, 
      x: 0, // x-offset:Crop canvas x-coordinate
      y: 0, // y-offset:Crop canvas y-coordinate
      logging: false,
}).then();
Error reporting using HTML to image

An error will be reported on Android:
Error: Failed to execute 'toDataURL' on 'HTMLCanvasElement'......

Please change it tohtml2canvas

The problem of picture size adaptation in HTML 2 canvas

The main problem here is DPR. The library will automatically match deviceswindow.devicePixelRatio。 After the above configuration, there will be no problem.

Html2canvas remote pictures [allow cross domain] do not show problems occasionally

html2canvasThe screenshot principle is node cloning and style cloning, so even if the pictures on the page are loaded, the pictures in the screenshot cannot be guaranteed to be normal. Remote image loading takes time, which is uncertain. If the picture is converted to a picture before it is loaded back, there will be a problem that the picture will not be displayed.

The solution is:

  • Loading local images will not have this problem
  • Instead of returning the HTTP address, the remote image can be directly rendered by the browser by returning the base64 encoding, which is equivalent to returning the image content directly. Or manually request the picture address and transfer the binary to Base64.

Possible difficulties

There are a lot of pictures in the page content. Because the picture content needs to be requested during the screenshot process, a large number of picture requests may not guarantee the integrity of the screenshot. Even if the picture request is OK,html2canvasSome pictures may also be lost in screenshot processing. (Reference:https://zhuanlan.zhihu.com/p/…)

So using a remote picture address is risky. After the real machine test, it is found that the probability of error is indeed very high (50% is available).

How to solve it?

  • Convert to Base64
  • Convert to blob and store it in memory with url.createbloburl

Convert to Base64

This needs to discuss the advantages and disadvantages of Base64.
advantage:

  • Using Base64 resources can reduce the number of network requests (usually remote pictures)
    The picture content is displayed directly, and the screenshot problem caused by time-consuming or failed picture loading will not occur
  • Todataurl () has strong compatibility. It is basically supported by other browsers except ie < = 8 Safari < = 3(https://caniuse.com/?search=c…)
    Disadvantages:
  • The size is 1 / 3 larger than the original picture
  • Pay attention to the format when converting, PNG   → JPEG transparent area will turn black background (solution:https://evestorm.github.io/po…

transformation:
Dynamically create a picture label, draw the picture on the canvas, and usecanvas.toDataURLexportbase64

function toDataURL(url) {
    //Skip Base64 pictures
    const base64Pattern = /data:image\/(.)+;base64/i;
    if (base64Pattern.test(url)) {
      return Promise.resolve(url);
    }

    const fileType = getFileType(url) || 'png'; //  The default is PNG. PNG supports transparency

    return new Promise((resolve, reject) => {
      const canvas = document.createElement('canvas');
      const ctx = canvas.getContext('2d');
      const img = document.createElement('img');

      img.crossOrigin = 'Anonymous'; //  Solve cross domain image problems,
      img.src = url;
      img.onload = () => {
          canvas.width = img.width;
          canvas.height = img.height;
          ctx.drawImage(img, 0, 0);
          const dataUrl = canvas.toDataURL(`image/${fileType}`);
          resolve(dataUrl);
      };
      img.onerror = e => {
        reject(e);
      };
    });
  }

  //Get picture file format
  function getFileType(url) {
    const pattern = /.*\.(png|jpeg|gif)$/i; //  Image types supported by data URI scheme
    return (url.match(pattern) || [undefined, undefined])[1];
  }

Convert to blob

advantage:

  • There are rich formats that support conversion. In addition to image, other types can also be supported. There is no need to care about file type conversion

Disadvantages:

  • There are still some problems with compatibility. IE6 ~ 9 Safari < = 10 are not supported(https://caniuse.com/?search=c…)
  • The revokeobjecturl needs to be called to free up memory

transformation:

function toBlobURL(url) {
    return new Promise((resolve, reject) => {
      const canvas = document.createElement('canvas');
      const ctx = canvas.getContext('2d');
      const img = document.createElement('img');

      img.crossOrigin = 'Anonymous'; //  Solve cross domain image problems,
      img.src = url;
      img.onload = () => {
          canvas.width = img.width;
          canvas.height = img.height;
          ctx.drawImage(img, 0, 0);
          //Key
          canvas.toBlob(blob => {
             const blobURL = URL.createObjectURL(blob);
             resolve(blobURL);
            //Release bloburl
            URL.revokeObjectURL(blobURL)
         });;
      };
      img.onerror = e => {
        reject(e);
      };
    });
  }

Realize the SRC of all pictures on the page, and use Base64

const urlmap = {};

  //Get picture file format
  function getFileType(url) {
    const pattern = /.*\.(png|jpeg|gif)$/i; //  Image types supported by data URI scheme
    return (url.match(pattern) || [undefined, undefined])[1];
  }

  //Remote URL to Base64
  function toDataURL(url) {
    //Filter duplicate values
    if (urlMap[url]) {
      return Promise.resolve(urlMap[url]);
    }

    //Skip Base64 pictures
    const base64Pattern = /data:image\/(.)+;base64/i;
    if (base64Pattern.test(url)) {
      return Promise.resolve(url);
    }

    const fileType = getFileType(url) || 'png'; //  The default is PNG. PNG supports transparency

    return new Promise((resolve, reject) => {
      const canvas = document.createElement('canvas');
      const ctx = canvas.getContext('2d');
      const img = document.createElement('img');

      img.crossOrigin = 'Anonymous'; //  Solve cross domain image problems,
      img.src = url;
      img.onload = () => {
          canvas.width = img.width;
          canvas.height = img.height;

          ctx.drawImage(img, 0, 0);
          const dataUrl = canvas.toDataURL(`image/${fileType}`);
          resolve(dataUrl);
      };
      img.onerror = e => {
        reject(e);
      };
    });
  }

  //Batch transfer to Base64
  function convertToBlobImage(targetNode) {
    if (!targetNode) { return Promise.resolve(); }

    let nodeList = targetNode;
    if (targetNode instanceof Element) {
      if (targetNode.tagName.toLowerCase() === 'img') {
        nodeList = [targetNode];
      } else {
        nodeList = targetNode.getElementsByTagName('img');
      }
    } else if (!(nodeList instanceof Array) && !(nodeList instanceof NodeList)) {
      Throw new error ('[converttoblobimage] must be of type element or NodeList');
    }

    if (nodeList.length === 0) {
      return Promise.resolve();
    }

    //Consider only < img >
    return new Promise(resolve => {
      let count = 0;
      //Replace < img > resource addresses one by one
      for (let i = 0, len = nodeList.length; i < len; ++i) {
        const v = nodeList[i];
        let p = Promise.resolve();
        if (v.tagName.toLowerCase() === 'img') {
          p = toDataURL(v.src).then(blob => {
            v.src = blob;
          });
        }

        p.finally(() => {
          if (++count === nodeList.length) {
            return resolve(true);
          }
        });
      }
    });
  }

matters needing attention:

  • When the remote picture is PNG, it should be converted to PNG format(toDataURL(image/png)), because PNG images support transparency, the alpha channel will be closed when converting to JPEG files, resulting in the transparent area becoming a black background. The recommended adaptive file type is PNG by default
  • Image type supported by data URI scheme: png | JPEG | gif
  • When drawing pictures on canvas, errors may be reported:"Failed to execute 'toBlob' on 'HTMLCanvasElement': Tainted canvases may not be exported"The reason is that CORS pictures will pollute the canvas and make it impossible to read its data. Add the following code to solve the cross domain image problem:img.crossOrigin = 'Anonymous'

Expand knowledge

<img /> srcThree types are supported:url/base64/blob

1. Talk about several common web image formats: gif, JPG, PNG, webp

Front end screenshot implementation

2. Process arraybuffer file
// buffer -> base64
function getData(){
  let binary = '';
  const bytes = new Uint8Array(buffer);
  const len = bytes.byteLength;
  for (let i = 0; i < len; i++) {
   binary += String.fromCharCode(bytes[i]);
  }
  return 'data:image/png; base64,' + btoa(binary); //  The window. Btoa () method is used to create a base-64 encoded string.
}
3. Base64 to blob:
//Principle: use url.createobjecturl to create temporary URL for blob object
function base64ToBlob ({b64data = '', contentType = '', sliceSize = 512} = {}) {
  return new Promise((resolve, reject) => {
    //Decode the data using the atob () method
    let byteCharacters = atob(b64data);
    
let byteArrays = [];
    for (let offset = 0; offset < byteCharacters.length; offset += sliceSize) {
        let slice = byteCharacters.slice(offset, offset + sliceSize);
        let byteNumbers = [];
        for (let i = 0; i < slice.length; i++) {
            byteNumbers.push(slice.charCodeAt(i));
        }
        //A typed array of 8-bit unsigned integer values. Content will be initialized to 0.
        //If the requested number of bytes cannot be allocated, an exception is thrown.
        byteArrays.push(newUint8Array(byteNumbers));
      }
      let result = new Blob(byteArrays, {
        type: contentType
      })
      result = Object.assign(result,{
        //Jartto: be sure to deal with the URL. Createobjecturl here
        preview: URL.createObjectURL(result),
        Name: ` picture example.png`
      });
      resolve(result)
    })
 }
4. Blob to Base64:

Compatibility: ie > 9 Safari > 5   https://caniuse.com/?search=r…

//Principle: use readasdataurl of FileReader to convert blob to Base64
function blobToBase64(blob) {
  return new Promise((resolve, reject) => {
      const fileReader = new FileReader();
      fileReader.onload = (e) => {
        resolve(e.target.result);
      };
      // readAsDataURL
      fileReader.readAsDataURL(blob);
      fileReader.onerror = () => {
        Reject (newerror ('File stream exception ');
      };
    });
}

Scenario 2: the screenshot content is an extension of the page content

The main content of the screenshot is the content displayed on the page, and the bottom is spliced to guide the banner.

How to realize dynamic splicing

have access tohtml2canvasofoncloneProperty to modify the copied node content. Modifying elements in onclone will not affect the original elements of the page.

Detach dynamic display content from the document stream and make it invisible:
<!-- html -->
<div class="wrapper">
  <!--  Main contents -- >
  <div class="content" id="share"></div>
  <!--  Dynamically added content -- >
  <div class="dynamic-contetn" id="share-footer"></div>
</div>
// css
.wrapper{
  position:relative;
}
.dynamic-content{
   
position: absolute;
  top:0;
  left:0;
  right: 0;
  z-index:-100;
}
During the screenshot, modify the onclone to make the dynamic content appear in the normal position and make it visible:
const node = document.getElementById('share'); //  Share the main content area of the poster
  const 
shareFooter = document.getElementById('share-footer'); //  Bottom content of poster
  html2canvas(node,{
      onclone: (doc: Document) => {
        //Display the content area at the bottom of the poster for screenshots with the content
        shareFooterNode.style.position = 'relative';
        shareFooterNode.style.display = 'flex';
        shareFooterNode.style.zIndex = '1';
      }
   }).then();
Control screenshot size:

Because the height of the screenshot is the offset height of the node element by default, now the height will becomeNode height + dynamic area height

const node = document.getElementById('share'); //  Share the main content area of the poster
    const shareFooter = document.getElementById('share-footer'); //  Bottom content of poster
    const { height: footerHeight} = shareFooter.getBoundingClientRect();

    html2canvas(node, {
        useCORS: true,
        logging: false,
        height: node.offsetHeight + footerHeight,
       }).then();

Another practice of intercepting visual area content

Add a shell to the sub element of the body to make the web content scroll in the shell.

html:

<html>
 <body>
   <div class="wrapper">
     <div id="share" class="content"></div>
   </div>
 </body>
</html>
// css

.wrapper{
  position:fixed;
  left:0;
  right:0;
  top:0;
  bottom:0;
}

javascript:

const wrapper = document.getElementsByClassName('wrapper')[0]; //  Wrapper is used to listen for scroll
    const node = document.getElementById('share'); //  Share the main content area of the poster

    const { scrollTop } = wrapper;
    const { height: footerHeight} = shareFooter.getBoundingClientRect();

    return this.convertToBlobImage(node).then(res => {
      return html2canvas(node, {
        useCORS: true,
        logging: false,
        x: 0,
        y: - scrolltop, // important
        Height: node.offsetheight, // important
      }).then();

This can also be a normal screenshot.
Therefore, the X / y setting has something to do with your layout. You can try all kinds and use the scheme with normal screenshots.

Summary under two scenarios

The screenshot content is completely different from the page content:

  • Prepare a screenshot of the node content in the area and hide it
  • usex:0 y:0To solve the screenshot area problem

The screenshot content is consistent with the main content of the page:

  • Reuse page node content
  • Use the shell and under the bodyy:-scrollTopSolve the problem of screenshot area
  • If there is dynamic content, useheight: node.offsetHeight + dynamicHeight

Images should be converted into Base64 content to ensure the integrity of screenshots

Two major problems in special scenarios

onecanvasCaused by excessive sizecanvas.toDataURL()returndata:/Question of

terms of settlement:
Check whether it is legal. If it is not, reduce the html2canvas scale (which will affect the clarity). It’s not legal to drop to 1, so there’s no way.

There are no restrictions on each browser. There is no API directly used for detection. A detection library is recommendedcanvas-size。 Start the test from the current DPR of the device, and downward compatibility:

  private async getAvailableScale(node) {
    let scale = 1;

    const {offsetWidth, offsetHeight} = node;
    const nodeWidth = Math.ceil(offsetWidth);
    const nodeHeight = Math.ceil(offsetHeight);
    const dpr = window.devicePixelRatio;

    const sizes = [];
    for (let i = dpr; i >= 1; i--) {
      sizes.push([nodeWidth * dpr, nodeHeight * dpr]);
    }

    try {
      const { width, height, benchmark } = await canvasSize.test({
        sizes,
        usePromise: true,
        useWorker : true,
      });
      console.log(`Success: ${width} x ${height} (${benchmark} ms)`);
      scale =  width / nodeWidth;
    } catch ({ width, height, benchmark }) {
      console.log(`Error: ${width} x ${height} (${benchmark} ms)`);
    }
    return scale;
  }

II. 512k limit of wechat SDK sharing circle of friends

Solution:
Using the second parameter of canvas.todataurl (type, quality) to compress the picture will reduce the picture size (but also the picture quality).quality: (0,1), default is0.92

public getPosterUrl(canvas): string {
    const img = canvas.toDataURL('image/jpeg');
    const size =  Math.ceil(getImgSize(img) / 1024); //  Get Base64 picture size
    Console.log ('poster picture size (KB): ', size);
    Console.log ('poster picture quality: ', 400 / size)// If the value range is beyond 0-1, the default value of 0.92 will be used
    //Huawei mate20 / Android 10 sharing will fail if it is not compressed under the long graph
   //400KB is a good reference value based on 512KB
    return canvas.toDataURL('image/jpeg', 400 / size).replace(/data:image\/\w+;base64,/, '');
  }

DOM to image principle

There may one day be (or may it already exist?) a simple and standard way to export a part of HTML to an image (then this script becomes evidence of all the problems I have experienced in order to achieve this goal), but so far I haven’t found it.

The library uses the function of SVG, which allows<foreignObject>Tag contains arbitrary HTML content. Therefore, in order to present theDOMNode, the following steps are performed:

  1. Recursive cloning of original DOM nodes
  2. Calculate the style of the node and each child node and copy it to the appropriate clone

    • And don’t forget to recreate the pseudo elements, because of course they won’t be cloned in any way
  3. Embedded network font

    • Find all possible network fonts@font-facestatement
    • Parse the file URL and download the corresponding file
    • base64Encode and use inline content asdataURLs
    • Connect all processed CSS rules, place them in a < style > element, and attach them to the clone
  4. Embedded picture

    • Embed the image URL into the < img > element
    • Embedded images used in background CSS properties in a manner similar to fonts
  5. Serialize the cloned node into XML
  6. Wrap XML into<foreignObject>Tag, then wrap it in SVG, and make it a data URL
  7. (optional) to obtain PNG content or original pixel data in the form of uint8array, create aSVGSourceImageElement and render it on the off screen canvas you also created, and then read the content from the canvas
  8. complete

reference material

Postscript

There will be new scenes and problems accumulated in the follow-up, which will continue to be supplemented.