Summary of principles and methods of file breakpoint continuation in node

Time:2021-7-27

Introduction: I have done a small project before, in which file upload is used, and breakpoint continuation is used on large files, which reduces the pressure on the server. Now I make a detailed summary of this development experience.

catalogue

  • Principle introduction
  • Method summary
  • Actual combat drill

Principle introduction

Here is the principle of file upload to help clarify this clue.

Normal upload

Generally, there are many ordinary uploads on websites. Most of them upload some user’s avatars, user’s dynamic comments and pictures, so let’s talk about the principle of this first.

  • After the user selects a file, JS checks whether the file size exceeds the limit and whether the format is correct;
  • After checking, AJAX is used to submit the request, and the server also performs secondary verification and stores it in the server;
  • The back end returns the file address to the front end and renders the page data;

Large file upload

  • After the user selects a file, JS checks whether the file size exceeds the limit and whether the format is correct;
  • Depending on the file size, usefile.sliceMethod for file segmentation;
  • useSparkMD5andFileReaderThe unique MD5 value of the API generated file;
  • Submit the request using Ajax, and the server returns the information of the file on the server after receiving it;

    • If there is a folder with this MD5 value, the number of files uploaded in the folder will be returned;
    • If it does not exist, create a new folder with this MD5 value and return empty content;
  • After receiving the information, the front end makes a judgment according to the returned information;

    • If the length of the returned file is equal to the total length of the slice, request to merge the file;
    • If the length of the returned file is less than the total length of the slice, start uploading the corresponding slice file until the last slice is uploaded, and then request to merge the file;
  • After receiving the merge request, the backend will merge the files in the corresponding MD5 value folder and return the file address;
  • After receiving the file address, the front end renders the page data;

Breakpoint continuation

This means that in the process of file upload, in case of force majeure, such as network interruption, server abnormality, or other reasons leading to upload interruption;

In the next upload, the server will find out how many files have been uploaded and how many have not been uploaded according to the MD5 value of the file, and send them to the client. After receiving them, the client will continue to upload the non uploaded files, merge the files and return the address.

This avoids repeated uploading of files, wastes server space, saves server resources, and is faster and more efficient than uploading a large file.

Method summary

Next, analyze the steps according to the above logic principle to realize the code function.

Ordinary file

This section describes the uploading of ordinary files, including the front-end part and the back-end part.

Front end part

  • HTML part

Let’s build a small house first

<div class="upload">
        <h3>Normal upload</h3>
        <form>
            <div class="upload-file">
                < label for = "file" > please select the file < / label >
                <input type="file" name="file" id="file" accept="image/*">
            </div>
            <div class="upload-progress">
                Current progress:
                <p>
                    <span style="width: 0;" id="current"></span>
                </p>
            </div>
            <div class="upload-link">
                File address: < a id = "links" href = "javascript: void();" target="_ Blank "> file link</a>
            </div>
        </form>
    </div>
    <div class="upload">
        <h3>Large file upload</h3>
        <form>
            <div class="upload-file">
                < label for = "file" > please select the file < / label >
                <input type="file" name="file" id="big-file" accept="application/*">
            </div>
            <div class="upload-progress">
                Current progress:
                <p>
                    <span style="width: 0;" id="big-current"></span>
                </p>
            </div>
            <div class="upload-link">
                File address: < a id = "big links" href = "" target = "" "_ Blank "> file link</a>
            </div>
        </form>
    </div>

introduceaxiosandspark-md5Two JS files.

<script></script>
<script></script>
  • CSS part

Let’s decorate the house.

body {
    margin: 0;
    font-size: 16px;
    background: #f8f8f8;
}
h1,h2,h3,h4,h5,h6,p {
    margin: 0;
}

/* * {
    outline: 1px solid pink;
} */

.upload {
    box-sizing: border-box;
    margin: 30px auto;
    padding: 15px 20px;
    width: 500px;
    height: auto;
    border-radius: 15px;
    background: #fff;
}

.upload h3 {
    font-size: 20px;
    line-height: 2;
    text-align: center;
}

.upload .upload-file {
    position: relative;
    margin: 30px auto;
}

.upload .upload-file label {
    display: flex;
    justify-content: center;
    align-items: center;
    width: 100%;
    height: 150px;
    border: 1px dashed #ccc;
}

.upload .upload-file input {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
    opacity: 0;
}

.upload-progress {
    display: flex;
    align-items: center;
}

.upload-progress p {
    position: relative;
    display: inline-block;
    flex: 1;
    height: 15px;
    border-radius: 10px;
    background: #ccc;
    overflow: hidden;
}

.upload-progress p span {
    position: absolute;
    left: 0;
    top: 0;
    width: 0;
    height: 100%;
    background: linear-gradient(to right bottom, rgb(163, 76, 76), rgb(231, 73, 52));
    transition: all .4s;
}

.upload-link {
    margin: 30px auto;
}

.upload-link a {
    text-decoration: none;
    color: rgb(6, 102, 192);
}

@media all and (max-width: 768px) {
    .upload {
        width: 300px;
    }
}
  • JS part

Finally, add the interactive effect.

//Get element
const file = document.querySelector('#file');
let current = document.querySelector('#current');
let links = document.querySelector('#links');
let baseUrl = 'http://localhost:3000';

//Listen for file events
file.addEventListener('change', (e) => {
    console.log(e.target.files);
    let file = e.target.files[0];
    if (file.type.indexOf('image') == -1) {
        Return alert ('File format can only be pictures! ');
    }
    if ((file.size / 1000) > 100) {
        Return alert ('File cannot be larger than 100kb! ');
    }
    links.href = '';
    file.value = '';
    this.upload(file);
}, false);

//Upload file
async function upload (file) {
    let formData = new FormData();
    formData.append('file', file);
    let data = await axios({
        url: baseUrl+'/upload',
        method: 'post',
        data: formData,
        onUploadProgress: function(progressEvent) {
            current.style.width = Math.round(progressEvent.loaded / progressEvent.total * 100) + '%';
        }
    });
    if (data.data.code == 200) {
        links.href = data.data.data.url;
    } else {
        Alert ('upload failed! ')
    }
}

Back end part

Open last folderdemoFirst, download and install a package for processing filesformidable, and then start processing the uploaded files!

Create a new folder and don’t forget toapp.jsIntroduce a new file.

const upload = require('./upload/index');

app.use(express.static(path.join(__dirname, 'public')));
app.use('/file', express.static(path.join(__dirname, 'static')));

app.use('/upload', upload);

The following is the file hierarchy diagram.

-- static
    -- big
    -- doc
    -- temp
-- upload
    - index.js
    - util.js
-- app.js
const express = require('express');
const Router = express.Router();
const formidable = require('formidable');
const path = require('path');
const fs = require('fs');
const baseUrl = 'http://localhost:3000/file/doc/';
const dirPath = path.join(__dirname, '../static/')

//Ordinary file upload
Router.post('/', (req, res) => {
    let form = formidable({
        multiples: true,
        uploadDir: dirPath+'temp/'
    })

    form.parse(req, (err,fields, files)=> {
        if (err) {
            return res.json(err);
        }
        let newPath = dirPath+'doc/'+files.file.name;
        fs.rename(files.file.path, newPath, function(err) {
            if (err) {
                return res.json(err);
            }
            return res.json({
                code: 200,
                msg: 'get_succ',
                data: {
                    url: baseUrl + files.file.name
                }
            })
        })
        
    })

});

module.exports = Router;

Large file

This big file breakpoint continuation is actually a further extension of the previous file upload. Therefore, the structure and style of the front-end part are the same, but the methods are different.

Front end part

Here is the method introduction.

  • Get element
const bigFile = document.querySelector('#big-file');
let bigCurrent = document.querySelector('#big-current');
let bigLinks = document.querySelector('#big-links');
let fileArr = [];
let md5Val = '';
let ext = '';
  • Test file
bigFile.addEventListener('change', (e) => {
    let file = e.target.files[0];
    if (file.type.indexOf('application') == -1) {
        Return alert ('File format can only be document application! ');
    }
    if ((file.size / (1000*1000)) > 100) {
        Return alert ('File cannot be greater than 100MB! ');
    }
    this.uploadBig(file);
}, false);
  • Cut file
//Cut file
function sliceFile (file) {
    const files = [];
    const chunkSize = 128*1024;
    for (let i = 0; i < file.size; i+=chunkSize) {
        const end = i + chunkSize >= file.size ? file.size : i + chunkSize;
        let currentFile = file.slice(i, (end > file.size ? file.size : end));
        files.push(currentFile);
    }
    return files;
}
  • Gets the MD5 value of the file
//Get file MD5 value
function md5File (files) {
    const spark = new SparkMD5.ArrayBuffer();
    let fileReader;
    for (var i = 0; i < files.length; i++) {
        fileReader = new FileReader();
        fileReader.readAsArrayBuffer(files[i]);
    }
    return new Promise((resolve) => {
        fileReader.onload = function(e) {
            spark.append(e.target.result);
            if (i == files.length) {
                resolve(spark.end());
            }
        }
    })
}
  • Upload fragment file
async function uploadSlice (chunkIndex = 0) {
    let formData = new FormData();
    formData.append('file', fileArr[chunkIndex]);
    let data = await axios({
        url: `${baseUrl}/upload/big?type=upload&current=${chunkIndex}&md5Val=${md5Val}&total=${fileArr.length}`,
        method: 'post',
        data: formData,
    })

    if (data.data.code == 200) {
        if (chunkIndex < fileArr.length -1 ){
            bigCurrent.style.width = Math.round((chunkIndex+1) / fileArr.length * 100) + '%';
            ++chunkIndex;
            uploadSlice(chunkIndex);
        } else {
            mergeFile();
        }
    }
}
  • Merge files
async function mergeFile () {
    let data = await axios.post(`${baseUrl}/upload/big?type=merge&md5Val=${md5Val}&total=${fileArr.length}&ext=${ext}`);
    if (data.data.code == 200) {
        Alert ('upload succeeded! ');
        bigCurrent.style.width = '100%';
        bigLinks.href = data.data.data.url;
    } else {
        alert(data.data.data.info);
    }
}

Back end part

  • Get parameters

Obtain upload parameters and make judgment.

let type = req.query.type;
let md5Val = req.query.md5Val;
let total = req.query.total;
let bigDir = dirPath + 'big/';
let typeArr = ['check', 'upload', 'merge'];
if (!type) {
    return res.json({
        code: 101,
        msg: 'get_fail',
        data: {
            Info: 'upload type cannot be empty!'
        }
    })
}

if (!md5Val) {
    return res.json({
        code: 101,
        msg: 'get_fail',
        data: {
            Info: 'file MD5 value cannot be empty!'
        }
    })
}

if (!total) {
    return res.json({
        code: 101,
        msg: 'get_fail',
        data: {
            Info: 'the number of file slices cannot be empty!'
        }
    })
}

if (!typeArr.includes(type)) {
    return res.json({
        code: 101,
        msg: 'get_fail',
        data: {
            Info: 'upload type error!'
        }
    })
}
  • The type is detection
let filePath = `${bigDir}${md5Val}`;
fs.readdir(filePath, (err, data) => {
    if (err) {
        fs.mkdir(filePath, (err) => {
            if (err) {
                return res.json({
                    code: 101,
                    msg: 'get_fail',
                    data: {
                        Info: 'get failed!',
                        err
                    }
                })
            } else {
                return res.json({
                    code: 200,
                    msg: 'get_succ',
                    data: {
                        Info: 'get success!',
                        data: {
                            type: 'write',
                            chunk: [],
                            total: 0
                        }
                    }
                })
            }
        })
    } else {
        return res.json({
            code: 200,
            msg: 'get_succ',
            data: {
                Info: 'get success!',
                data: {
                    type: 'read',
                    chunk: data,
                    total: data.length
                }
            }
        })
    }
    
})
  • The type is uploaded
let current = req.query.current;
if (!current) {
    return res.json({
        code: 101,
        msg: 'get_fail',
        data: {
            Info: 'the current partition value of the file cannot be empty!'
        }
    })
}

let form = formidable({
    multiples: true,
    uploadDir: `${dirPath}big/${md5Val}/`,
})

form.parse(req, (err,fields, files)=> {
    if (err) {
        return res.json(err);
    }
    let newPath = `${dirPath}big/${md5Val}/${current}`;
    fs.rename(files.file.path, newPath, function(err) {
        if (err) {
            return res.json(err);
        }
        return res.json({
            code: 200,
            msg: 'get_succ',
            data: {
                info: 'upload success!'
            }
        })
    })
    
})
  • Types are merged
let ext = req.query.ext;
if (!ext) {
    return res.json({
        code: 101,
        msg: 'get_fail',
        data: {
            Info: 'file suffix cannot be empty!'
        }
    })
}

let oldPath = `${dirPath}big/${md5Val}`;
let newPath = `${dirPath}doc/${md5Val}.${ext}`;
let data = await mergeFile(oldPath, newPath);
if (data.code == 200) {
    return res.json({
        code: 200,
        msg: 'get_succ',
        data: {
            Info: 'file merge succeeded!',
            url: `${baseUrl}${md5Val}.${ext}`
        }
    })
} else {
    return res.json({
        code: 101,
        msg: 'get_fail',
        data: {
            Info: 'file merge failed!',
            err: data.data.error
        }
    })
}

The merge function mainly usesfsYescreateWriteStreamas well ascreateReadStreamMethod.

  • Merge files
const fs = require('fs');

function mergeFile (filePath, newPath) {
    return new Promise((resolve, reject) => {
        let files = fs.readdirSync(filePath),
        newFile = fs.createWriteStream(newPath);
        let filesArr = arrSort(files).reverse();
        main();
        function main (index = 0) {
            let currentFile = filePath + '/'+filesArr[index];
            let stream = fs.createReadStream(currentFile);
            stream.pipe(newFile, {end: false});
            stream.on('end', function () {
                if (index < filesArr.length - 1) {
                    index++;
                    main(index);
                } else {
                    resolve({code: 200});
                }
            })
            stream.on('error', function (error) {  
                reject({code: 102, data:{error}})
            })
        }
    })
}
  • File sorting
function arrSort (arr) {
    for (let i = 0; i < arr.length; i++) {
        for (let j = 0; j < arr.length; j++) {
            if (Number(arr[i]) >= Number(arr[j])) {
                let t = arr[i];
                arr[i] = arr[j];
                arr[j] = t;
            }
        }
    }
    return arr;
}

Actual combat drill

Now the method has been written to test whether it is OK.

Two files are prepared here to test the two functions respectively.

Summary of principles and methods of file breakpoint continuation in node

  • Ordinary file

This is an ordinary file upload interface

Summary of principles and methods of file breakpoint continuation in node

After uploading successfully:

Summary of principles and methods of file breakpoint continuation in node

Back end return content:

Summary of principles and methods of file breakpoint continuation in node

Open file address Preview:

Summary of principles and methods of file breakpoint continuation in node

You can see success!

  • Large file

This is the big file upload interface

Summary of principles and methods of file breakpoint continuation in node

After uploading successfully:

Summary of principles and methods of file breakpoint continuation in node

This is a fragment file being uploaded:

Summary of principles and methods of file breakpoint continuation in node

This is the content returned after the file is uploaded in pieces:

Summary of principles and methods of file breakpoint continuation in node

Open file address Preview:

Summary of principles and methods of file breakpoint continuation in node

Upload again and find that the file address will be returned soon:

Summary of principles and methods of file breakpoint continuation in node

Summary of principles and methods of file breakpoint continuation in node

This is a screenshot of nodejs directory. You can see that the file fragments are well preserved and merged.

Summary of principles and methods of file breakpoint continuation in node
Summary of principles and methods of file breakpoint continuation in node

That’s all for the file upload and breakpoint continuation. Of course, the method I mentioned above is only one for reference. If you have a better way, please contact me.