[convert] javascript high performance array de duplication

Time:2019-11-13

Original address: https://www.cnblogs.com/wisewrong/p/9642264.html

 

I. test template

Array de duplication is a commonplace problem. There are various solutions on the Internet

In order to test the performance of these solutions, I wrote a test template to calculate the time-consuming of array de duplication

// distinct.js

let arr1 = Array.from(new Array(100000), (x, index)=>{
    return index
})

let arr2 = Array.from(new Array(50000), (x, index)=>{
    return index+index
})

let start = new Date().getTime()
Console. Log ('Start array de duplication ')

function distinct(a, b) {
    //Array de duplication
}

Console. Log ('length after de duplication ', distinct (Arr1, arr2). Length)

let end = new Date().getTime()
Console.log ('time consuming ', end - start)

Two arrays of 10W and 5W are created here

Then merge the two arrays through the distinct () method and remove the duplicates

The amount of data is not small, but it can explain some problems

 

II. Array. Filter() + indexof

The idea of this method is to splice two arrays into an array, then use the array. Filter(), in ES6, to traverse the array, and use indexof to exclude duplicates

function distinct(a, b) {
    let arr = a.concat(b);
    return arr.filter((item, index)=> {
        return arr.indexOf(item) === index
    })
}

This is the way I repeat the array of tucks, which looks very simple, but the actual performance.

Yes, the reality is so cruel. It takes 8427ms to process an array with a length of 15W

 

III. double for cycle

The easiest way to understand, outer loop traverses elements, inner loop checks for repetition

When there are duplicate values, you can use push() or splice()

function distinct(a, b) {
    let arr = a.concat(b);
    for (let i=0, len=arr.length; i

But this method takes up more memory and has the lowest efficiency

 

 

IV. for… Of + includes ()

Upgrade version of double for loop, replace for loop with for… Of statement in the outer layer, and change the inner loop to include ()

Create an empty array first. When includes() returns false, push the element into the empty array

Similarly, indexof() can be used instead of includes()

function distinct(a, b) {
    let arr = a.concat(b)
    let result = []
    for (let i of arr) {
        !result.includes(i) && result.push(i)
    }
    return result
}

This method is similar to filter + indexof

Just use for loop to implement the internal logic of filter (), and then replace indexof with includes

So the time is close

 

  

V. array. Sort ()

First, sort the array using sort()

Then compare adjacent elements for equality to exclude duplicates

function distinct(a, b) {
    let arr = a.concat(b)
    arr = arr.sort()
    let result = [arr[0]]

    for (let i=1, len=arr.length; i

This method only has one sorting and one cycle, so the efficiency will be higher than the above method

 

  

Vi. new set ()

ES6 adds the data structure of set, which is similar to array, butMembers of set are unique

Based on this feature, it is very suitable for array de duplication

function distinct(a, b) {
    return Array.from(new Set([...a, ...b]))
}

How long does it take to process 15W data with set?

Meow, meow, meow 57ms?? I’m blind, right??

Then I add a 0 after the length of two arrays, under the data amount of 150W

There is such a high-performance and concise array de duplication method?!

 

VII. For… Of + object

I’ve only seen this method in some articles, but it’s not very useful in practical work

First create an empty object, and then loop through it with for

utilizeObject’s properties do not repeatThis feature checks whether the array elements are repeated

function distinct(a, b) {
    let arr = a.concat(b)
    let result = []
    let obj = {}

    for (let i of arr) {
        if (!obj[i]) {
            result.push(i)
            obj[i] = 1
        }
    }

    return result
}

When I see the processing time of this method is long, I am stupid again

15W data is only 16ms Faster than set()???

Then I tried 150W data

Emmmmmmm

Recommended Today

Using native es module in node.js

Original: using ES modules natively in node.js By Axel rauschmayer Starting from version 8.5.0, node.js supports native es modules, which can be turned on through command-line options. Much of the new functionality is due to Bradley Farias. 1. demo The code directory structure of this example is as follows: esm-demo/ lib.mjs main.mjs lib.mjs: export function […]