[convert] javascript high performance array de duplication


Original address: https://www.cnblogs.com/wisewrong/p/9642264.html


I. test template

Array de duplication is a commonplace problem. There are various solutions on the Internet

In order to test the performance of these solutions, I wrote a test template to calculate the time-consuming of array de duplication

// distinct.js

let arr1 = Array.from(new Array(100000), (x, index)=>{
    return index

let arr2 = Array.from(new Array(50000), (x, index)=>{
    return index+index

let start = new Date().getTime()
Console. Log ('Start array de duplication ')

function distinct(a, b) {
    //Array de duplication

Console. Log ('length after de duplication ', distinct (Arr1, arr2). Length)

let end = new Date().getTime()
Console.log ('time consuming ', end - start)

Two arrays of 10W and 5W are created here

Then merge the two arrays through the distinct () method and remove the duplicates

The amount of data is not small, but it can explain some problems


II. Array. Filter() + indexof

The idea of this method is to splice two arrays into an array, then use the array. Filter(), in ES6, to traverse the array, and use indexof to exclude duplicates

function distinct(a, b) {
    let arr = a.concat(b);
    return arr.filter((item, index)=> {
        return arr.indexOf(item) === index

This is the way I repeat the array of tucks, which looks very simple, but the actual performance.

Yes, the reality is so cruel. It takes 8427ms to process an array with a length of 15W


III. double for cycle

The easiest way to understand, outer loop traverses elements, inner loop checks for repetition

When there are duplicate values, you can use push() or splice()

function distinct(a, b) {
    let arr = a.concat(b);
    for (let i=0, len=arr.length; i

But this method takes up more memory and has the lowest efficiency



IV. for… Of + includes ()

Upgrade version of double for loop, replace for loop with for… Of statement in the outer layer, and change the inner loop to include ()

Create an empty array first. When includes() returns false, push the element into the empty array

Similarly, indexof() can be used instead of includes()

function distinct(a, b) {
    let arr = a.concat(b)
    let result = []
    for (let i of arr) {
        !result.includes(i) && result.push(i)
    return result

This method is similar to filter + indexof

Just use for loop to implement the internal logic of filter (), and then replace indexof with includes

So the time is close



V. array. Sort ()

First, sort the array using sort()

Then compare adjacent elements for equality to exclude duplicates

function distinct(a, b) {
    let arr = a.concat(b)
    arr = arr.sort()
    let result = [arr[0]]

    for (let i=1, len=arr.length; i

This method only has one sorting and one cycle, so the efficiency will be higher than the above method



Vi. new set ()

ES6 adds the data structure of set, which is similar to array, butMembers of set are unique

Based on this feature, it is very suitable for array de duplication

function distinct(a, b) {
    return Array.from(new Set([...a, ...b]))

How long does it take to process 15W data with set?

Meow, meow, meow 57ms?? I’m blind, right??

Then I add a 0 after the length of two arrays, under the data amount of 150W

There is such a high-performance and concise array de duplication method?!


VII. For… Of + object

I’ve only seen this method in some articles, but it’s not very useful in practical work

First create an empty object, and then loop through it with for

utilizeObject’s properties do not repeatThis feature checks whether the array elements are repeated

function distinct(a, b) {
    let arr = a.concat(b)
    let result = []
    let obj = {}

    for (let i of arr) {
        if (!obj[i]) {
            obj[i] = 1

    return result

When I see the processing time of this method is long, I am stupid again

15W data is only 16ms Faster than set()???

Then I tried 150W data


Recommended Today

The way of nonlinear optimization

Mathematical knowledge 1、 Nonlinear functionLinear function is another name of a function of first degree, then nonlinear function means that the function image is not a function of a straight line.Nonlinear functions include exponential function, power function, logarithmic function, polynomial function and so on. 2、 Taylor expansion1. Taylor formula:Taylor’s formula is to add a_ The […]