Analysis of time consuming method of multi thread scanning folder

Time:2021-3-20

ad locumJava methods have thread recursion, do not know what method to run timeEncountered an interesting problem, multi-threaded scan folder for runtime. Generally, this kind of time-consuming scanning folder seems to use recursive traversal for timing. It’s the first time to see this kind of folder and thread. Here we try to solve the problem according to the original idea of the owner.

At the beginning, I used static variables to share the total time-consuming, adding the time-consuming of each run method.

package com.brewin.codetuning.test;

import java.io.File;
import java.text.SimpleDateFormat;
import java.util.Date;

public class ThreadDemo02
{
    public static void main(String args[]) {
        File file=new File("D:\AA");
        MyThread mt1 = new MyThread("Thread 1",file);
        mt1.start();
    }
}

class MyThread extends Thread
{
    private String name;
    private File file;
    public static int sum;
    static {
        sum = 0;
    }

    public MyThread(String name,File file) {
        this.name = name;
        this.file=file;
    }

    @Override
    public void run() {
        long startTime =  System.currentTimeMillis();
        File[] files=file.listFiles();
        if(files!=null &&files.length>0) {
            for(File f:files) {
                if(f.isDirectory()) {
                    new MyThread(f.getName(),f).start();
                }
//It's not convenient to print the file name. It's time-consuming to print the last file, so you can comment it out
//                else {
//                    System.out.println(Thread.currentThread().getName()+":"+f);
//                }
            }
        }
//        try {
//            Thread.sleep(5000);
//        }
//        catch (InterruptedException e) {
//            // TODO Auto-generated catch block
//            e.printStackTrace();
//        }
        long endTime =  System.currentTimeMillis();
        long usedTime = (endTime-startTime);
        sum+=usedTime;
        System.out.println(name + End of run, time taken:+usedTime+"Ms, sum is now:  +MyThread.sum+" ms");
    }

}

What does the console print? Let’s take the last part

End of storage running, time consumption:0MS, sum is now:3156ms1MS, sum is now:3157ms1MS, sum is now:3158ms1MS, sum is now:3159 ms

But it didn’t take more than three seconds, and it was over soon.

Here’s what’s added Thread.sleep (5000) is more obvious

XS running end, time consuming:5004MS, sum is now:2611580ms5001MS, sum is now:2606576ms5001MS, sum is now:2616581 ms

Multithreading is a concurrent execution, so this idea is wrong. We can’t calculate their total, but should calculate their first time and last time. Change the way of thinking, initialize a start time at the beginning, then update the last time after each thread ends, and calculate the difference between them.

package com.brewin.codetuning.test;

import java.io.File;

public class ThreadDemo02
{
    public static void main(String args[]) {
        File file=new File("D:\AA");
        MyThread mt1 = new MyThread("Thread 1",file);
        mt1.start();
    }
}

class MyThread extends Thread
{
    private static long startTime;
    private String name;
    private File file;

    static {
        startTime =  System.currentTimeMillis();
    }

    public MyThread(String name,File file) {
        this.name = name;
        this.file=file;
    }

    @Override
    public void run() {
        File[] files=file.listFiles();
        if(files!=null &&files.length>0) {
            for(File f:files) {
                if(f.isDirectory()) {
                    new MyThread(f.getName(),f).start();
                }
//It's not convenient to print the file name. It's time-consuming to print the last file, so you can comment it out
//                }else {
//                    System.out.println(Thread.currentThread().getName()+":"+f);
//                }
            }
        }
        long usedTime = System.currentTimeMillis()-startTime;
        System.out.println(name + End of run, time consumed:+usedTime+" ms");
    }
}

The last section of the console is printed as follows:

IO finished running, time consumed:164ms164ms164ms165ms165ms166ms166ms166ms166ms166ms167ms167ms167ms167ms167ms167ms168ms168ms168ms169ms169ms169ms169ms169ms169ms170ms170ms170ms170ms170ms170 ms
CORBAEnd of running, time consumed:171ms171ms171ms171ms171ms171ms171ms171ms172ms172ms172ms172ms172ms172ms172ms173ms173ms173ms173ms174ms174ms174ms174ms174ms175ms175ms175ms175ms175ms176ms177ms177 ms

When I saw the original problem, the landlord mentioned countdown latch This is not very suitable for this article. The initial size of the countdownlatch counter should be consistent with the number of tasks (independent of the number of threads). Every time a task is executed, the counter will be subtracted by one (countdown). The await() method will block the main thread until the value of the counter is reduced to 0, and then the lock will be released. In this way, it can ensure that all tasks are completed Continue the effect of the next step.

However, in this scenario, new threads are recursively generated, and the number of tasks and threads is required to be equal in this scenario. If the number of tasks cannot be determined, the countdownlatch size cannot be initialized.

If you want to use countdownlatch, you can only add a statistical method to obtain the number of folders, as follows:

package com.brewin.codetuning.test;

import java.io.File;
import java.util.concurrent.CountDownLatch;

public class ThreadDemo02
{
    static int count;
    static {
        count = 0;
    }
    public static void main(String args[]) {
        File file = new File("D:\AA");
        countNumberOfFolders(file);
        long startTime = System.currentTimeMillis();
        MyThread mt1 = new MyThread("Thread 1", file);
        mt1.start();
        try {
            MyThread.latch.await();
        }
        catch (InterruptedException e) {
            e.printStackTrace();
        }
        long usedTime = System.currentTimeMillis() - startTime;
        System.out.println(Time consuming: + usedTime + " ms");

    }

    /**
    public static void countNumberOfFolders(File f) {
        count++;
        File[] files = f.listFiles();
        if (files != null && files.length > 0) {
            for (File file : files) {
                if (file.isDirectory()) {
                    countNumberOfFolders(file);
                }

            }
        }
    }

}

class MyThread extends Thread
{
    private String name;
    private File file;
    static CountDownLatch latch;
    static {
        latch = new CountDownLatch(ThreadDemo02.count);
    }

    public MyThread(String name, File file) {
        this.name = name;
        this.file = file;
    }

    @Override
    public void run() {
        File[] files = file.listFiles();
        if (files != null && files.length > 0) {
            for (File f : files) {
                if (f.isDirectory()) {
                    new MyThread(f.getName(), f).start();
                }
//It's not convenient to print the file name. It's time-consuming to print the last file, so you can comment it out
//                }else {
//                    System.out.println(Thread.currentThread().getName()+":"+f);
//                }
            }
        }
        System.out.println(name + "End of run");
        latch.countDown();
    }
}

The results are as follows

End of HTTP run350 ms

You can see that the next step is really to wait until all threads are finished.

Conclusion as above

This work adoptsCC agreementThe author of this article must be noted

Recommended Today

What is “hybrid cloud”?

In this paper, we define the concept of “hybrid cloud”, explain four different cloud deployment models of hybrid cloud, and deeply analyze the industrial trend of hybrid cloud through a series of data and charts. 01 introduction Hybrid cloud is a computing environment that integrates multiple platforms and data centers. Generally speaking, hybrid cloud is […]