Prompt Argument list too long solution for batch operation of Linux

Time:2019-8-12

In the first case:

In practice, if there are too many files or folders in a directory, Argument list too long will be prompted when executing the “*” command, such as rm-rf/tmp/* command.
Solution:

 


ls | xargs ­n 10 rm ­rf ls

SSH executes the above command and outputs all filenames (separated by spaces) xargs, which is the output of ls, with 10 as a group (separated by spaces), as a parameter of RM rf, that is to say, all filenames are 10 as a group and deleted by RM rf.

In the second case:

Rm, cp, MV are commonly used file processing instructions under unix. When we need to delete a large number of log files, this message will appear if there are too many files [bin/rm: Argument list too long]

The solutions are as follows:
For example, delete / tmp /*.log
Then execute [ls/tmp/*.log | xargs rm-f] so that Lo ^^, other cp, MV are the same!

The third case:

Today, we need to delete all the files in the / tmp directory. The number of files is quite large.


ls -lt /tmp | wc -l  
385412  

Userm *After that, the system prompted an errorArgument list too long
The reason is that under linux, when trying to pass too many parameters to a system command (ls *; CP *; RM *; cat *; etc.), there will be an Argument list too long error.

The solutions are as follows:
Use find-exec to traverse and then perform deletion.


sudo find /tmp -type f -exec rm {} \;  

When deleting a large number of files in linux, the error of using RM directly will appear: – bash: / bin / rm: parameter list is too long.

This can be combined with the find command.

Example:
1. Replace rm*-rf with:
Find. -name “*” | xargs Rm-rf’*’is fine.

2. RM test*-rf should be changed to:
find . -name “test*” | xargs rm -rf “test*”

The MV Times parameter list is too long.
for i in *.m;do mv $i ${i%.m};done
So turn to Google and the process of exploration is omitted. Let’s just say the solution.
ls dir1 | xargs -t -I {} mv {} dir2/{}
Here, a pair of braces is used in the example given in the original text. Later, I saw the use of parameters. In fact, braces can be replaced by any string, such as:
ls dir1 | xargs -t -I asdf mv asdf dir2/asdf
The effect is exactly the same as the bracketed version, but it looks a little less serious.
It should be noted that the second parameter of xargs above is the capital i, the letter read “love”, not the lowercase L. As for the meaning of parameters, I forgot.
 

Linux reported “too long list of command parameters” and failed to move more than 30,000 files at a time with the MV command, similar to the original command: “mv $(ls dir1) dir2”. The central idea of the error prompt is, “You have too many TM parameters.”
According to LZ, you can probably do this: find / dir1 / maxdepth 1 | xargs – I MV {}/ dir2/
If the parameters are too long, tar is easy to use.
tar -C /dir1/ -cf – . | tar -xf – -C /dir2/

So turn to Google and the process of exploration is omitted. Let’s just say the solution.
  ls dir1 | xargs -t -I {} mv {} dir2/{}
Here, a pair of braces is used in the example given in the original text. Later, I saw the usage of parameters. In fact, braces can be replaced by arbitrary strings, such as:
  ls dir1 | xargs -t -I asdf mv asdf dir2/asdf
The effect is exactly the same as that in brackets, except that it looks a little less serious.
It should be noted that the second parameter of xargs above is the capital i, the letter pronounced “love”, not the lowercase L. As for the meaning of parameters, I forgot.


Four other methods
As a Linux user/system administrator, sometimes you will encounter the following error prompts:

[[email protected] foo]$ mv * ../foo2
bash: /bin/mv: Argument list too long

The “Argument list too long” parameter list error often occurs when the user provides too many parameters in a simple command line, and often occurs in ls*, cp*, rm*, etc.
According to the causes of the problem, the following four methods are provided, which can be chosen according to their own circumstances.
Method 1: Manually divide the file group into smaller combinations
e.g 1:

[[email protected] foo]$ mv [a-l]* ../foo2
[[email protected] foo]$ mv [m-z]* ../foo2

This is the most basic method, but simply to make the number of parameters meet the requirements. This method has limited scope of application. It only applies to the file list with fairly uniform name distribution. This is also a solution that can be considered by primary users, but it requires a lot of repetitive commands and observation and guess of file name distribution.

Method 2: Use the find command
e.g 2:

[[email protected] foo]$ find $foo -type f -name ‘*’ -exec mv {}$foo2/. \;
Method 2 outputs the list of files to the MV command through the find command, so that it can process one at a time, which completely avoids the existence of excessive parameters. In addition, different parameters can specify matching patterns such as timestamp, permission, inode and so on, besides name.
The disadvantage of method 2 is that it is time-consuming.

Method 3: Create shell functions
e.g 3.1:

function huge_mv ()
{whileread line1; do
mv foo/$line1 ../foo2
done
}
ls -1 foo/ | huge_mv
Writing a shell function does not involve some degree of complexity. This method is more flexible than methods 1 and 2.

Let’s expand on Example 3.1:
e.g 3.2:
 


function huge_mv ()
{whileread line1; do
md5sum foo/$line1 >> ~/md5sums
ls -l foo/$line1 >> ~/backup_list
mv foo/$line1 ../foo2
done
}
ls -1 foo/ | huge_mv

Compared with 3.1, 3.2 generates md-checked hash files and name backups of files, which conforms to the philosophy of leaving a way for oneself.
In addition, according to their own needs, unlimited expansion functions.

Method 4: Ultimate solution, recompile the kernel
Be cautious before using this solution first, because it involves modifying the source code of the kernel. In production environment, it is necessary to consider and test it well.
In addition, this method is fundamentally solved, once and for all.
This is also one of the benefits of open source
First, find the include/linux/binfmts.h file in the kernel source code and search for the following fields:
/*
* MAX_ARG_PAGES defines the number of pages allocated for arguments
* and envelope for the new program. 32 should suffice, this gives
* a maximum env+arg of 128kB w/4KB pages!
*/
#define MAX_ARG_PAGES 32
Modifying the value of MAX_ARG_PAGES to 64 or higher can solve the problem of parameter constraints perfectly.
Then recompile and enable the new kernel.