Shell (bash) script programming III: redirection

stayThis oneIn, we introduced some basic knowledge about input-output redirection and pipeline. This article will continue the topic of redirection.
Before we start, let’s talk about the in the shellquote。

quote

Like many programming languages, bash also supports character escape, which is used to change the original meaning of characters and make someMetacharacter(e.g&）Can appear in the command.
There are three types of references in Bash, which are slightly different from each other:
The first is a backslash (\), which is used to escape the next character

[[email protected] temp]# echo $PATH
$PATH
[[email protected] temp]# 

The second is the single quotation mark (”), which prohibits parsing of the contained text.
The third is double quotation marks (“”), which prevent partial parsing, but allow some words（word）Expansion of.
Characters that retain special meaning in double quotation marks include:

$ ` \ !
#Where $(extender: variable extension, mathematical extension, command replacement) and ` (command replacement) maintain their special meaning;
#Backslash in double quotation marks \ keeps its special meaning only when it is followed by the following characters: $` "\! < newline >;
#By default, exclamation point! (historical expansion, described in the next chapter) it is only used in the interactive shell, and historical recording and expansion cannot be carried out in the script.
#As described in Chapter 1, when @ and * are used for position variables and array variables in double quotation marks, the meanings are different:
#Each element is a separate word after the extension of "[email protected]" and "${array [@]}"
#"$*" and "${array [*]}" are extended as a whole

There is also a special reference in Bash: $’string ‘. Where stringstringThe characters escaped by the inner backslash have special meanings and follow ANSI C standard. See for some explanationshere
example:

[[email protected] ~]# echo $'\u4f60\u597d\uff0c\u4e16\u754c\uff01'
Hello, world!
[[email protected] ~]#

redirect

In the following description, if the numbernOmitted, the first redirection operation symbol is<, then this redirection refers toStandard input(file descriptor 0), if the first redirection operation symbol is>, then this redirection refers tostandard output (file descriptor 1).
Following the redirection operatorwordWill be expanded.

1. Input redirection

[n]<word

2. Output redirection

[n]>word

wordThe extended result file will be overwritten by the output of the command (if the file does not exist, it will be created). Through built-in commandssetSetnoclobberThe bash process of option is using the redirection operator>When, subsequent files are not overwritten. Use operator>|You can force overrides.

3. Append output redirection

[n]>>word

4. Redirect standard output and standard error

&>word
>&word

The two ways of writing are the same, equivalent to>word 2>&1。

5. Append redirection standard output and standard error

&>>word

amount to>>word 2>&1

6. Open the file read-write

[n]<>word

Redirection abovewordThe extension result of cannot be more than one, and can only be a file. The order in which multiple redirects appear in a command is important, but it doesn’t matter where a redirect is in the command.

#!/bin/bash
#The order in which multiple redirects occur sometimes affects the results
#Both standard output and standard error are redirected to the file file
ls hello file >file 2>&1
#The standard error is output to the terminal, and the standard output is redirected to the file
ls hello file 2>&1 >file
#It doesn't matter where redirection occurs. The following three commands are equivalent:
head -1 </etc/passwd >>newfile
>>newfile head -1 </etc/passwd
head</etc/passwd>>newfile -1
#View validation
cat newfile

Execution:

[[email protected] ~]# ./test.sh 
Ls: cannot access that directory or file
root:x:0:0:root:/root:/bin/bash
root:x:0:0:root:/root:/bin/bash
root:x:0:0:root:/root:/bin/bash

7、Here Documents

<<[-]word
    here-document
delimiter

HerewordCannot be extended ifwordAny character in isquote(as previously cited),delimiteryeswordRemove the remaining characters after the reference, andhere-documentNone of the words in are interpreted by the shell. IfwordNot quoted,here-documentWords in can experienceVariable extension、Command replacementandMathematical extension(similar to double quotation marks).
If the redirection operator is<<-, then inhere-documentThe beginning tab character in will be deleted.

8、Here Strings

<<<word

herewordThe extended result of is redirected as a string.
Script example:

#!/bin/bash
VAR='hello'
#Here Documents
cat <<EOF >file
#The content of the document will not be used as a comment
When not referenced, variables can be extended within the document:
$VAR world
EOF
cat file
#Here Strings
echo ${parameter:=$[`tr "," "+" <<<"1,2,3"`]}
#Variable temporary scope
IFS=':' read aa bb cc <<<"11:22:33"
echo -e "$aa $bb $IFS $cc"

Execution result:

[[email protected] ~]# ./test.sh   
#The content of the document will not be used as a comment
When not referenced, variables can be extended within the document:
hello world
6
11 22  
 33
[[email protected] ~]#

9. Copy file descriptor

[n] < & word # copy input file descriptor
[n] > & word # copy output file descriptor

therewordThe extended value must be a number to copy this file descriptor ton, ifwordIf the result of the extension is not a file descriptor, a redirection error will occur. IfwordThe value of is-, it means that the file descriptor is closedn。
[n]>&wordHere is a special case: ifnOmitted andwordIf the result is not a number, it indicates redirection of standard error and standard output (as described earlier).

10. Transfer file descriptor

[n] < & Digital - # transfer input file descriptor
[n] > & Digital - # transfer output file descriptor

These two represent moving file descriptorsdigitTo file descriptorn, file descriptor after movedigitClosed.
Since the redirection in Bash is only valid in the current command, the redirection is revoked after the command is executed. You can use built-in commandsexecMake redirection valid throughout the script.
Script example:

#!/bin/bash
#Open input file descriptor 3 and associate the file file
exec 3<file
#First copy the file descriptor to the standard input, and the cat command reads the contents of the file from the standard input
cat <&3
#Close file descriptor 3
exec 3<&-

#Open descriptors 3 and 4 as output and associate files respectively.
exec 3>./stdout
exec 4>./stderr
#Transfer the standard output to descriptor 3 and close the original file descriptor 1.
exec 1>&3-
#Transfer the standard error to descriptor 4 and close the original file descriptor 2.
exec 2>&4-
#The standard output of the command is written to a file/ Stdout, standard error writing to file/ stderr
ls file newfile
#Close both file descriptors
exec 3>&-
#It doesn't matter whether the redirection symbol is > or < when closing
exec 4<&-

#Define remote host and port
host=10.0.1.251
port=80
#Open the file descriptor 5 in read-write mode and associate it to the file (this file represents a TCP link to the remote end)
if ! exec 5<>/dev/tcp/$host/$port
then
    exit 1
fi
#Test link availability
echo -e "GET / http1.1\n" >&5
#Get output
cat <&5
#Close file descriptor
exec 5<&-

Execution result:

[[email protected] ~]# ./test.sh 
#I am the content of the file
<! DOCTYPE html... # The rest is the HTTP response information
...
[[email protected] ~]# 
[[email protected] ~]# cat stderr 
Ls: cannot access newfile: there is no such file or directory
[[email protected] ~]# cat stdout 
file
[[email protected] ~]#

coproc

In the last article, we describedcoprocThe syntax of the command. Here are the use cases:

#!/bin/bash
#Simple command
#Simple command使用不能通过NAME指定协进程的名字
#At this time, the process name is unified as coproc. (it also indicates that there can only be one cooperative process with simple commands at the same time)
coproc cat file
#Coprocess PID
echo $COPROC_PID
#Transfer the output file descriptor of the coprocess to the standard input and use it for the cat command:
cat <&${COPROC[0]}-

#Compound command
#For named coprocesses, subsequent commands must be compound commands
coproc ASYNC while read line
do
    if [ "$line" == "break" ];then
        break
    else
        awk -F: '{print $1}' <<<"$line"
    fi
done
#Pass data to the asynchronous program (SED command appends the string "break" at the bottom of the file)
sed '$abreak' /etc/passwd >&${ASYNC[1]}
#Get output
while read -u ${ASYNC[0]} user_name
do
    echo $user_name
done

Execution result:

[[email protected] ~]# ./test.sh 
28653
The standard output and standard input of the command are respectively connected to the two file descriptors of the current shell through two-way pipes,
Then the file descriptor is assigned to the array elements name [0] and name [1] respectively
root
bin
daemon
...
[[email protected] ~]#

The Conduit

Pipeline is one of the main means of inter process communication. There are two types of Linux pipes:Anonymous Pipe andname pipes。
By control operator|or|&The pipes created when connecting commands areAnonymous Pipe 。Anonymous Pipe It can only be used between processes with kinship.
name pipesCan be used between two unrelated processes, you can use commandsmknodormkfifoTo createname pipes。
We have seen many examples of anonymous pipes. Here is an example of using named pipes to control the number of concurrent processes:

#!/bin/bash
#Number of processes
NUM=10
tmpfile="$$.fifo"
#Generate temporary named pipes
[ -e $tmpfile ] && exit || mkfifo $tmpfile
#Open the file descriptor 5 in a read-write manner and associate it to the named pipe
exec 5<>$tmpfile
#Delete temporary named pipe file
rm $tmpfile
#Writes a specified number of blank lines for read use
while((NUM-->0))
do
    echo
done >&5
 
for IP in `cat ip.list`
do
    #The read command reads one line of input at a time, ensuring that there are 10 following composite commands running at the same time
    read
    {
        #Count the IP address in the log file access Number of log occurrences
        grep -c $IP access.log >>result.txt
        echo
    #After the command is run, a blank line is still written to file descriptor 5
    #The symbol at the end & ensures that this composite command runs in the background
    } >&5 &
done <&5
#The built-in command wait is used to wait for the end of the child process
wait
#Close file descriptor 5
exec 5>&-

Implementation strategy.
Of course, the compound command executed in the for loop here can be replaced by any task that needs to be executed concurrently.