An article that teaches you to write 90% shell scripts

Time:2020-8-14

Shell script?

Before we talk about shell scripts, let’s talk about what shells are.

Shell is the shell, which means the shell of the operating system. We can operate and control the operating system through shell commands. For example, the shell commands in Linux include ls, CD, PWD and so on. In conclusion, shell is a command interpreter, which starts, pauses, stops the running of programs or controls the computer by accepting shell commands from users.

Shell is an application program, it connects the user and the Linux kernel, so that users can use the Linux kernel more efficiently, safely and cheaply, which is the essence of shell.

Shell itself is not a part of the kernel, it is just an application written on the basis of the kernel.

So what is a shell script?

Shell script is an execution file composed of shell commands. Some commands are integrated into a file to process business logic. The script can be run without compiling. It runs through an interpreter, so it is relatively slow.

Shell script is the most important use and combination of shell commands, and then use some language features supported by shell script to complete the desired function.

Notes

Comments begin with “ා” and are ignored by the compiler

  • Single line comment:#
  • Multiline comments: < < EOF EOF or: < < ! (: < < identifies the start of a multiline comment and specifies an identifier as the start and end flag)

variable

Variable type

When running a shell, there are three variables at the same time:

  • Local variables: local variables are defined in scripts or commands and are only valid in the current shell instance. Programs launched by other shells cannot access local variables.
  • Environment variables: all programs, including shell launched programs, can access environment variables. Some programs need environment variables to ensure their normal operation. If necessary, shell scripts can also define environment variables.
  • Shell variables: shell variables are special variables set by shell programs. Some shell variables are environment variables and some are local variables. These variables ensure the normal operation of the shell

Variable operation

  • Create a common variable: name = “test” (= no spaces on both sides)
  • Create a local variable that can only be used in the function body: local name = “test” (variables decorated with local cannot be accessed outside the function body, and local can only be used inside the function body)
  • Use variables: echo $name or echo ${name} (curly bracket version is recommended)
  • Variable reassignment: name = New_ Test “
  • Read only variable: name = only_ Read “- > readonly name (the variable identified by readonly cannot be modified)
  • To delete a variable: unset name; (after deletion, it is not accessible, and the read-only variable cannot be deleted)

String variable

1) Single quotation mark

  • Single quotation mark variable var = Test ‘can only be output as is, and the variable is invalid
  • A single quotation mark cannot appear in a single quotation mark, nor can it be escaped

2) Double quotation marks

  • Double quotation mark variable var = my name is ${name} “, the variable is valid
  • An escape character may appear

3) Concatenated string

  • There are no characters like +, in the middle
  • Name = “this is” “my name”; name = “this is my name”; name = “this” is “my name” is equivalent
  • Name =’This is’ ‘my Nam’; name =’This is my name ‘; name =’ this’ is’ my name ‘is equivalent

4) Get string length

  • Use “ා” to get the length in ${}
  • name=“test”;
  • Echo ${name}; output 4

5) Extract substring

  • 1: 4 take four characters from the second
  • :: 4 start with the first character and cut back 4 characters
  • name=“this is my name”;
  • echo ${ name:1 : 4} ා output is I
  • Echo ${Name:: 4} output this

array

Bash only supports one-dimensional arrays, not multidimensional arrays

  • Define array: array_ Name = (Li Wang Xiang Zhang) (bracket as boundary, space as separation)
  • Define the elements of an array separately: array_ para[0]=“w”; array_ The subscript can also be defined as “Para S]”
  • Assigning array elements: array_ name[0]=“zhao”;
  • Get array elements:
    • array_name[0]=“li”
    • array_name[3]=“zhang”
    • echo ${array_ Name [0]} output “Li”
    • echo ${array_ Name [1]} output
    • echo ${array_ Name [3]} output “Zhang”
    • echo ${array_ Name [@]} ා output “Li Zhang”, output all elements of the array, and no subscript of the elements is omitted
  • Get the number of elements: ${array_ Name [@]} or ${array_ name[*]}
  • Gets the length of a single element: ${array_ name[1]}

Parameter transfer

  • Get parameter value:
    • $0: fixed, represents the file name of the execution
    • $1: represents the first parameter passed in
    • $n: represents the nth parameter passed in
  • $ා: number of parameters
  • $*: displays all parameters passed to the script in a single string. If “$*” is enclosed by “”, “1 $2 Output all parameters in the form of $n “
  • [email protected]: same as $*, but with quotation marks when used, and returns each parameter in quotation marks.
  • $$: the current process number of the script running
  • $! : the ID of the last process running in the background
  • $?: displays the exit status of the last command. Any other value of 0 indicates no error.
  • The difference between $* and [email protected]
    • Same point: all references all parameters.
    • The difference: only in double quotation marks. Suppose that three parameters 1, 2, 3 are written when the script is running, then “*” is equivalent to “1 2 3” (one parameter is passed), and “@” is equivalent to “1”, “2”, “3” (three parameters are passed).

operator

arithmetic operation

  • +, -, *, \: the multiplication sign must be preceded by an escape to perform multiplication
  • Addition operation
    • Val = ` expr 2 + 2 ‘(use Linux command expr for auxiliary operation)
    • Val = $[2 + 2] (4 spaces are not necessary, different from conditional judgment)
    • val=$((2+2))

Number relation operator

Relational operators only support numbers, not strings, unless the value of the string is a number.

Let’s assume that variable a is 10 and variable B is 20

  • -EQ: checks whether two numbers are equal and returns true. [$a – EQ $b] returns false.
  • -NE: check whether two numbers are not equal, and return true if they are not equal. [$a – Ne $b] returns true.
  • -GT: detects whether the number on the left is greater than that on the right, and returns true if it is. [$a – GT $b] returns false.
  • -LT: detects whether the number on the left is less than that on the right, and if so, returns true. [$a – LT $b] returns true.
  • -Ge: detects whether the number on the left is greater than or equal to that on the right. If so, returns true. [$a – Ge $b] returns false.
  • -Le: detects whether the number on the left is less than or equal to that on the right, and returns true if it is. [$a – Le $b] returns true.

String operator

The following table lists the common string operators, assuming that variable a is “ABC” and variable B is “EFG”:

  • =: checks whether two strings are equal and returns true. [$a = $b] returns false.
  • ! =: checks whether two strings are equal. If not, returns true. [$a! = $b] returns true.
  • -z: Detects whether the string length is 0 and returns true for 0. [- Z $a] returns false.
  • -n: Check whether the string length is 0. If not, return true. [- n “$a”] returns true.
  • $: checks whether the string is empty. If not, returns true. [$a] returns true.

Boolean operator

The following table lists the common Boolean operators, assuming that variable a is 10 and variable B is 20:

  • !: non operation. If the expression is true, it will return false, otherwise it will return true. [! False] returns true.
  • -o: Or operation. If an expression is true, it returns true. [$a – LT 20 – o $B – GT 100] returns true.
  • -a: And operation, return true if both expressions are true. [$a – LT 20 – a $B – GT 100] returns false.

Logical operator

The following describes the shell’s logical operator, assuming that variable a is 10 and variable B is 20:

  • &&: the logical and [[$a – LT 100 – & $B – GT 100]] returns false
  • ||: the logical or [[[$a – LT 100 | $B – GT 100]] returns true

File operator

  • -B file: detects whether the file is a block device file. If so, returns true. [- B $file] returns false.
  • -C file: checks whether the file is a character device file. If so, returns true. [- C $file] returns false.
  • -D file: checks whether the file is a directory. If so, returns true. [- D $file] returns false.
  • -F file: detects whether the file is a normal file (neither a directory nor a device file). If so, it returns true. [- f $file] returns true.
  • -G file: detects whether the sgid bit is set in the file. If so, returns true. [- G $file] returns false.
  • -K file: detects whether the file has set sticky bit. If so, it returns true. [- K $file] returns false.
  • -P file: detects whether the file is a named pipeline, and returns true if it is. [- p $file] returns false.
  • -U file: detects whether the suid bit is set in the file. If so, returns true. [- U $file] returns false.
  • -R file: checks whether the file is readable. If so, returns true. [- R $file] returns true.
  • -W file: checks whether the file is writable. If so, returns true. [- W $file] returns true.
  • -X file: checks whether the file is executable, and returns true if it is. [- x $file] returns true.
  • -S file: check whether the file is empty (whether the file size is greater than 0). If not, return true. [- S $file] returns true.
  • -E file: detects whether the file (including the directory) exists. If so, returns true. [- E $file] returns true.

Execution related

Command replacement

Command replacement is almost the same as variable replacement. It is used to reorganize the command line. First complete the command line in quotation marks, and then replace the result to form a new command line.
Execute command:

  • `Ls / etc ‘: back quotation marks (supported by all UNIX Systems)
  • $(LS / etc): $+ () (not supported by some UNIX Systems)

When multiple nesting is used, it is executed from the inside out

For file in / S / etc or for file in $(LS / etc) loops
`Dirname $0 ‘gets the directory where the script file is located
Path = $(CD ` dirname $0 ‘; PWD): get the current directory of the script, execute the CD command to reach the directory, use PWD to get the path and assign it to the path variable

Arithmetic operation

  • $[]: addition, subtraction, multiplication and division, no need to add spaces
  • $(()): addition, subtraction, multiplication, division, etc., without adding spaces

Logical judgment

  • []: spaces must be added next to brackets and operators (can be used, not recommended)
  • [[]]: spaces must be added next to brackets and operators (recommended for string validation)
  • (()): spaces must be added next to brackets and operators (recommended for numeric validation)
  • [[]] and (()) are enhanced versions of the mathematical comparison expression and string expression of [], respectively.
  • Use [[…] ]]Conditions determine structure, not […] ]To prevent many logical errors in scripts. For example, &, |, < and > operators can normally exist in the [[]] conditional judgment structure, but if it appears in the structure of [], it will report an error. For example, you can directly use if [[$a! = 1 – & $a! = 2]]. If double brackets are not applicable, it is if [$a – Ne 1] & – [$a! = 2] or if [$a – Ne 1 – a $a! = 2].

Add pattern matching effect in [[]];
(()) there is no need to escape the size of the expression to the symbol. In addition to using the standard mathematical operators, the following symbols are added

output

echo

It is only used for the output of string. It has good portability without using printf as the output. It is recommended to use printf

printf

Printf does not automatically add line breaks like echo, we can manually add
No braces, separated directly by spaces

Format:printf format-string [arguments...] Where (format string: format control string, arguments: parameter list)

Case study:Printf "% - 10s% - 8s% - 4.2f / N" Guo Jingnan 66.1234

%S% C% d% F are format substitutes

  • d: Decimal decimal integer corresponding position parameter must be decimal integer, otherwise error is reported!
  • s: String string corresponding position parameter must be string or character type, otherwise an error is reported
  • c: Char character corresponding position parameter must be string or character type, otherwise error will be reported
  • f: Float floating point corresponding position parameter must be numeric type, otherwise error will be reported

%-10s: refers to a width of 10 characters (- left alignment, none means right alignment). Any character will be displayed in the character width of 10 characters. If it is insufficient, it will be filled with spaces automatically. If it is more than 10 characters, all contents will be displayed.

%-4.2f: it means that the format is decimal and the width is 4 characters, in which. 2 means to reserve 2 decimal places.

Escape character:

  • \a: Warning characters, usually ASCII bel characters
  • \b: Back off
  • \c: Suppresses (does not display) any ending newline characters in the output (valid only in parameter strings controlled by the% B format indicator), and any characters left in the parameters, any subsequent parameters, and any characters left in the format string are ignored
  • \f: Form feed
  • \n: Line break
  • \r: Carriage return
  • \t: Horizontal tab
  • \v: Vertical tab
  • \: a literal backslash character
  • \DDD: a character representing an octal value from 1 to 3 digits. Valid only in format strings
  • \0ddd: an octal value character from 1 to 3 digits

Process control

Unlike Java, PHP and other languages, the process control of SH cannot be empty, that is, there is no statement in the braces of if or else

if else


if
if condition
then
 command1 
 command2
 ...
 commandN 
fi

if else


if condition
then
 command1 
 command2
 ...
 commandN
else
 command
fi

if else-if else


if condition1
then
 command1
elif condition2 
then 
 command2
else
 commandN
fi

for


for var in item1 item2 ... itemN
do
 command1
 command2
 ...
 commandN
done

while

while condition


while condition
do
 command
done

While infinite loop


while :
do
 command
done

until

The until loop executes a series of commands until the condition is true.
The until loop is just the opposite of the while loop.


until condition
do
 command
done

case

The shell case statement is a multiple choice statement. A case statement can be used to match a value with a pattern, and if the match is successful, the matching command is executed.
A case needs an ESAC (that is, the reverse of case) as the end marker. Each case branch uses a right parenthesis, and a two semicolon indicates a break, where “;;” is not a jump out of the loop, but does not match the following pattern
The format of the case statement is as follows:

Case value in
 Mode 1)
 command1
 command2
 ...
 commandN
 ;;
 Mode 2)
 command1
 command2
 ...
 commandN
 ;;
esac

Jump out of the loop

  • Break: jump out of the total cycle
  • Continue: jump out of the current loop and continue with the next loop

Defining functions

It can be defined with function fun() or directly with fun() without any parameters.

Function definition


[ function ] funname()
{
 action;
 [return int;]
}

Parameter transfer

  • Calling function: Fun_ name 2 3 4
  • Function: the same as the shell function, or add {}
funWithParam(){
 Echo "the first parameter is $1!"
 Echo "the second parameter is $2!"
 Echo "the tenth parameter is $10!"
 Echo "the tenth parameter is !"
 Echo "the eleventh parameter is !"
 Echo "the total number of parameters is $"
 Echo "outputs all parameters $*!" as a string}
funWithParam 1 2 3 4 5 6 7 8 9 34 73
Echo $

Function return value

  • The word return can or may not exist
  • Return can only be return [0-255]. The return here can be used as the status of function execution. This is the return value obtained through $
  • If no return is added, the return value of the function execution status of the last statement is default. If the last statement is executed successfully, $? Is 0, otherwise it is not 0

Use function return value (janusgraph diagram database official startup service script fragment)

  • The number returned by return is only the return value of the function execution status, that is, the value obtained by $? Next
  • For bin = / ABS_ Path ‘` statement, which gets a string composed of all echo and printf outputs in the function body
abs_path() {
 SOURCE="${BASH_SOURCE[0]}"
 while [ -h "$SOURCE" ]; do
  DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
  SOURCE="$(readlink "$SOURCE")"
  [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE"
 done
 echo "test" 
 echo "$( cd -P "$( dirname "$SOURCE" )" && pwd )" 
 #The two echo outputs of this function are combined into a string as the values of the bin below
}

 
BIN=`abs_ Path 'ා bin assignment function returns a value. If there is no return, all echo and printf outputs in the function are combined into a string and passed into bin
Path = ${bin} / nodetool ා can be used directly

I / O redirection

In general, each UNIX / Linux command runs with three files open:

  • Standard input file (stdin): the file descriptor of stdin is 0, and UNIX programs read data from stdin by default.
  • Standard output file (stdout): the file descriptor of stdout is 1, and UNIX programs output data to stdout by default.
  • Standard error file (stderr): the file descriptor of stderr is 2, and UNIX program will write error information to stderr stream.

By default, command > file redirects stdout to file and command < file redirects stdin to file.
If you want to execute a command, but you don’t want the output to be displayed on the screen, you can redirect the output to / dev / null:

input redirection

  • bash.sh < file: redirects the input of the script to file, and the file provides the parameters

Output redirection

  • bash.sh >File: redirect the output data of the script to the file, and overlay the data
  • bash.sh >>File: redirect the output data of the script to the file and append the data
  • Command > > file 2 > & 1: merge stdout and stderr to file

Read external input

Command: read Arg (script reads external input and assigns values to variables)
When the shell script is executed to the above command, stop the script execution and wait for the external input, assign the external input to the Arg variable, and continue to execute the script

Document reference

After referencing other files, you can use their variables, functions, and so on, which is equivalent to including the referenced file into the current file

There are two ways:

  • . file_path\file_name
  • source file_path\file_name

Color identification


printf "3[32m SUCCESS: yay 3[0m\n";
printf "3[33m WARNING: hmm 3[0m\n";
printf "3[31m ERROR: fubar 3[0m\n";

Output results:

Line break in long sentences

In the shell, you can use “\” to wrap a statement
Using “\” to wrap lines is still executed as a line by statement during script execution, which is different from entering directly

Note: add a space before. \Line feed without spaces.


 /mysql/bin/mysql \
 -h test_host -P 000 \
 -u test_user -ptest_password ;

Shell operation MySQL

The following example is to log in to MySQL, select the operation database, and then import the data

/mysql/mysql/bin/mysql \
 -h test_host -P 000 \
 -u test_user -ptest_password \
 -e"use test_ database; source data_ Fail; "ා - e represents the execution of SQL statements

-U user name
-P user password
-H server IP address
-D connected database
-N does not output column information
-B uses the tab key instead of the separator
-SQL statement executed by e

Exit script

Command:exit

Use different error codes when you exit the script, so you can judge what error has occurred based on the error code.

In most shell scripts, exit 0 indicates successful execution and Exit 1 indicates an error.
One to one mapping of errors and error codes helps script debugging.

Command:Set - E or set + e

Set – e means to start from the current position. If any error occurs, exit will be triggered. Instead, set + e means that the script continues to execute regardless of any errors.

If the script is stateful (each subsequent step depends on the previous step), use set – e to exit the script as soon as an error occurs.
If all commands are required to be executed (rarely), use set + E.

Shell script debugging

Check for syntax errors – N:


bash -n script_name.sh

Use the following command to execute and debug the shell script – X:


bash -x script_name.sh

Debug count_ odd_ number.sh Procedure case:

#!/usr/bin.env bash

#Used to calculate the sum of odd numbers in an array
# @author liyangyang
# @time 2019/09/17

sum=0
for num in 1 2 3 4;do
 re=${num}%2
 if (( ${re} == 1 ));then
  sum=$[${sum}+${num}]
 fi
done
echo ${sum}

First check for grammatical errors


bash -n count_odd_number.sh

There is no output, indicating that there is no error. Start the actual debugging:


bash -x count_odd_number.sh

The debugging results are as follows:


+ sum=0
+ for num in 1 2 3 4
+ re=1%2
+ (( 1%2 == 1 ))
+ sum=1
+ for num in 1 2 3 4
+ re=2%2
+ (( 2%2 == 1 ))
+ for num in 1 2 3 4
+ re=3%2
+ (( 3%2 == 1 ))
+ sum=4
+ for num in 1 2 3 4
+ re=4%2
+ (( 4%2 == 1 ))
+ echo 4
4

The output shows each step of the program execution, and the debugging effect is achieved by observing whether the program execution steps meet the expectations
With + is the output of the shell debugger, without + is the output of the program.

Case study:

This is the script of ES (elastic search) to start the service. Can you understand it~


#!/usr/bin/env bash

# CONTROLLING STARTUP:
#
# This script relies on a few environment variables to determine startup
# behavior, those variables are:
#
# ES_PATH_CONF -- Path to config directory
# ES_JAVA_OPTS -- External Java Opts on top of the defaults set
#
# Optionally, exact memory values can be set using the `ES_JAVA_OPTS`. Note that
# the Xms and Xmx lines in the JVM options file must be commented out. Example
# values are "512m", and "10g".
#
# ES_JAVA_OPTS="-Xms8g -Xmx8g" ./bin/elasticsearch

source "`dirname "$0"`"/elasticsearch-env

parse_jvm_options() {
 if [ -f "$1" ]; then
 echo "`grep "^-" "$1" | tr '\n' ' '`"
 fi
}

ES_JVM_OPTIONS="$ES_PATH_CONF"/jvm.options

ES_JAVA_OPTS="`parse_jvm_options "$ES_JVM_OPTIONS"` $ES_JAVA_OPTS"

# manual parsing to find out, if process should be detached
if ! echo $* | grep -E '(^-d |-d$| -d |--daemonize$|--daemonize )' > /dev/null; then
 exec \
 "$JAVA" \
 $ES_JAVA_OPTS \
 -Des.path.home="$ES_HOME" \
 -Des.path.conf="$ES_PATH_CONF" \
 -cp "$ES_CLASSPATH" \
 org.elasticsearch.bootstrap.Elasticsearch \
 "[email protected]"
else
 exec \
 "$JAVA" \
 $ES_JAVA_OPTS \
 -Des.path.home="$ES_HOME" \
 -Des.path.conf="$ES_PATH_CONF" \
 -cp "$ES_CLASSPATH" \
 org.elasticsearch.bootstrap.Elasticsearch \
 "[email protected]" \
 <&- &
 retval=$?
 pid=$!
 [ $retval -eq 0 ] || exit $retval
 if [ ! -z "$ES_STARTUP_SLEEP_TIME" ]; then
 sleep $ES_STARTUP_SLEEP_TIME
 fi
 if ! ps -p $pid > /dev/null ; then
 exit 1
 fi
 exit 0
fi

exit $?

This article about a teaching you to write 90% shell script (Introduction summary) article introduced here, more relevant shell script introduction content, please search the previous articles of developeppaer or continue to browse the related articles below, I hope you can support developeppaer more in the future!