Linux_ Summary of Chapter 15 of Linux command line and shell script programming

Time:2021-10-15

Time: Monday, September 25, 2017
Note: some of the contents of this article are taken from the book “complete collection of Linux command line and shell script programming”, and the copyright belongs to the original author. Summary of Chapter 15 of Linux command line and shell script programming (Third Edition)

Chapter 15: presenting data

Contents of this chapter

Rediscover redirection
Standard inputs and outputs
Report error
Discard error
Discard data
Create log file

15.1 understanding inputs and outputs

Two methods of displaying script output

Displays the output on the display screen
Redirect output to a file

15.1.1 standard document descriptor

The Linux system treats each object as a file, including input and output processes
Linux uses a file descriptor to identify each file object
The file descriptor is a non negative integer that can uniquely identify the file opened in the reply
Each process can have up to nine file descriptors at a time
For special purposes, the bash shell retains the first three file descriptors

Standard file descriptor for Linux

File descriptor: abbreviation: description
0: stdin: standard input
1: Stdout: standard output
2: Stderr: standard error

1.STDIN

Stdin file descriptors represent the standard input of the shell
For the terminal interface, the keyboard is used for standard input
The shell obtains input from the keyboard corresponding to the stdin file descriptor and processes each character as the user enters
When using the input redirection symbol (<), the file is read and the data can be extracted

Use the cat command with the lines in the testfile file file as input

Write testfile file

This is first line.
This is a test line
This is third line.

Execute command

cat < testfile

2.STDOUT

The stdout file descriptor represents the standard output of the shell
On the terminal interface, the standard output is the terminal display
All output from the shell is directed to the standard output, the display
By default, most bash commands direct output to stdout file descriptors
You can also use output redirection to change (>)

Execute command

ls -l > test2
cat test2
who >> test2
cat test2
ls -al badfile > test3
cat test3

When an incorrect command is executed, the shell does not redirect the error message to the file

3.STDERR

The shell processes error messages through stderr file descriptors with special characters
The stderr file descriptor represents the shell's standard error output
Error messages generated when an error occurs in the shell or in programs and scripts running in the shell are sent to this location
By default, error messages are also output to the display output
Stderr does not change with stdout redirection

15.1.2 redirection error

1. Redirect only errors

Stderr file descriptor is 2, and you can choose to redirect only error messages
Place the file descriptor before the redirection symbol
The value must be placed firmly in front of the redirection symbol, otherwise it will not take effect

Execute command

ls -al badfile 2> test4
cat test4

Stdout and stderr can also be used in combination

Execute command

ls -la test badtest test2 2> test5
cat test5

2. Redirect errors and data

If you want to redirect error and normal output, you must use two redirection symbols
You need to put the file descriptor corresponding to the redirected data in front of the symbol
Then point to the output file used to hold the data

Execute command

ls -al test test2 test3 badtest 2> test6 1>test7
cat test6
cat test7

Redirect the output of stdout and stderr to the same output file using the special symbol (& >)

Execute command

ls -al test test2 test3 badtest &>test7
cat test7

15.2 redirecting output in scripts

15.2.1 temporary redirection

Command Demo: echo “this is an error message” > & 2
Command description: manually generate error information and output redirect to stderr

Write test8.sh script

#!/bin/bash
echo "This is an error" >&2
echo "This is normal output"

Execute command

./test8.sh
./test8.sh 2>test9
cat test9

15.2.2 permanent redirection

Use the exec command to tell the shell to redirect a specific file descriptor during script execution

Write test10.sh script

#!/bin/bash
exec 1>testout

echo "This is a test of redirecting all output"
echo "from a script to another file"
echo "without having to redirect every individual line"

Execute command

./test10.sh
cat testout

Stdout can also be redirected during script execution

Write test11.sh script

#!/bin/bash
exec 2>testerror

echo "this is the start of the script"
echo "now redirecting all output to another location"

exec 1>testout

echo "Theis out should go to the testout file"
echo "but this should go to testerror file" >&2

Execute command

./test11.sh
cat testout
cat testerror

15.3 redirecting input in scripts

Command demonstration: exec 0 < testfile
Command description: redirect stdin from the keyboard to another location using the same method as script redirection stdout and stderr

Write test12.sh script

#!/bin/bash
exec 0< testfile
count=1

while read line
do
    echo "Line #$count:$line"
    count=$[ $count + 1 ]
done

15.4 create your own redirection

When redirecting input and output in a script, these three default file descriptors are not limited
There can be up to 9 open file descriptors in the shell
Any of the other 6 file descriptors from 3 to 8 can be assigned to the file

15.4.1 create output file descriptor

Use the exec command to assign a file descriptor to the output

Write test13.sh script

#!/bin/bash

exec 3>test13out

echo "This should display on the monitor"
echo "and this should be stored in the file" >&3
echo "Then this should be back on the monitor"

Execute command

./test13.sh
cat test13out

Command demonstration: exec 3 > > test13out
Command description: use the exec command to append the output to an existing file

15.4.2 redirect file descriptor

Temporarily redirect the output in the script, and then restore the default output settings

Write test14.sh script

#!/bin/bash

exec 3>&1
exec 1>test14out

echo "This should store in the output file"
echo "along with whis line"

exec 1>&3

echo "Now things should be back normal"

Execute command

./test14.sh
cat test14out

15.4.3 create input file descriptor

You can save the stdin file descriptor to another file descriptor, and then restore stdin to its original location after reading the file

Write test15.sh script

#!/bin/bash

exec 6<&0
exec 0< testfile

count=1
while read line
do
    echo "Line #$count:$line"
    count=$[ $count+1 ]
done

exec 0<&6
read -p "Are you done now [Y/N]?" answer
case $answer in
Y|y) echo "Goodbye" ;;
N|n) echo "Sorry,this is the end."
esac

15.4.4 create read / write file descriptor

You can use the same file descriptor to read and write the same file
When reading and writing data to the same file, the shell maintains an internal pointer
Any read or write starts at the last position of the pointer

Write test16.sh script

#!/bin/bash

exec 3<> testfile
read line <&3
echo "Read: $line"
echo "This is a test line" >&3

Execute command

cat testfile
./test16.sh
cat testfile

15.4.5 close file descriptor

Command format: exec 3 >&-
Command description: manually close the file descriptor before the end of the script and redirect it to a special symbol&-

When closing the file descriptor, if the same output file is subsequently opened in the script, the existing file is overwritten

Write test17.sh script

#!/bin/bash

exec 3> test17file
echo "This is a test line of data" >&3
exec 3>&-

cat test17file

exec 3> test17file
echo "This'll be bad" >&3

Execute command

./test17.sh
cat test17file

15.5 list open file descriptors

Use the lsof command to display open file descriptors

Write test18.sh script

#!/bin/bash

exec 3> test18file1
exec 6> test18file2
exec 7< testfile

/usr/bin/lsof -a -p $$ -d0,1,2,6,7

15.6 block command output

Stderr can be redirected to a special file of a null file
On Linux systems, / dev / null is the standard location for null files
Any data redirected to that location will be discarded
Usually used to clear log files

Command demonstration: cat / dev / null > testfile
Command description: clear the data in the testfile file file

15.7 creating temporary files

15.7.1 create local temporary files

Command demonstration: mktemp testing.xxxxxx
Command description: the mktemp command creates a temporary file and automatically replaces the six X’s with six character codes to ensure that the file name is unique in the directory. Multiple temporary files can be created, and each file is unique.

Write test19.sh script

#!/bin/bash

tempfile=$(mktemp test19.XXXXXX)

exec 3>$tempfile

echo "This script writes to temp file $tempfile"

echo "This is the first line" >&3
echo "This is the second line" >&3
echo "This is the last line" >&3

exec 3>&-

echo "Done creating temp file.The contents are:"
cat $tempfile
rm -f $tempfile 2> /dev/null

Execute command

./test19.sh
ls -al test19*

15.7.2 create temporary files in / tmp directory

Use the - t option to force the mktemp command to create the file in the temporary directory of the system
The mktemp command returns the full path of the temporary file

Write the test20.sh script

#!/bin/bash

tempfile=$(mktemp -t tmp.XXXXXX)

echo "This is a test file." > $tempfile
echo "This is the second line of the test." >> $tempfile
echo "The temp file is located at: $tempfile"

cat $tempfile
rm -f $tempfile

15.7.3 create temporary directory

Use the – D option to tell the mktemp command to create a temporary directory

Write test21.sh script

#!/bin/bash

tempdir=$(mktemp -d dir.XXXXXX)
cd $tempdir
tempfile1=$(mktemp temp.XXXXXX)
tempfile2=$(mktemp temp.XXXXXX)
exec 7> $tempfile1
exec 8> $tempfile2

echo "Sending data to directory $tempdir"
echo "This is a test line of data for $tempfile1" >&7
echo "This is a test line of data for $tempfile2" >&8

Execute command

./test21.sh
ls -al
ls -al dir.zKQTCL/
cat dir.zKQTCL/temp.EZBifQ
cat dir.zKQTCL/temp.sO4oa4

15.8 logging messages

Send output to both the display and the log file
Using the tee command, it is equivalent to a T-joint of the pipe
Send the data from stdin to two places at the same time
One is stdout, and the other is the file name specified by the tee command: tee filename

Use the – a option to append the data to the file, and the data will be displayed on the screen and permanently saved in the file

Write test22.sh script

#!/bin/bash

tempfile=test22file

echo "This is the start of the test" | tee $tempfile
echo "This is the second line of the test" | tee -a $tempfile
echo "This is the end of the test" | tee -a $tempfile

Execute command

./test22.sh
cat test22file

15.9 examples

Case description: File redirection is common in scripts that need to read in and output files. Insert the data into the database by reading the data file in. CSV format and outputting SQL insert statement

Write the members.csv text

Blum,Richard,123 Main st.,Chicago,IL,60601
Blum,Barbara,123 Main st.,Chicago,IL,60601
Bresnahan,Christine,456 Oak Ave.,Columbus,OH,43201
Bresnahan,Timothy,456 Oak Ave.,Columbus,OH,43201

Write test23.sh script

#!/bin/bash

outfile='members.sql'
IFS=','

while read lname fname address city state zip in 
do
    cat >> $outfile << EOF
    INSERT INTO members (lname,fname,address,city,state,zip) 
    VALUES('$lname','$fname','$address','$city','$state','$zip');
EOF
done

Execute command

./test23.sh < members.csv
cat members.sql

15.10 summary

The bash shell allows you to create your own file descriptors in scripts. Using mktemp, you can easily create temporary directories and files. The tee command makes it easy to send output to both standard output and log files.