Detailed explanation of sed usage of shell script

Time:2022-9-23

The role of sed

Sed is the abbreviation of Stream Editor (stream editor), referred to as stream editor; used to process files

sed is to read the file content line by line and process it as required, and output the processed result to the screen

  • First, sed reads a line in the file and saves it in a temporary buffer (also known as pattern space)
  • Then process the line in the temporary buffer as needed, and send the line to the screen when done

Summarize:

  • Since sed stores each line in a temporary buffer and edits this copy, it will not directly modify the original file
  • Sed is mainly used to automatically edit one or more files; simplify repeated operations on files, filter and convert files

Introduction to the use of sed

There are two common syntax formats for sed, one is called command line mode and the other is called script mode.

command line format

sed [options] ‘processing action’ filename

  • Common options
Options illustrate Remark
-e Make multiple (multiple) edits Multiple match parameters can be edited
-n cancel default output Do not print pattern spaces automatically
-r Using extended regular expressions  
-i Edit in place will modify the source file
-f Specify the filename of the sed script  
  • Common processing actions

All of the following actions must be enclosed in single quotes

action illustrate Remark
‘p’ Print  
‘i’ insert content before the specified line Similar to capital O in vim  
‘a’ Insert content after the specified line Similar to lowercase o in vim
‘c’ Replace all contents of the specified line very commonly used
‘d’ delete the specified line  

for example

document preparation

vim a.txt  

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
298374837483
172.16.0.254
10.1.1.1

Add, delete, modify and check files

print file content

sed '' a.txt does nothing to the file
sed -n 'p' a.txt prints each line and suppresses the default output
sed -n '1p' a.txt prints line 1
sed -n '2p' a.txt prints line 2
sed -n '1,5p' a.txt print lines 1 to 5
sed -n '$p' a.txt print the last line

add file content

iInsert above address location

aInsert below

sed '$a99999' a.txt Add content below the last line of the file
sed 'a99999' a.txt file adds content below each line
sed '5a99999' a.txt file, add content below line 5
sed '$i99999' a.txt file adds content to the last line on the last line
sed 'i99999' a.txt file adds content to the previous line of each line
sed '6i99999' a.txt file, line 6, add content to the previous line
sed '/^adm/ihello' insert content on the previous line of the line starting with adm

cReplace the specified entire line

sed '5chello world' a.txt replace the content of line 5 of the file
sed 'chello world' a.txt replace the entire contents of the file
sed '1,5chello world' a.txt replace the contents of files 1 to 5 with hello world
sed '/^daemon/c888888' a.txt replace lines starting with user01

delete file contents

sed '1d' a.txt delete file line 1
sed '1,5d' a.txt delete lines 1 to 5 of the file
sed '$d' a.txt delete the last line of the file

Search and replace files

Syntax: sed option ‘s/searched content/replaced content/action’ file to process
Among them, s means search search; slash ==/ means delimiter, which can be defined by yourself; the action is generally to print p and globally replace g== (by default, only the first matching content of each line is replaced)

sed -n 's/root/ROOT/p' 1.txt 
sed -n 's/root/ROOT/gp' 1.txt 
sed -n 's/^#//gp' 1.txt 
sed -n '[email protected]/sbin/[email protected]@gp' a.txt
sed -n 's/\/sbin\/nologin/itcast/gp' a.txt
sed -n '10s#/sbin/nologin#itcast#p' a.txt 
uucp:x:10:14:uucp:/var/spool/uucp:itcast

sed -n '[email protected]/sbin/[email protected]@p' 2.txt 
Note: The delimiter in search and replace can be specified by yourself

sed -n '1,5s/^/#/p' a.txt Comment out lines 1-5 of the file
#root:x:0:0:root:/root:/bin/bash
#bin:x:1:1:bin:/bin:/sbin/nologin
#daemon:x:2:2:daemon:/sbin:/sbin/nologin
#adm:x:3:4:adm:/var/adm:/sbin/nologin
#lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

other commands

Order explain Remark
r Read content from another file  
w Save content as  
& Save the search string for reference in the replacement string the same as
= print line number  
apply command to all but selected lines, after line number ‘1,5!’
q quit  

for example:

r reads input lines from a file
w write selected lines to file

sed '3r /etc/hosts' 2.txt 
sed '$r /etc/hosts' 2.txt
sed '/root/w a.txt' 2.txt 
sed '/[0-9]{4}/w a.txt' 2.txt
sed  -r '/([0-9]{1,3}\.){3}[0-9]{1,3}/w b.txt' 2.txt

! Applies the command to all lines except the selected line, after the line number
sed -n '1!p' 1.txt 
sed -n '4p' 1.txt 
sed -n '4!p' 1.txt 
cat -n 1.txt 
sed -n '1,17p' 1.txt 
sed -n '1,17!p' 1.txt 

& saves the search string to quote \(\) in the replacement string

sed -n '/root/p' a.txt 
root:x:0:0:root:/root:/bin/bash
sed -n 's/root/#&/p' a.txt 
#root:x:0:0:root:/root:/bin/bash

sed -n 's/^root/#&/p' passwd comment out lines starting with root
sed -n -r 's/^root|^stu/#&/p' /etc/passwd Comment out lines starting with root or stu
sed -n '1,5s/^[az].*/#&/p' passwd Comment out lines starting with any lowercase letter in lines 1 to 5
sed -n '1,5s/^/#/p' /etc/passwd comment 1~5 lines
or
sed -n '1,5s/^/#/p' passwd start with empty plus #
sed -n '1,5s/^#//p' passwd replace the ones starting with # with empty

sed -n '/^root/p' 1.txt 
sed -n 's/^root/#&/p' 1.txt 
sed -n 's/\(^root\)/#/p' 1.txt 
sed -nr '/^root|^stu/p' 1.txt 
sed -nr 's/^root|^stu/#&/p' 1.txt 


= print line number
sed -n '/bash$/=' passwd print the line number of the line ending in bash
sed -ne '/root/=' -ne '/root/p' passwd 
sed -n '/nologin$/=;/nologin$/p' 1.txt
sed -ne '/nologin$/=' -ne '/nologin$/p' 1.txt

q quit
sed '5q' 1.txt
sed '/mail/q' 1.txt
sed -r '/^yunwei|^mail/q' 1.txt

sed -n '/bash$/p;10q' 1.txt
ROOT:x:0:0:root:/root:/bin/bash


Integrated use:
sed -n '1,5s/^/#&/p' 1.txt 
#root:x:0:0:root:/root:/bin/bash
#bin:x:1:1:bin:/bin:/sbin/nologin
#daemon:x:2:2:daemon:/sbin:/sbin/nologin
#adm:x:3:4:adm:/var/adm:/sbin/nologin
#lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

sed -n '1,5s/\(^\)/#/p' 1.txt 
#root:x:0:0:root:/root:/bin/bash
#bin:x:1:1:bin:/bin:/sbin/nologin
#daemon:x:2:2:daemon:/sbin:/sbin/nologin
#adm:x:3:4:adm:/var/adm:/sbin/nologin
#lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

other options

-e multiple edits
-r extended regular
-i modify the original file

sed -ne '/root/p' 1.txt -ne '/root/='
root:x:0:0:root:/root:/bin/bash
1
sed -ne '/root/=' -ne '/root/p' 1.txt 
1
root:x:0:0:root:/root:/bin/bash

Insert "hello world" before line 5 in the 1.txt file; insert "cat cat cat" below line 8 in the 1.txt file

sed -e '5ihello world' -e '8a cat cat cat' 1.txt -e '5=;8='

sed -n '1,5p' 1.txt
sed -ne '1p' -ne '5p' 1.txt
sed -ne '1p;5p' 1.txt

Filter the vsftpd.conf file starting with # and blank lines:

grep -Ev '^#|^$' /etc/vsftpd/vsftpd.conf
sed -e '/^#/d' -e '/^$/d' /etc/vsftpd/vsftpd.conf
[sed '/^#/d;/^$/d' /etc/vsftpd/vsftpd.conf
sed -r '/^#|^$/d' /etc/vsftpd/vsftpd.conf

Filter valid lines in the smb.conf file:
sed -e '/^#/d' -e '/^;/d' -e '/^$/d' -e '/^\t$/d' -e '/^\t#/d' smb.conf
sed -r '/^(#|$|;|\t#|\t$)/d' smb.conf 

sed -e '/^#/d' -e '/^;/d' -e '/^$/d' -e '/^\t$/d' -e '/^\t#/' smb.conf


grep '^[^a-z]' 1.txt

sed -n '/^[^a-z]/p' 1.txt

Filter out IP addresses in the file:
grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}' 1.txt 
192.168.0.254
sed -nr '/([0-9]{1,3}\.){3}[0-9]{1,3}/p' 1.txt 
192.168.0.254

grep -o -E '([0-9]{1,3}\.){3}[0-9]{1,3}' 2.txt 
10.1.1.1
10.1.1.255
255.255.255.0

sed -nr '/([0-9]{1,3}\.){3}[0-9]{1,3}/p' 2.txt
10.1.1.1
10.1.1.255
255.255.255.0

Filter out the IP, subnet mask, broadcast address in the ifcfg-eth0 file
grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' ifcfg-eth0 
10.1.1.1
255.255.255.0
10.1.1.254

sed -nr '/([0-9]{1,3}\.){3}[0-9]{1,3}/p' ifcfg-eth0|cut -d'=' -f2
10.1.1.1
255.255.255.0
10.1.1.254

sed -nr '/([0-9]{1,3}\.){3}[0-9]{1,3}/p' ifcfg-eth0|sed -n 's/[A-Z=]//gp'
10.1.1.1
255.255.255.0
10.1.1.254

ifconfig eth0|sed -n '2p'|sed -n 's/[:a-Z]//gp'|sed -n 's/ /\n/gp'|sed '/^$/d'
10.1.1.1
10.1.1.255
255.255.255.0

ifconfig | sed -nr '/([0-9]{1,3}\.)[0-9]{1,3}/p' | head -1|sed -r 's/([a-z:]|[A-Z/t])//g'|sed 's/ /\n/g'|sed  '/^$/d'

ifconfig eth0|sed -n '2p'|sed -n 's/.*addr:\(.*\) Bcast:\(.*\) Mask:\(.*\)/\n\n/p'
10.1.1.1 
10.1.1.255 
255.255.255.0

The -i option directly modifies the original file
sed -i 's/root/ROOT/;s/stu/STU/' 11.txt
sed -i '17{s/YUNWEI/yunwei/;s#/bin/bash#/sbin/nologin#}' 1.txt
sed -i '1,5s/^/#&/' a.txt

Notice:
-ni do not use together
p command don't use -i anymore

sed combined with regular use

sed option ‘sed command or regular expression or address location’ filename

Addressing is used to decide which lines to edit. The address can be in the form of numbers, regular expressions, or a combination of the two.

If no address is specified, sed will process all lines of the input file.

Regular illustrate Remark
/key/ Query for rows containing keywords sed -n ‘/root/p’ 1.txt
/key1/,/key2/ Matches lines that contain between two keywords sed -n ‘/^adm/,/^mysql/p’ 1.txt
/key/,x Lines starting from the line matching the keyword to line x of the file (including the line where the keyword is located) sed -n ‘/^ftp/,7p’
x,/key/ Lines starting at line x of the file to lines matching the keyword  
x,y! does not contain rows x to y  
/key/! Lines that do not include keywords sed -n ‘/bash$/!p’ 1.txt

Supplementary extension summary

1. Regular expressions must be separated by "/" before and after the specification
For example: sed '/root/d' file
For example: sed '/^root/d' file

2. If the match is an extended regular expression, you need to use the -r option to extend sed
grep -E
sed -r
+ ? () {n,m} | \d

Notice:         
If special characters (^$.*/[]) appear in regular expressions, they need to be escaped with leading "\"
eg:sed '/$foo/p' file


3. Comma separator
For example: sed '5,7d' file delete lines 5 to 7
For example: sed '/root/,/ftp/d' file    
Delete all lines from the first matching string "root" to the first matching string "ftp" This line is not found and the loop is executed

4. Combination method
For example: sed '1,/foo/d' file delete all lines from the first line to the first matching the string "foo"
For example: sed '/foo/,+4d' file deletes lines starting with the matching string "foo" and ending four lines after it
For example: sed '/foo/,~3d' file deletes lines (in the file) that are multiples of 3 starting from the matching string "foo"
For example: sed '1~5d' file delete one line every five lines from the first line
For example: sed -nr '/foo|bar/p' file shows lines with the configuration string "foo" or "bar"
For example: sed -n '/foo/,/bar/p' file show lines matching from foo to bar
For example: sed '1~2d' file delete odd lines
For example: sed '0-2d' file delete even lines sed '1~2!d' file

5. Special circumstances
For example: sed '$d' file delete the last line
For example: sed '1d' file delete the first line

6. Others:
sed 's/.//' a.txt delete the first character in each line
sed 's/.//2' a.txt removes the second character in each line
sed 's/.//N' a.txt starts from the Nth line in the file and deletes the Nth character in each line (N>2)
sed 's/.$//' a.txt delete the last character in each line

Summarize

So far, this article about the detailed usage of sed in shell scripts is introduced here. For more information on the usage of sed in shell scripts, please search for previous articles of developpaer or continue to browse the related articles below. I hope you will support developpaer more in the future!

Recommended Today

Detailed explanation of the usefulness of having in msSQL

Table of contents where to filter data The meaning of having Daily use of having having check having statistical screening: a collection that achieves a certain indicator Must be used after group by Sorting out the specific execution order Give a simple example to understand the execution order here Summarize where to filter data To […]