Restart failed git pull / push operation with expect

Time:2021-5-3

Raising of the problem

Recently, when I use GitHub to upload and download project code, I often get stuck for a long time. Sometimes I hit git push on the command line and then go to the toilet. As a result, when I came back, I found that the push had already failed and had to submit it again. If there is a tool that can restart the failed git push until it succeeds, it will be good.

What is expect

Before introducing the use of expect to restart git operation, let’s briefly explain this command. In fact, it is not a trendy thing. It has existed for a long time, so that some systems do not carry this command by default. You need to install it manually

$ sudo yum install expect
$ expect -vexpect version 5.44.1.15

 

In short, expect is to complete some tasks that need to interact with users. For example, when Telnet, FTP and SSH remotely log on to the machine, these commands will require users to enter user name, password and other related information, which cannot be completed by shell script. This is because these commands are read from the control terminal rather than the standard input, so it is impossible to redirect the information to the standard input in advance to realize automatic operation. Expect is used to solve such problems. Here is an example of SSH login using expect:

 #!/usr/bin/expect -f
 set ipaddr "localhost"
 set passwd "iforgot"
 
 spawn ssh [email protected]$ipaddr
 expect {
 "yes/no" { send "yes\r"; exp_continue}
 "password:" { send "$passwd\r" }
 }
 
 expect "]# "
 send "touch a.txt\r" 
 send "exit\r"
 expect eof
 exit

 

There are several key actions in the expect script:

  • Splash: start the command to be executed;
  • Expect: parse the command output and enter the sub control block according to the following matching statement;
  • Send: send information to the command, which is equivalent to the information read by the command from the control terminal;
  • Interaction: continue the interaction between command and control terminal. At this time, the user can normally input information to the command (not shown in this example).
  • ……

Well, after getting familiar with the usage of expect, some people may have doubts. This git pull / push operation does not involve passwords. What can I do with it? This is different from person to person. Some people use it because of the password, but I only see its expect action.

Failure log and normal log

Take git pull as an example. When it fails, its output is as follows:

$ git pull
ssh: connect to host github.com port 22: Connection refused
fatal: The remote end hung up unexpectedly

 

When successful, its output is as follows:

$ git pull 
remote: Enumerating objects: 38, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 36 (delta 24), reused 24 (delta 12), pack-reused 0
Unpacking objects: 100% (36/36), done.
From github.com:goodpaperman/apue
   86b80d3..e0cc835  master     -> origin/master
Updating 386fd43..e0cc835
Fast-forward
 apue.c |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

 

If there is no updated content to pull, its output is as follows:

$ git pull
Already up-to-date.

 

For git push, the same is true

$ git push
Connection reset by 13.229.188.59 port 22
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

 

When successful:

$ git push
Counting objects: 16, done.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (10/10), 1.05 KiB, done.
Total 10 (delta 7), reused 0 (delta 0)
remote: Resolving deltas: 100% (7/7), completed with 6 local objects.
To [email protected]:goodpaperman/apue.git
   87748c7..08e3a1e  master -> master

 

It’s up to date:

$ git push
Everything up-to-date

 

So it’s natural to come up with a solution: splash git pull / push until we expect the output “XXX up-to-date.”

Restart failed operation

Using the above ideas, write the following expect script

pull.exp

 #! /usr/bin/expect -f
 set timeout 30
 for {set i 0} {$i<=10} {incr i} {
     puts "start pulling git $i"
     spawn git pull 
     expect "Already up-to-date." { puts "pulling ok"; exit }
 }

 

This script uses the expect loop. Try 10 times at most. If the pull still fails, it may be caused by other reasons. At this time, exit the loop.

push.exp

 #! /usr/bin/expect -f
 set timeout 30
 for {set i 0} {$i<=10} {incr i} {
     puts "start pushing git $i"
     spawn git push 
     expect "Everything up-to-date" { puts "pushing ok"; exit }
 }

 

Similar to pull, except that the feature string of expect is different. Here, “everything up-to-date” is used instead of “already up-to-date.”

But there is a disadvantage in this way, that is, if the script is placed in a directory, I need to specify the corresponding path before I can call it. Is there any way to call this script anytime and anywhere like a command?

Using alias

Enter alias on your system to view the currently opened command alias.

$ alias
alias cd='cdls'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias vi='vim'
alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'

 

You can see that the CD command on my machine is redefined as CdLS. What’s the magic thing? Open ~ /. Bashrc and you can see its definition:

 $ cat ~/.bashrc
 # .bashrc
 
 # Source global definitions
 if [ -f /etc/bashrc ]; then
     . /etc/bashrc
 fi
 
 # User specific aliases and functions
 cdls() {  
     cd "";  
     ls;  
 }  
 
 alias cd='cdls' 

 

In fact, it is a shell function, in which the CD and LS commands are combined to switch to a new directory and list the contents of the directory. Here, it is not difficult to implement a gpull / gpush

 git_pull() { 
     expect -c 'set timeout 30; for {set i 0} {$i<=10} {incr i} { puts "start pulling git $i"; spawn git pull; expect "Already up-to-date." { puts "pulling ok"; exit } }'
 }
 
 git_push() {
     expect -c 'set timeout 30; for {set i 0} {$i<=10} {incr i} { puts "start pushing git $i"; spawn git push; expect "Everything up-to-date" { puts "pushing ok"; exit } }'
 }
 
 alias gpull='git_pull'
 alias gpush='git_push'

 

Here, use the – C option of expect to enter all script statements in one line, separated by semicolons. Add the above in ~ /. Bashrc, and then execute the following command to reload the bashrc file

$ . ~/.bashrc

 

You can make the newly added gpull and gpush aliases take effect! Of course, after doing so, it will only take effect for the current user, and other users will not be able to use it after logging in. You can define this alias in / etc / bashrc, so that all users can use it. The following is the implementation effect:

$gpullstart pulling git 0spawn git pullremote: Enumerating objects: 5, done.remote: Counting objects: 100% (5/5), done.remote: Compressing objects: 100% (1/1), done.remote: Total 3 (delta 2), reused 3 (delta 2), pack-reused 0Unpacking objects: 100% (3/3), done.From github.com:goodpaperman/apue   65d83a6..8560ad0  master     -> origin/masterUpdating 65d83a6..8560ad0Fast-forward apue.c |   11 +++++------ 1 files changed, 5 insertions(+), 6 deletions(-)start pulling git 1spawn git pullAlready up-to-date.pulling ok
$gpushstart pushing git 0spawn git pushCounting objects: 5, done.Compressing objects: 100% (3/3), done.Writing objects: 100% (3/3), 316 bytes, done.Total 3 (delta 2), reused 0 (delta 0)remote: Resolving deltas: 100% (2/2), completed with 2 local objects.To [email protected]:goodpaperman/apue.git   8560ad0..0d3c3c7  master -> masterstart pushing git 1spawn git pushEverything up-to-datepushing ok

 

From the above output, we can see a problem, that is, the first time we actually pulled / pushed successfully, but because we did not get the output we wanted, the operation was restarted again until it output XXX up-to-date. It can be seen that our expect is not very intelligent. I’ll leave the improvements to you

epilogue

In fact, we only use the script syntax of expect, but we don’t use its deeper part: terminal control. In fact, similar to expect, there are commands such as script and tee, which open a pseudo terminal pair internally to achieve the redirection of terminal input / output. For the content related to the terminal, please refer to a mistake about the pseudo terminal in my previous article: [APUE]

reference resources

[1] . Linux expect command

[2] . expect usage

[3] . expect syntax basis: usage examples of while, for loop and if statements

[4] . expect (splash) automated git submission and SCP copy — CentOS (Linux)

Recommended Today

Large scale distributed storage system: Principle Analysis and architecture practice.pdf

Focus on “Java back end technology stack” Reply to “interview” for full interview information Distributed storage system, which stores data in multiple independent devices. Traditional network storage system uses centralized storage server to store all data. Storage server becomes the bottleneck of system performance and the focus of reliability and security, which can not meet […]