The function of FTP upload and download file by shell script

Time:2019-11-13

Some time ago, we need to extract some information verification data from our platform and upload it to the customer’s FTP server so that they can compare and verify relevant information. Because there are only four hosts containing these information, the strategy is to gather the four generated files on one host, and then upload the files to the target FTP server on this host.

1. Establish a trust relationship between host a and the other three hosts to facilitate remote copying of files

#To generate the local authentication secret key of host a, you can choose to generate the RSA or DSA type secret key. Here, choose RSA
[[email protected] ~]#ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa >/dev/null 2>&1
#Among the authorized keys files of the other three hosts, only three hosts can be copied. If there are many hosts, you can use expect script to execute in batches
[[email protected] ~]#ssh-copy-id -i ~/.ssh/id_rsa.pub "-p 22 [email protected]"
[[email protected] ~]#ssh-copy-id -i ~/.ssh/id_rsa.pub "-p 22 [email protected]"
[[email protected] ~]#ssh-copy-id -i ~/.ssh/id_rsa.pub "-p 22 [email protected]"
#When the root password of B, C and D is entered, enter the password and enter. The following prompt indicates that the trust relationship between the local machine and the target machine is established successfully
Now try logging into the machine, with "ssh '-p 22 [email protected]'", and check in:
 .ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.

2. Establish the script to upload the file to the target FTP server

The customer’s requirement is to upload the data of the previous day in the early morning every day. The data files we generate are of the type of “host name” and “result. CSV” on the same day.

#The following shows the content of the script
[[email protected] getfile] cat upload_csv.sh
#!/bin/bash
#This script is used to upload the generated data file to the client FTP server regularly every day
#Specify the home directory of the uploaded file
SRCDIR=/tmp/test_jr/getfile/files/
#Specify the directory to upload the peer FTP server
DESDIR=/JRAQ_FILE/
#Specify the user name and password of the peer FTP server
USER=finftp
PASSWD="[email protected]"
#Specify the IP address of the target FTP server
IP=3.3.3.3
#Specify the target FTP server port, which is generally the default
PORT=21
#Specify the date of the file to be uploaded
targetDay=`date -d "-1 days" +"%Y%m%d"`
#Get the file to upload
cd $SRCDIR
for host in 192.168.1.B 192.168.1.C 192.168.1.D
do
  scp [email protected]$host:$SRCDIR/$targetDay_*_Result.csv ./
done
#Determine whether the document is obtained correctly
[ $? -eq 0 ] || echo "Copy romote files failed, pls check." >>$SRCDIR/upload_file.log
#Upload files to FTP server
ftp -ivn <<EOF
  open $IP $PORT
  user $USER $PASSWD
  binary
  cd $DESDIR
  lcd $SRCDIR
  put ${targetDay}_*_Result.csv
  quit
EOF
#Determine whether the file is uploaded successfully
[ $? -eq 0 ] && echo "Upload $targetDay's files to romote FTP server successful." >>$SRCDIR/upload_file.log || echo "Upload files failed, pls check." >>$SRCDIR/upload_file.log

3. Possible transcoding

Because the CSV file can be opened by Excel, but when excel opens the utf-8-encoded file, all Chinese names will display garbled code. At this time, transcoding is needed. For example, if UTF-8 format is converted to GBK encoding, Excel can be opened perfectly.

[[email protected] ~]iconv -futf8 -tgbk -c -o file2.csv file1.csv
#File1 is the file to be transcoded
#File2 is the file name after transcoding
#-F -- from code
#-T -- to code into that format
#-O -- filename of output output
#-C ignore the illegal characters in the output and add this parameter to prevent some illegal strings in the file. If iconv does not add this parameter, it will automatically terminate transcoding after encountering illegal strings, even if there are normal untranslated characters.

summary

The above is the shell script introduced by Xiaobian to realize FTP upload and download file function. I hope it can help you. If you have any questions, please leave me a message, and Xiaobian will reply to you in time. Thank you very much for your support of the developepaer website!
If you think this article is helpful to you, welcome to reprint, please indicate the source, thank you!