Shell script combined with git to realize incremental project deployment

Time:2020-10-2

Small hub reading:

There are so many script codes. If it’s not operation and maintenance, you may not be able to write it. Ha ha. It’s good for ordinary people to use Jenkins in full, but it’s troublesome when there are too many codes. Moreover, Jenkins should be used with scripts, which is more in line with the business.


Author: catalpaflat

https://juejin.im/post/5cf0ed…

Application deployment is a process that must be faced with in development, testing and online. Especially with the emergence of microservice architecture, operation and maintenance deployment is gradually separated from the single deployment, and becomes more and more complex.

However, apart from multi language, multi environment, cluster and distributed deployment. Just talk about incremental deployment and full deployment

1. Incremental and full deployment

Deployment: in addition to the initial deployment of the project, the ideal situation is as follows:What is changed in the new version will be updated

1.1 incremental deployment

1.1.1 introduction to incremental deployment

Incremental deployment generally refers to extracting the increment (including code, executable file or configuration) between the current version and the version to be deployed in each deployment process, and only the incremental part is updated during the deployment process.

1.1.2 common deployment process

  1. Use code management tools (SVN, GIT, etc.) to extract the increment between the two versions, and combine the incremental changes in other aspects.
  2. According to the incremental part, the specific deployment method is formulated, the deployment script is written, and the incremental deployment package (including confused code, etc.) is prepared.
  3. Distribute and deploy the incremental deployment package to the target environment that has run the previous version, and complete the version upgrade of the system.

1.1.3 advantages of incremental deployment

  1. Fast deployment. Only the incremental part is updated each time to shorten the deployment time
  2. Reduce the amount of change. To reduce the change range of the whole system, some configuration contents do not need to be updated every time
  3. Improve security. Due to the increment of each detachment, it can avoid the leakage of all code

1.1.4 disadvantages of incremental deployment

  1. Incremental deployment will reduce deployment efficiency if there are other external deployment environment dependencies

    Incremental deployment is not like

  2. When there are many deployment environments, the requirement for repeatability is high
  3. Incremental deployment becomes unfriendly to rollback operations

1.2 how to choose increment or total quantity

Most of the existing automated deployment are full deployment, but full deployment also has some disadvantages. However, there are some strategies for filtering:

  • All configurations and materials (deployment packages, external configuration files, etc.) deployed in advance can improve efficiency and speed
  • Using gray publishing or load balancing methods to reduce the impact of full deployment on application availability

    For most state independent deployment units (applications, modules, microservices, etc.) in modern systems,In general, full deployment should be the best choice。 The state dependent deployment unit (database, etc.) is still suitable for incremental deployment logic.

2. Enter the theme

Some of the incremental and full deployment scenarios were described earlier. Next, we will talk about how to use shell script and git log for incremental deployment

2.1 premise environment

  • Java project
  • Maven
  • Git as code repository

2.2 shell script

Shell novice, not perfect writing, light spray.

2.2.1 module of the whole shell script

  • Git environment preparation
  • Maven compiles the project to be built
  • Create incremental deployment folder
  • Retrieve project target directory
  • Git diff is used to retrieve the difference between two commit, and then the corresponding file is copied to the “incremental folder”

2.2.2 git environment preparation

#Git environment

if [[ ! -d ".git" ]]; then
    ECHO error: please init  Git Repository
    exit 1;
fi

if [[ ! -z ${branch} ]]; then
    git checkout ${branch}
fi

#Get default commit hash
if [[ -z "$begin_hash" ]] && [[ -z "$end_hash" ]] ; then
    for p in $(git log --pretty=oneline -2) ; do
        if [[ ${#p} -eq 40 ]]; then
            if [[ -z ${begin_hash} ]]; then
                begin_hash=${p}
            else
                end_hash=${p}
                break
            fi
        fi
    done
fi

is_begin_has=false

#Is it the latest commit
if [[ $(git log --pretty=oneline -1) == *${begin_hash}* ]]; then
    is_begin_has=true
fi

#If it is not the latest branch commit, it will be rolled back to the original version. At that time, the original Maven configuration does not support compile or there may be build failure (such as using local warehouse / private warehouse, etc.)
if [[ ${is_begin_has} = false ]]; then
    project_path=$(pwd)
    project_name=${project_path##*/}
    cd ..
    build_project_name=${project_name}_build_temp_project
    if [[ ! -d ${build_project_name} ]]; then
        mkdir ${build_project_name}
    fi
    \cp -rf  ${project_name}/.  ${build_project_name}
    cd ${build_project_name}
    git reset --hard ${begin_hash}
fi
Copy code
2.2.2.1 check whether git warehouse code is used
if [[ ! -d ".git" ]]; then
    ECHO error: please init  Git Repository
    exit 1;
fi
Copy code
2.2.2.2 check whether it is necessary to switch branches
if [[ ! -z ${branch} ]]; then
    git checkout ${branch}
fi
2.2.2.3 do you need to set the commit value of the default build

If you do not add — begin_ Hash = and — end_ Hash = to assign a value, the latest two commit are used for incremental deployment by default.

Get the hash of the last two commit through git log — pretty = oneline – 2

#Get default commit hash
if [[ -z "$begin_hash" ]] && [[ -z "$end_hash" ]] ; then
    for p in $(git log --pretty=oneline -2) ; do
        if [[ ${#p} -eq 40 ]]; then
            if [[ -z ${begin_hash} ]]; then
                begin_hash=${p}
            else
                end_hash=${p}
                break
            fi
        fi
    done
fi
Copy code
2.2.2.4 verify the begin of parameter transfer_ Whether the hash value is the latest commit hash of the current branch

If it is not the latest commit hash of the current branch, you need to roll back to the corresponding commit to build and compile the project

if [[ $(git log --pretty=oneline -1) == *${begin_hash}* ]]; then
    is_begin_has=true
fi
Copy code
2.2.2.5 if begin_ Hash is not the latest commit hash

Begin_ The value of hash is not the current latest commit hash. You need to roll back to the corresponding commit for build compilation.

  1. Copy the existing project to a new directory environment
  2. Reset the project to the new directory environment to build the project
if [[ ${is_begin_has} = false ]]; then
    project_path=$(pwd)
    project_name=${project_path##*/}
    cd ..
    build_project_name=${project_name}_build_temp_project
    if [[ ! -d ${build_project_name} ]]; then
        mkdir ${build_project_name}
    fi
    \cp -rf  ${project_name}/.  ${build_project_name}
    cd ${build_project_name}
    git reset --hard ${begin_hash}
fi
Copy code

2.2.3 Maven compiles the project to be built

Compile the project to generate the corresponding class file and related configuration file

mvn clean compile -q -DskipTest
Copy code

If the local warehouse is used in the historical version but not configured in maven, it can be reconfigured and introduced through scope and SystemPath, such as:

<dependency>
    <groupId>cn.catalpaflat</groupId>
    <artifactId>core</artifactId>
    <version>1.0.0</version>
    <scope>system</scope>
    <systemPath>${project.basedir}/lib/core-1.0.jar</systemPath>
</dependency>
Copy code

2.2.4 create incremental deployment folder

In order to prevent the incremental folder from being deleted or committed to git repository, it can be unified into a directory and ignored by. Gitignore. You can compare the differences between each incremental deployment

build_path=build-path/
current_date=`date +%Y%m%d%H%m%s`

if [[ ! -d "$build_path$current_date" ]]; then
    mkdir -p ${build_path}${current_date}
else
    rm -rf ${build_path}${current_date}
    mkdir -p ${build_path}${current_date}
fi
Copy code

2.2.5 search project target directory

If the project is a maven project and it is a java project, due to the Maven multi module situation, it is necessary to retrieve the compiled code path under each module for subsequent copy of class and other files.

default_target_paths=()
default_java_file=java

module_index=0
#Retrieve whether the current project is Maven multi module development, retrieve recursively, and set the compiled code location (only Java type is provided temporarily)
obtain_module(){
    for module in ` cat ./pom.xml | grep '<module>' | awk -F '>' '{print $2}' | awk -F '<' '{print $1}' `
    do
        cd ${module}

        if [[ ! -d "/pom.xml" ]]; then
           module_exist=`cat ./pom.xml | grep '<module>' | awk -F '>' '{print $2}' | awk -F '<' '{print $1}'`
           if [[ -z ${module_exist} ]]; then
                if [[ ! -d "/target" ]]; then
                    if [[ -z $1 ]]; then
                        default_target_paths[module_index]=${module}/target/classes
                    else
                        default_target_paths[module_index]=$1/${module}/target/classes
                    fi
                    ((module_index++))
                fi
           else
                 if [[ -z $1 ]]; then
                      obtain_module ${module}
                 else
                      obtain_module $1/${module}
                 fi
           fi
        fi
        cd ..
    done
}

obtain_module
Copy code

2.2.6 retrieve and copy change files to incremental folder

  1. Git diff — name only check the file differences between two commit
  2. And start_ Hash commit compiled code is copied to the increment folder for subsequent packaging and deployment
#Git diff -- name only is used to realize the file difference between two commit, and start_ After the hash code is compiled, copy the difference files to the "incremental folder" for subsequent incremental deployment

for file_path in $(git diff --name-only ${begin_hash} ${end_hash}) ; do
    package_path=${file_path%/*}
    file_name=${file_path##*/}
    file_type=${file_name##*.}
    #Check whether the file folder is created
    if [[ ${package_path} != *.* ]]; then
          if [[ ! -d "./${build_path}${current_date}/$package_path" ]] ; then
                mkdir -p ./${build_path}${current_date}/${package_path}
           fi
    fi
    #Java or not
    if [[ ${file_type} = ${default_java_file} ]]; then
        module_path=${package_path##*java}
        file_class_name=${file_name%.*}
        module_type=${package_path%%/*}
        #Check which Maven module path is in
        for default_target_path in ${default_target_paths[@]}; do
            target_module_path=$(echo ${default_target_path} | awk -F '/target/' '{print $1}')
            file_target_module_path=$(echo ${package_path} | awk -F '/src/' '{print $1}')
            file_target_package_path=$(echo ${package_path} | awk -F '/src/main/java/' '{print $2}')
            default_module_type=${default_target_path%%/*}
            if [[ ${target_module_path} = ${file_target_module_path} ]]; then
                #Check the target directory of the corresponding Maven module and perform CP operation
                cp -afx ${default_target_path}/${file_target_package_path}/${file_class_name}* ./${build_path}${current_date}/${package_path}
            fi
        done

    else
    #Non java files, directly copy the files to the corresponding directory
        if [[ ${package_path} != *.* ]]; then
            if [[ ! -d "./${build_path}${current_date}/$package_path" ]] ; then
                mkdir -p ./${build_path}${current_date}/${package_path}
            fi
        else
             package_path=${package_path%/*}
        fi

        cp -afx ${file_path} ./${build_path}${current_date}/${package_path}

    fi
done

Source code through train!!! : https://github.com/CatalpaFla…

So far, the preliminary version 1.0 can be used


(end)

Recommended reading:

Great, spring boot + Vue front and back separation complete introductory tutorial!

Share a set of springboot development blog system source code, as well as the complete development document! Speed save!

GitHub’s top 100 Java open source projects, covering a variety of technology stacks!

The latest interview questions and answers in 2020