Coding agile practical combat series lesson 4: building continuous integration Devops pipeline from scratch


Building continuous integration Devops pipeline from scratchSenior agile coach, senior lecturer of extreme programming college and invited agile consultant by codingLi XiaoboThe teacher will show how to write Jenkins file to build Ci / CD pipeline based on coding, including unit test, end-to-end test, code specification check, product library and docker deployment.


Hello everyone, the main content of today’s course is how to build Devops pipeline from scratch and its significance in R & D work. Finally, it summarizes the relationship between Devops pipeline practice and agile development.


At first, continuous integration was proposed in extreme programming, then continuous delivery was proposed by ThoughtWorks, and then the concept of Devops was proposed. Why do we do this? I think the reason isAs time goes on, productivity will continue to decline.At the beginning of the team, the efficiency is very high. From 0 to 1, the function goes online quickly, but in the later stage, the speed will become lower and lower, until the development stagnates finally. There are three difficulties in the later stage of system developmentFirst, it’s hard to add new features.With the accumulation of system functions, there will be a lot of repeated code and unreasonable design, resulting in a lot of places to change when adding a new feature, and the cost of modification is very high;Second, it is difficult to repair defects.Users may report some defects or bugs, but it is difficult to locate defects because the code is too messy and complex;Third, it leads to new defects.When making a new feature, it’s easy to beat a hamster. When you press a bug, three bugs will pop up, which will bring difficulties to the team.


The reasons for these phenomenaFirst of all, in the process of requirements evolution, some code will continue to corruptMethods get longer and longer, classes get bigger and bigger, and there’s a lot of code duplication. Although some teams will develop code specifications, in practical applications, they may be “lying down” in word or PDF, so it is difficult to adhere to the inspection and Implementation for a long time.


The second is that the architecture will also rot.At the beginning of a project, the architect usually designs the architecture according to the business, how many modules and objects are divided into several layers, which layer can be adjusted, which layer can not be adjusted, and how to depend on each other. However, in the process of continuous evolution, the architecture often becomes messy.


William Edwards Deming proposedQuality built inThe concept of,That is, the product quality has been embedded in the construction process, and it is not found by the later detection.Later detection can not improve the quality of code and product. The earlier the problem is found, the lower the cost of repair. Therefore, we hope that every time we submit code to the code base, we can get feedback immediately: is the modification good or bad, whether it increases repeated code, whether it reduces test coverage, whether it destroys some functions, and so on. With such a yardstick, you can always ensure that every time you submit code, you add value to the product, not destroy it. In this context, the concept of pipeline is introduced.

Pipeline is a metaphor, which means to connect all aspects of software development. I think assembly line plays three roles in R & D management processPainstaking temporary workers, selfless guardians and fast and accurate operators.

The assembly line is a laborious temporary worker.Today’s build pipeline can be created on demand. For example, coding, with its continuous integration function, it is impossible to allocate fixed computing and storage resources to each user. If you want to be able to grow linearly, the strategy should be to create it on demand when the user needs to build it, and then destroy it when it is used up. From this point of view, the assembly line is like a temporary worker. In addition, the pipeline can also take the trouble to do repetitive things. In particular, continuous integration teams have to submit code many times a day. It is painful to manually check each submission, but the machine can repeat the mechanical movement.

The assembly line is an impartial guardian. The first is the code specification.Code specifications of many development teams are “living” in word or PDF. Even if there are senior coaches or technical leaders who occasionally do some code reviews, the execution is far from enough. However, by means of integration into the pipeline, such as specifying how many lines a method cannot exceed, how many lines a class cannot exceed, how much code repetition rate cannot exceed, code width and naming, etc., to effectively ensure the implementation of code specifications;The second is test coverage.In the daily development work, there will be a variety of tests, such as unit test, component test, interface test, integration test, end-to-end test. Every time you submit the code, you need to check whether the test coverage has decreased. However, many teams are missing in this area. They just build, package, deploy and other actions on the pipeline. Instead of running test coverage, they rely on large-scale manual testing to ensure quality, resulting in the inability to quickly iterate;The third is architecture constraints.The code is generally layered. There are constraints on what files should be placed in each layer, which files can be transferred to which files. A set of automatic mechanism is needed to ensure the landing;Secondly, there is security inspection.For example, web development will introduce some third-party open source code. These open source codes often have security defects, which need to be checked every time new content is introduced.

The assembly line is a fast and accurate operator.The more complex the system is, the more environments there will be, including development joint debugging environment, test environment, pre release environment, etc., and there will be multiple instances in the formal environment. The URL of database access in each environment is different, and the environment of accessing other services is also different. How to ensure that there is no error in the operation process? We can rely on pipelining to standardize, process, and automate these actions. Each time the code is submitted, we check the specifications, print different packages for different environments, and continuously deploy them to different environments.

So how to build an assembly line? There’s a way calledPipeline as codePut the pipeline into the code. I think the advantage of this is versioning. The problem with the traditional build method is that there is no record of the operation and the review cannot be forced. When a server fails and a server is changed, the original pipeline configuration needs to be operated again. If the pipeline is changed into code, it can be traced and repeatedly created to improve production efficiency. In addition, there are two main ways to construct pipeline, one is called Ming type, the other is called instruction type. Declarative is to specify the links and steps, what kind of things are needed. The specified form needs to write a lot of conditional judgments, which is logical, and the maintenance cost will be higher. Therefore, declarative formula is a widely used method at present.


A typical pipeline should containFour key links. The first is to build.Both the front-end and back-end code need to be compiled. The front-end, such as HTML, JS, CSS, and so on, may also use some templates, which need to be compiled to convert them into formats that the browser can support;The second is inspection.After compiling, it is necessary to check whether the code conforms to the specification, whether there are many duplicate codes, and how much repetition rate is exceeded;The third is testing.Testing is very important and will affect the team’s confidence in the product launch. Here is a test pyramid Theory: the bottom layer is a large number of unit tests, the middle is component testing or interface testing, and the top is end-to-end testing. Because a lot of logic is in various if else branches, unit tests can cover these branches, so the upper level test does not need to cover the logic that has been covered by the lower level. The value of the upper level test is to integrate the code and use it from the user’s point of view to see if it can work properly. The test focus of the upper layer is different from that of the lower layer, and the problems to be solved are also different;The fourth is deployment.Finally, the image needs to be built and pushed to the production library to update the server. After this series of things are completed, the pipeline instance will be destroyed.


Finally, we summarize why Devops practice and its relationship with agile development are derived

First, our most important goal is to satisfy our customers by consistently delivering valuable software early.I think it’s in line with agility. According to the agile principle, working software is higher than detailed documentation. Customers prefer to see available software, not just document description. However, due to the opacity of the development process, customers like to do micro management, which also makes the development team very passive. There is a value in agile called respect, which needs to be won by the team itself. Through the establishment of a process, improve the transparency of the development process, so as to establish trust between the customer and the team. Another thing that fits with agile principles is that responding to change is better than following the plan. This is not a slogan, but a kind of ability, which includes: project management ability, demand management ability, configuration management ability and quality assurance ability. After these four abilities are built, the team can have the ability to respond to changes. Only by combining the core capabilities such as continuous integration, automated testing, and automatic deployment with such practices as code specification, codereview, TDD, etc., can we really enhance the strength of the development team, instead of just importing scrum in and holding planning meetings and standing meetings.

Second, working software is the primary measure of progress.It’s not just a daily station meeting or a weekly email to tell customers how much work has been completed this week. It must be that after the deployment is completed, the function that can be seen and experienced can be regarded as the real progress.

Third, the persistent pursuit of technical excellence and good design will enhance the agility ability.After the development team has the quality control such as code specification inspection and automatic testing, they have the foundation to continuously optimize and get a sustainable and rapid iteration rate. Practice will change with the change of technology, and the team’s ability will also change continuously. However, whether the team can keep agile or not depends on whether the values and principles are consistent.


Click to watch the full video