The main article shared and introduced how Xita team used Devops technology system to make development and testing, operation and maintenance efficient cooperation, how to use CI / CD to carry out automatic test and software release of blockchain software, and design CI / CD What problems should be paid attention to in the pipeline; what are the difficulties in the automatic testing of blockchain software; how to improve the running efficiency of automated testing; what kind of test report is a good report; how to share the practical experience of test framework optimization.
Devops process – integration of requirements, development, testing, operation and maintenance and monitoring
Agile development and Devops – how should development and testing, operation and maintenance work together
Why automated testing?
Introduction of CI / CD assembly line
How to design test pipeline
Practical experience: optimizing assembly line
Practical experience: optimizing test framework
In order to improve the efficiency of software R & D, ensure the quality of software and service operation and maintenance efficiently, Xita team practices agile development, uses Devops for automatic operation and maintenance, and implements continuous automated testing. Before introducing how to do it, let’s talk about what xiaxita team has done, so that we can understand why the team is doing this.
Xita technology team developed a blockchain software CITA in 2016, and carried out the project open source in 2017 https://github.com/cryptape/cita
CITA is a high-performance blockchain core independently developed by Xita team from scratch, with the characteristics of micro service architecture, horizontal expansion, pluggable components and high performance. The blockchain service is simple and can be regarded as a distributed ledger synchronization system. In order to simplify the use of application developers, Xita team has developed various development and debugging tools and components, such as:
The diagram of the relationship between the CITA kernel and various components:
These tools are open source:
How to do it
The Sita team uses the scrum framework in agile development to manage team goals and progress. The goal of agile development is to deliver working software through iterative development. The Sita team will use a large version as a milestone, define the overall goals and time points that must be completed, and then divide them into several sprints with shorter work cycles, each of which has its own stage goal to complete.
The Xita team uses story point to evaluate the completion time of each task, and uses burn out chart to feedback task completion progress, so as to control the rhythm and progress of each sprint. The following figure shows the burnout chart of each sprint in the sub product development process of Xita team
In terms of specific task management, Xita team uses Kanban tool to track the task status of each member, as shown in the following figure:
Kanban is a graphical display tool. Because human beings are born with visual information easier to understand, it can directly reflect the status of each task, and it is also an intuitive presentation of workload, so that each member of the development team can clearly know themselves and understand the progress of other people’s work.
Software development is not only finished after the code is written, but also needs to be tested to ensure the integrity, deploy the tested software to the server, and set up monitoring to ensure that the service runs normally, and can collect feedback to the product and the development team to continuously improve the software function and quality. This is a complete and available service development to the online process. The whole process is as follows As shown in the figure above, one link connects with another.
How to practice Devops
As shown in the figure above, Devops is not only a matter for the operation and maintenance team, but also a process of integrated implementation of development, operation and maintenance, and testing personnel of various positions. The Xita team needs to standardize the process, so that each functional member can understand each other’s work output expectations, so as to effectively cooperate. After the process is standardized, tools can be used for automation to improve work efficiency and reduce man-made faults. Make the cooperation of each link as clear, perceptible and predictable as an assembly line.
Some people think that to practice Devops, team members should be full stack engineers, while Xita team thinks that professional division of labor can be in-depth and proficient. Therefore, the team can be divided into blockchain development, test development and operation and maintenance development teams. It is noted that each post has the word “development”, and a team with programming ability is the fundamental to effectively practice Devops.
Set three goals for the Sita team:
- Development of blockchain software – software code for automatic code quality inspection, testing, construction and release with CI
- Test blockchain software – Test code to test smoke, regression, performance and stability with CI
- Operation and maintenance blockchain Service – monitoring service uses CI for active monitoring and passive alarm
Requirements for such a team:
Programming ability is required for development, testing, operation and maintenance
The code of developing, testing and monitoring services follows the same development process and uses CI tools
Use GitHub flow for code management, create feature branch locally, send PR, code review / CI check, merge master branch
Automatic operation and maintenance
Use CI / CD pipeline to build, test, release and deploy automatically
After code development, developers will conduct code review, static code scanning, code specification and code security check, and run automatic testing. Automatic deployment of different links will be carried out according to different branch merging strategies. For example, merging into the development branch will trigger the deployment of staging environment. After successful deployment, testers will be automatically informed to carry out acceptance test. After merging to The master branch will trigger the deployment of the production environment. After the deployment is successful, the online notification will be automatically sent to the operators for operation. The use of automated CI / CD Tools and processes can shorten the time of function development to the last moment when developers submit code, and there are records on CI services. Everyone can know who is, when, what changes have been deployed and to which link. This improves the operation and maintenance efficiency and reduces the possibility of human accidents in the manual operation process.
The elk service is used for log collection and data analysis in operation and maintenance. As shown in the figure below, Xita team collects and analyzes the logs of services on each node of the test chain, which is convenient for operation and maintenance and developers to understand the causes when software failure occurs.
In order to facilitate the operation and maintenance of CITA services, Xita’s operation and maintenance team has developed CITA monitoring services, which can be used to monitor the operation of CITA services in the nodes, including collecting blockchain data, monitoring the survival of CITA processes, monitoring the operating environment, and automatic fault alarm notification. It provides a friendly and visual panel for development and operation and maintenance personnel, as shown in the following figure:
Similarly, the development process of the monitoring tool itself follows the same Devops practice and is also open-source https://github.com/cryptape/c…
Continuous automated testing
It improves the efficiency of development, operation and maintenance, and also ensures the quality of software, which is the focus of Xita test team. The following is the famous test pyramid in software testing. The test pyramid illustrates a dilemma: the higher the level of testing, the more energy, time and cost will be consumed.
The focus of Xita’s test team is to automate the most time-consuming and resource intensive system testing. This is not to say that the development team does not have to worry about testing. The development team mainly writes white box tests such as unit tests and integration tests.
Because the Xita team believes that:
Automated testing can maximize the value of QA;
Automation requires CI tools for management and execution.
Blockchain software is a multi instance system, which only guarantees the results of unit test and integration test, which is far from enough to ensure that the system is ultimately available and the results are credible. Blockchain is a tool to solve “trust” with computer software. The software must be trusted first, and the data generated can be trusted. Therefore, the Xita team has invested a lot in the testing work Quantity of energy to ensure the quality of software.
The system test done by Xita team is an end-to-end black box test of software running results in the simulation environment. In order to improve the test efficiency, we use CI pipeline to manage a variety of test strategies.
What is CI / CD pipeline?
As shown in the figure below, Jenkins is one of the CI tools used by the Xita team. A CI / CD pipeline is like a pipeline in sections, and the CI will be triggered after the code is submitted Different pipeline links have their own input and output, and any problem in any link will affect the work of the whole pipeline.
Before setting up CI / CD pipeline, we should first consider three questions
- How to design a reasonable assembly line?
- How to design test pipeline?
- Why set up multiple pipelines?
There are two key elements in pipeline design
Design appropriate workflow: workflow, such as stage 1 > stage 2 > stage 3
Design stage job in workflow: stage work, such as unit test, build, integration test, code audit, staging deploy, production deploy
Each stage job also has five key design elements:
Up / down stream: upstream and downstream stage jobs
Trigger: trigger condition
Input / output: input parameter; output component product, output parameter
Steps: execution steps
Notification: result notification; the final stage should have a clear notification object
As we all know, software development needs PRD (functional requirements document), and the same pipeline settings also need design documents. Before setting up specific CI / CD assembly lines, the downstream water lines should be designed. As shown in the figure below, the design documents of two pipelines of Xita team are as follows:
After designing the functions, execution strategies and operation steps of various pipelines, you can set the corresponding settings on CI tools. Do not change the CI configuration while thinking about it. Especially for complex pipelines, you should design and then set them. In this way, the pipeline has its own documents, which is convenient for other operation and maintenance colleagues to cooperate, and the development and testing can also clear the workflow.
As shown in the figure below, the distribution and testing pipeline set up by Xita team is as follows:
The release test pipeline is divided into four stages
Build the test object: get the software code, automatically compile and generate the binary file bin of different algorithm versions
Generate test run: put the bin file into a runnable environment and build docker image
Run test cases: run automated test cases for different bin versions; parallel execution saves time
Generate test report: automatically summarize the test results of each version, automatically generate a report including test object, test environment and test results; and automatically archive the build products for easy release
Multiple pipeline strategies
The release pipeline is undoubtedly the most important pipeline in the daily development of Xita team. In order to enable developers to find code quality problems as soon as possible, the team also set up various test pipelines, such as the system test pipeline used to guarantee the development main branch, the smoke to ensure the test code quality, the review test pipeline, and the performance test to guarantee the software performance Trial production line, etc.
System test pipeline:
When the development branch is merged, the regression test of the full test case set is performed to ensure the code quality of the main development branch.
Smoke test line:
It is used for incremental test cases to ensure the code quality of the test code; it is executed when the tester submits the feature branch of the test code; it only runs the P1 level test case set, which speeds up the code review and merge process; reuses the test objects generated by the release pipeline.
Regression test pipeline:
It is used to ensure the code quality of the test code; it is executed when the tester merges the development branch of the test code; it runs a full set of test cases, which is slow in execution, but avoids introducing destructive code; it also reuses the test objects generated by the release pipeline.
Other test lines
Performance test pipeline:
In addition to ensuring the usability of the software, the Xita team is very concerned about the running efficiency of the software, so the team has set up a performance test pipeline. Performance testing focuses on some performance indicators, such as TPS, latency and response time, which reflect the service processing capacity. However, the running cost of performance testing is very high, which needs to monopolize the host resources and run for a long time. Therefore, the Xita team set up to automatically run the performance test pipeline at 2:30 a.m. every night. After running, it will generate a graphical report and send an email to the development team. In this way, the development team can learn about the performance changes of the main branch every day. When there are significant performance changes, they can optimize and repair them as soon as possible, instead of waiting for the release test stage to know the results.
Stability test line:
The Sita team also focuses on the stability of software, which means that all kinds of key functions can work normally when the service has a lot of request pressure. Therefore, the Xita team has set up the stability test pipeline, which is executed in the release stage, and the pressure test parameters can be selected as required.
Usability test pipeline:
Developers have made some major changes. They want to know whether there are other side effects before merging them into the main branch. Therefore, the operation and maintenance department provides the usability test pipeline, and the branch regression test results can be obtained by developing the branch execution that needs to be tested.
Performance improvement test pipeline:
Developers have made some performance related changes and want to understand the overall performance changes. Therefore, the operation and maintenance provides performance improvement test pipeline. The branch regression test results can be obtained by developing the branch execution that needs to be tested.
Focus of collaboration in Devops mode
In Devops mode, the cooperation of development, testing and operation and maintenance personnel focuses on:
The goal of operation and maintenance is to provide convenient processes and tools for development and testing by using automation technology, so as to reduce repetitive work
The goal of testing is to use automation technology to improve test efficiency and ensure the quality of development results
Development enjoy the convenience of automation, concentrate on development, improve the quality of software
If there is no automation, there will be no efficiency improvement and continuous quality assurance. Of course, the maintenance of automation tools also has a price. This is exactly what the Devops team really needs to do.
Practical experience sharing
How to optimize the pipeline and test framework, the following are some of our practical skills and gains:
Pipeline optimization skills
- Production lines restrict each other to ensure quality
System test pipeline to ensure the quality of development code
Regression testing pipeline to ensure the quality of test code
- Automatic trigger rules with GitHub flow Convention
Feature branch running smoke, smoke after merging into development
Merge development, run and return, and merge to master
- Reusing test objects
Release test build test object
Smoke and regress to use the built test object
The test object is put into docker image for version management, and the same test object is used for local test and CI
Archive build artifacts
- Combined test report
- The test environment is isolated by docker container
- Each test case runs in isolation
Test framework optimization techniques
- Using environment variables and docker image tag to get test objects
- Using docker image to get a unified test environment
- Error auto retrying use case
- Parallel execution of test cases
- Manage output logs
- Output intuitive test report
What kind of test report is a good report?
The final result of test work is test report. What kind of test report is a good report? The Sita team believes that a clear and accurate summary of the test report should include:
- Software name
- Software version
- Source code address
- Source code commit ID
- Version information output from bin file
- The commit ID of the test code
- test case
- test result
A good testing framework should be able to help testers automatically generate such a test report and improve their work efficiency and accuracy.
Here’s the report from the test framework used by the Sita team, using TestNG as the base framework:
The Xita team has continuously improved the testing framework, so that the automatic generation test report can output the test object version, test environment, test code version and other information that we are concerned about. When receiving the test report notice, everyone can clearly and accurately know what the test is, what the results are and how to test.
The Xita team believes that one of the secrets of efficient enterprise collaboration is: don’t ask, look by your self! (don’t ask, look by your self!).
The above is the experience sharing of Xita team on the engineering quality control of blockchain software. Please note that each enterprise has its own advantages, weaknesses or historical burdens. The experience can not be copied, but can only be referred to. The best one is the one that suits itself. Thank you.