The essence of architecture is to manage complexity. Abstract, layered, divide and conquer and evolutionary thinking are the four most basic weapons for our engineers / architects to deal with and manage complexity.
Recently, some newcomers have come to the team, some of whom have some work experience and come in as senior engineers / architects, but I find that most of them tend to think about application and details and have weak abstraction ability. Therefore, as part of the team’s technical training, I sorted out this article, hoping to help them establish correct architecture design thinking. In my opinion, the cultivation of thinking habits and thinking ability is far more important than the mastery of practical technical tools.
Due to the long content of this article, I will divide it into two small articles. In the first article, architectural thinking that excellent architects must master, I will first introduce the four basic thinking to deal with complexity: abstraction, layering, divide and conquer and evolution. In the second “four architecture design cases and their thinking methods”, I will explain how to comprehensively use these thinking to construct and design small systems, medium-sized systems, infrastructure and even organizational technology systems through four cases.
1、 Abstract thinking
If I ask what is the most important capability in software R & D / system architecture, I will not hesitate to answer that it is abstract capability. The word abstraction is often heard, but few people really understand and can explain what abstraction is. Abstract is actually defined as follows:
The process of simplifying the representation or description of something. Abstraction makes us pay attention to elements and hide additional details.
For an example, see the figure below:
What do you see? You see a door, don’t you? What you see is neither wood nor carbon atoms. This door is abstraction, and wood or carbon atoms are details. In addition, you can see a door handle on the door. What you see is neither iron nor iron atoms. The door handle is abstract, and iron and iron atoms are details.
In system architecture and design, abstraction helps us get our mind about big picture and hide details temporarily. The abstract ability directly determines the complexity and scale of the problem we can solve.
The following figure shows the building blocks we played when we were children. I found that children who liked to play with building blocks and built them quickly and well generally had strong abstract ability.
The building block castle on the right of the figure above is an abstraction. If you look closely at the castle, it is actually composed of several sub modules. These modules are sub Abstract units, and the building blocks of various shapes on the left are details. When building blocks, the child will first have a big picture (Abstract) of the castle in his / her head, and then he / she will have a preliminary sub module decomposition (completed subconsciously), then use the building blocks to build each sub module, and finally assemble the final castle. There is a top-down divide and conquer design, and then a bottom-up combination process. This divide and conquer thinking is very important, which we will talk about later.
I think there is no essential difference between software system architecture design and children’s building blocks, but the problem domain and scale are different. Architects should first form abstract concepts in the brain, then decompose sub modules, then implement sub modules in turn, and finally assemble and combine sub modules to form the final system. So I often say that programming and architecture design are building blocks. Influenced by professional habits, excellent architects see a modular world.
Abstract ability is not only important for software system architecture design, but also very important for human activities in other fields such as architecture, commerce, management and so on. In fact, we can think that the world we live in is built on the basis of abstraction, and it will be difficult for human beings to move without abstraction.
By the way, the problem of abstract level jump is very common in development. Experienced programmers will keep the consistency of abstraction level when writing code. The code reads like a story, which is clear and easy to understand; Inexperienced programmers will have obvious abstract level jumping problems, and the code is tired to read. This is caused by insufficient abstract ability. for instance:
When an e-commerce website processes orders, it generally follows this process:
- Inventory update
- Discount calculation
- Paycard verification
- Payment (pay)
The abstraction in the above process is at the same level, which is relatively clear and easy to understand. However, when an inexperienced programmer implements this process, the code level will jump. For example, when the main process goes to the payment card verification, his code will suddenly jump out of a bank API remote call in a line. This is an abstract jump. The bank API call is a detail, It should be encapsulated in the abstract paycard verification.
2、 Hierarchical thinking
In addition to abstraction, layering is also our basic thinking weapon to deal with and manage complexity. As shown in the figure below, in order to build a complex system, we divide the whole system into several levels, each of which focuses on solving problems in a certain field and providing services upward. Some layers are vertical and run through all other layers, called the shared layer. Layering can also be regarded as a way of abstraction, which decomposes the system abstraction into several hierarchical modules.
There are many cases of layered architecture. For a small and medium-sized spring web application, we usually design it into a three-tier architecture:
The operating system is a classic layered architecture, as shown in the following figure:
TCP / IP protocol stack is also a classic layered architecture, as shown in the following figure:
If you pay attention to the evolution of human civilization, you will find that today’s human world is also built and evolved layer by layer. Today’s Internet system can be regarded as a level of modern civilization. Above it is modern commerce based on the Internet, below it is the infrastructure of modern electronic industry, and so on.
3、 Divide and conquer thinking
Divide and combine or split and merge is also a general method to deal with and manage complexity. The following figure shows a thinking process of divide and Conquer:
For a large problem that cannot be solved at one time, we will first decompose the large problem into several sub problems. If the sub problem cannot be solved directly, we will continue to decompose it into sub problems until it can be solved directly. This is the process of dividing; Then, the solutions of the subproblems are combined into the solutions of the subproblems, and then the solutions of the subproblems are combined into the solutions of the original problems. This is a process of combination.
During the interview, in order to investigate the divide and conquer thinking of candidates, I often meet a divide and conquer question: how to sort a 100g large file with an ordinary computer with 8g memory / 500g disk space? It is assumed that there are string records in the file, with a line of about 100 characters.
This is a typical divide and conquer problem. 100g large files cannot be loaded into memory at one time and sorted directly, so they need to be cut into several small problems to solve. So how much data can a computer with 8g memory row at a time and finish it in a limited time? That is, how to cut 100g large files and how many copies are more appropriate? This is to investigate the candidate’s time and space complexity estimation ability. It requires certain computer organization and algorithm skills, as well as some practical experience and sense. In fact, if you use 8g of memory, the operating system needs to use some. If you use java to develop the sorting program, the JVM can roughly use 2 ~ 4G of memory. Based on the general empirical value, there should be no problem in sorting about 1g of data at a time (I actually sorted 1g of data on the computer, which is OK). Therefore, 100g files need to be cut into 100 copies, 1g each, so that each sub file can be directly loaded into memory for sorting. For the string sorting of 1G data, it is more appropriate to use the quick sorting algorithm provided in Java.
OK, after sorting for a limited time (depending on the computer performance, it can be finished in a fast day), assuming that 100 1g files have been arranged, which is equivalent to 100 ordered files on the hard disk, but what we finally need is an ordered file. What should we do next? At this time, we need to combine the solved sub problems into the final result file we need. What algorithm should be used at this time? Here, we examine the candidates’ mastery of the external sorting and merge sorting algorithms. We can merge and sort 100 ordered files in pairs. In this way, we will get 50 ordered files, each size of 2G. Then merge them in pairs and repeat until the last two files are merged into the target file. This file is 100g and in good order. Because it is external sort + merge sort, you only need to read the file pointed to by the current index and record it in memory for comparison. The smaller one is output to the target file, which takes up very little memory. In addition, the above algorithm is two-way merging, multi-way merging, or even heap sorting for optimization, but the overall divide and conquer idea has not changed.
In general, this is a very good interview question. In addition to investigating the divide and conquer thinking of candidates, it also investigates the understanding of various sorting algorithms (fast sorting, external sorting, merge sorting, heap sorting), the estimation of time and space complexity of calculation, the internal and external memory characteristics and organization of computers, file operation, etc. In fact, there are very few candidates who can fully answer this question. If I am lucky enough to meet one, I will be a treasure, because this person has the potential to grow into an excellent architect.
In addition, recursion is also a special divide and conquer technology. Developers who master recursion technology are equivalent to mastering a powerful programming weapon, which can solve some problems that ordinary developers cannot solve. For example, recently, my team is developing a new service framework, including contract parser, code generator, serializer and other components, which requires a lot of recursive thinking and technology. Developers without this thinking can’t do this. Therefore, when I interview candidates, I usually give recursion related programming questions to investigate the recursive thinking of candidates.
Recursive structures can be found everywhere in nature. As shown in the figure below, if you are interested, you might as well think about what enlightenment nature gives us through recursion?
4、 Evolutionary thinking
People in the community often discuss: is the architecture designed? Or evolved? Based on more than ten years of experience, I personally believe that architecture is not only designed, but also evolved. For the Internet system, it can basically be said that it is three-part design and seven part evolution, and it evolves in design. Design in evolution is a continuous iterative process.
In the whole life cycle of Internet software system, the early design and development only account for about three points. In the next seven minutes, the architect needs to constantly adjust the architecture according to the user’s feedback. I think architects should not only make use of their own architecture design ability, but also learn to use the power of user feedback and evolution to promote the continuous evolution of architecture. This is evolutionary architecture thinking.
Of course, the architecture design at the beginning is very important. The architecture system is basically formed, which can’t be careless. At the same time, excellent architects know that a system that can constantly respond to environmental changes is a viable system. The quality of the architecture largely depends on its flexibility to respond to changes. Therefore, architects with evolutionary thinking can consider the evolutionary characteristics of subsequent architectures at the beginning of design, and take the ability to respond flexibly to changes as the main consideration of architecture design.
At present, a new architecture methodology ~ evolutionary architecture is emerging in the community. Microservice architecture is a typical evolutionary architecture, which can quickly respond to the changes of market user needs, while single block architecture lacks this flexibility. Martin fuller once gave an evolution roadmap of microservice architecture on his blog [appendix 8.2], which can be used to explain the difference between design thinking and evolutionary thinking, as shown in the figure below:
The above route is to go straight to the micro service architecture from the beginning. In fact, it reflects the thinking of design architecture, and believes that architects can fully design the whole system and its evolution direction. Martin believes that the risk of this approach is very high. One is the high cost. The other is that at the beginning, the architects do not have a deep understanding of the business domain and can not clearly divide the domain boundary. The developed system is likely to fail to meet the needs of users.
The following route starts from the single block architecture. With the continuous deepening of the architect’s understanding of the business domain and the continuous expansion of the business and team scale, the idea of gradually dividing the single block architecture into micro service architecture is the idea of evolutionary architecture. If you look at the system architecture of some Internet companies in the real world (such as eBay, Alibaba, Netflix, etc.), most of them follow the route of evolutionary architecture.
The following figure shows the evolution history of architecture. At each stage, you can see the shadow of design, but if the timeline is long enough, the characteristics of evolution will come out.
5、 How to cultivate architecture design thinking
The cultivation of good architecture design thinking is inseparable from the practical training of a large number of high-quality projects in the work, and then the usual learning, thinking, refining and summary.
In addition, the basic architecture design thinking can be found in our university computer courses (such as data structures and algorithms), but at that time, it was mainly learning, and the problem domain was relatively small and idealized. Therefore, university education is actually very important. The basic architecture design thinking has been planted at that time. It will be further digested and applied in later engineering practice. With the accumulation of experience, the complexity and scale of the problem domain we can solve gradually become larger, but the basic weapon is still thinking of abstraction, stratification and division.
I think the height of an architect’s growth is closely related to the cultivation of his thinking habits in college. As far as I know, the world-class Internet companies, such as Google, have harsh requirements for data structures and algorithms when recruiting new engineers. It is understandable that the problems to be solved by Google level companies are super complex, and their basic thinking skills are weak, so they can’t deal with them at all.
For novice engineers with less than 5 years of working experience, if you were a waste in college, it is recommended to self-study the relevant courses after work. I recommend learning by referring to the data structure course cs61b [appendix 8.1] of Berkeley University in the United States, which is very helpful to establish Abstract programming thinking. I have self-taught this course in the graduate stage, and now I think it has benefited a lot. Note that all Lab / home work / projects in this course should be done by hands in order to have good results.
The course I taught myself was cs61b 2006 Autumn Edition. The above picture shows the course logo
For evolutionary design thinking, the current university education actually has little training. On the contrary, most of the current university education adopts simplified ideal models divorced from the real scene, and some are exam oriented teaching with fixed answers. This way will lead to the certainty of students’ thinking and is not conducive to the cultivation of evolutionary design thinking. From my personal experience, evolutionary design thinking is more tempered and cultivated in practical work.
- The essence of architecture is to manage complexity. Abstract, layered, divide and conquer and evolutionary thinking are the four fundamental weapons for architects to conquer complexity.
- Having mastered the four basic weapons of abstraction, layering, partition and evolution, you can design as small as a class, a module, a subsystem, or a medium-sized system, or as large as a company’s basic platform architecture, micro service architecture, technical architecture, even organizational architecture, business architecture, etc.
- Architecture design is not static, but dynamic evolution. Only a system that can constantly respond to environmental changes is a viable system. Therefore, even if you master the three basic thinking of abstraction, layering and divide and conquer, you still need evolutionary thinking to promote the continuous evolution of the architecture with the power of feedback and evolution while designing.
- While paying attention to technology and developing applications, architects need to regularly sort out their architecture design thinking. After a long time of accumulation, the way you look at things in the world will change fundamentally. You will find that the world we live in is actually built on the basis of abstraction, stratification, division and evolution. In addition, the formation of architecture design thinking will have a significant impact on your system architecture design ability. It can be said that the depth of abstraction, layering, division and evolution and the level of flexible application directly determine the complexity and scale of the problem domain that architects can solve. It is a watershed to distinguish between ordinary application architects and platform / system architects.
- Berkeley CS61B http://datastructur.es/sp17/
- Single block priority https://www.martinfowler.com/bliki/MonolithFirst.html