Speech by Cai Luwei (CTO of baby bus)
Tech minds No.08 Fuzhou
Sorting out Northwest China
At the tech minds No.08 Fuzhou field hosted by cloud, Cai Luwei, CTO of baobabus, as a guest, shared the “workflow and automation of baby bus”, revealing the secret of baobabus operating 150 apps at the same time.
Cai Luwei, CTO of baby bus, senior system architect. He led and developed large-scale Japanese system, 2D online game engine, automatic simulation / test engine, etc. he was the earliest promoter of agile development, advocating open source, automation and minimalism. A developer with rich team management experience and solid R & D skills.
Cai Luwei, CTO of baby bus
Pioneer of early childhood education in China
Baby bus is an Internet company providing intelligent early education products. It has successfully produced 150 apps, nearly 600 nursery rhymes and 50 episodes of animation.
App store traffic ranking
In August 2016, baby bus ranked second only to Tencent, Baidu and Alibaba in terms of global company traffic of app store.
After three years of steady development, each product of baby Bus supports 17 languages, covering more than 50 countries in the world, and is loved by many children in Southeast Asia, Russia, the United States, Japan and Arab regions.
Overseas market development of baby
Indonesia has the best development in overseas market, and Arab and UAE have a high share.
There are new products every week and upgrade every month
Baby bus has a series of products with huge user volume. We need to develop a set of universal standardized production process solutions to support a large number of product maintenance and innovation.
SOP for baby bus development
The workflow of baby bus includes topic selection, development, user verification, testing and release. One of the most important points is: to ensure the quality of the topic, the overall development, testing and rapid iteration can be highly suitable.
How to choose a topic?
First of all, the project is selected by planning, product experience group and planning PK, whether the topic selection conforms to the results of the questionnaire survey, whether it meets the actual needs of children and parents, and so on. After strict screening, the quality of topic selection is guaranteed.
After two rounds of Agile Development upgrade, our current development process is planning first, then art design, and finally programmer development. This way can ensure that when the programmer intervenes, he can operate immediately and cooperate with the artist efficiently.
After the development is completed, check out the version and conduct automatic testing, including simple monkeying test and flashback test, to ensure that the program does not appear big abnormalities, and then test the products, let the artists, technology and kindergarten children experience improvement together. We are strict with testing. From production development to testing, we will limit three rounds of testing. Once more than three rounds are exceeded, the product must be launched or returned.
Product structure of baby bus
From the technical structure of supporting products, “framework support” is an important part of APP R & D. the framework will develop common SDK packages, such as we media advertising and game effects. At the same time, linkage tool group and Automation Group will cooperate to provide automatic testing and packaging services, forming a set of standard pipeline.
Baby bus has developed more than 200 products. However, apple does not allow a large number of products in the market at the same time. We now have 150 products in the market, including more than 40 channels such as IOS, Android, TV, desktop (window and MAC), and third-party game machines at the peak.
In the early stage, we need to distribute the products to different channels for downloading, obtain the source, and determine which channel works best. At that time, there were a total of 40 channels and 11 languages, which meant that the baby bus would need more than 60000 installation packages. At this time, it is necessary to merge some long tail channels, focusing on the top 15 with large amount. At the same time, the language is optimized to minimize the difference package, leaving only Arabic package (the word left and right reversed) and general package. Later, through the further optimization of the formula, the optimized installation package is 150x2x (2 ~ 3).
How do we accomplish the weekly new product and monthly update? There must be a stable, scalable, customized workflow that can support various needs, multiple engines, apps and channels.
We will be divided into two development lines, a new function trial line, and a full line upgrade line. First of all, the new function trial line will try a new function, and conduct rapid online verification, and observe the product effect released through a / btest. If it runs stably and the effect is positive, then we will merge it into the real full line upgrade. Secondly, the whole upgrade line is fully developed and launched every month to ensure the active market version. So in fact, the two paths are parallel to achieve rapid cooperation. At the same time, the technical team requires that the technical strength and work efficiency be increased by more than 20% every year, so that no one else can catch up with us.
Reasonable personnel distribution
Baby bus team structure
The technical team of baby bus is divided into game team, application team and service team.
The game team is divided into 2D and 3D teams. There are four groups below. The first group is the new product group, which is only responsible for the continuous development of new products. The latter operations are completed by other groups, such as product plus international language, product re update, etc. The second is the framework group, which is responsible for linkage with other departments for the next test and release. The third is the plug-in department, which is responsible for the content integration of the next version, similar to the new function group or effect group. The fourth integrated group is responsible for things other than these three things, such as packaging and updating.
The application team refers to the family long-term team, including the application product group and wechat group. The application product group mainly serves the parents and makes the baby bus products. The wechat group is responsible for the establishment of account number and the communication between parents and users.
The service team is divided into service-side team and tool team. The tool group includes four parts of business: the first is the development of basic tools, which generates tools through process customization; the second is responsible for plug-ins such as IDE, 3DMAX, eclipse, sublime, etc.; the third data group is responsible for data statistics, and integrates analysis with the third-party data statistics platform to derive effective conclusions. The fourth automation group is responsible for automation development, automatic testing, automatic release, automatic operation and maintenance, and automatic simulation.
The production process of baby bus has gone through three generations. The first generation was before 2013. It is in the market dividend period. The products we launched are relatively high-end in the mobile early education industry. The market will actively help you recommend it to users and bring traffic. However, before 2013, IOS and Android were independently developed, and the development cost of Android using micro cloud engine was high, so the team basically only developed and planned. At that time, new products were available every week. We need to consider accelerating the whole product line process.
Our approach is: plan eight products, IOS and Android two departments at the same time, receive four products each, and develop at the same time. Three months later, the two groups will cross over the developed products, IOS to translate Android, Android to translate IOS, to ensure the fastest development speed.
We should have used Lua relatively early. In the second half of 2011, we started to use cocos2d iPhone to make games. We tried to integrate Lua and make it, but the game at that time was online games. In the second stage, from 2014, we merged IOS and Android platforms, invited experts to conduct agile development training for the whole team, and customized the agile development process for baby bus.
In the third stage, we are carrying out multi-dimensional incubation development. We don’t have a real group. A programmer may belong to either the technology department or a member of an incubation project group. In one stage, as a programmer, I want to finish the work together with artists and planners, and the position of the next stage will change. I will finish the work with other programs and artists, and adjust the work according to the needs of the construction team. At the same time of rapid incubation, agile development has also been comprehensively upgraded. Baby bus will develop a custom UI editor for planning. Programmers can write code synchronously with artists. When the artist draws the line diagram, the programmer assumes that this is a panda or other animal, and then writes the logic. Later, the art program is iterated and integrated to form a product that is truly presented to everyone.
After the framework group has built the whole framework system and a complete set of processes, the developers of baby bus 2D can start to develop games after three days as long as they know one development language, such as Java. Complex, such as unity3d, can be developed after about 15 days of training. Among them, we will carry out personnel training through one-to-one, pair programming, video tutorials, and the use of automation development tools.
Here we need to emphasize the idea of minimum programming, programmers only need to do the actual business development, do not need to pay attention to other things. Android development such as startup, error location and packaging is slow, debugging is troublesome, and the running efficiency of the old machine is very low. Baby bus has its own highly customized simulator, which can ensure rapid development, without recompiling and repackaging the modified product code. The function will come out within 1 second after refreshing.
Overall process of U3D
Taking U3D development as an example, after completing the planning content, we will invite the experience group and a product manager with decision-making power to start judging whether the planning content is high-quality.
If the plan is passed, it will enter the simple design of artists and the production of demo. Demo production will be verified, if the verification is passed, the detailed function development will be carried out. After the development is completed, the internal test will be carried out, and then the audio, online copywriting and American publicity will be added synchronously. The business process for 2D is similar.
Full line upgrade is the most complex. After the development is completed, there will be 5000 or even 7000 packages. The whole line upgrade team will use the automation tools to deliver the package, and realize the overnight package delivery (the automatic test is completed in the daytime, and all the installed packages that pass the test can be typed out at night, and the test team will be handed over to the test group for artificial functional test the next day). The comprehensive team is the team in charge of the later stage of the whole line upgrade, which will be completed by the machine and cooperate with the market for product docking.
Whole line upgrade workflow
There is only one important product of the parent line – Baby bus Daquan, so we will directly use group AB to carry out the parent line at the same time to increase work efficiency. At the same time, each week is the rapid iteration of the product iteration cycle and continuous integration.
Parent line development process
Efficient automated production tools
Because the development speed of the third-party framework of the game is not fast enough, and in the early stage, it does not support the functions of physical engine, bone animation, high-low clear image intelligent processing, image compression, audio and video compression, and the support of different devices is not complete. We independently encapsulate the version based on quickcocos2d-x 2.1.4, write the logic content needed by baby bus, rewrite the underlying business, and form a unique 1.0 game engine for baby bus.
In order to render a sprite in the center of the screen that can respond to the click, we can reduce the 15-20 lines of code required by the traditional programming method cocos2d-x into one line of code after adopting the minimalist tagging design and chain design.
2.0 architecture upgrade, the former is so easy to upgrade to the latter, IOS is also similar.
Before framework upgrade
After upgrading the framework
Automatic packaging is a link of automatic release. Baby bus now releases thousands of installation packages every month. We use the product background of APP admin to automatically synchronize the product information in the product background into the packaging configuration environment, select the packaging engine, channel and configure plug-ins according to the demand.
When packaging, it will have a set of templates according to the game framework version, quickly copy, modify and complete the package. In addition, the packaging mechanism has realized distributed order receiving, and can freely configure physical machines, such as 1 Mac Pro, 1 mac mini, and several windows.
Automatic packaging features:
1. The packer clicks to pack;
2. After the package is completed, it can also be uploaded to share with one key, which is convenient for communication with the test
3. Unified packaging template and easy to find out the package location, easy to debug
Distributed order receiving features:
1. If there is a machine, you can get involved in receiving orders and packing
2. Packaging machines are free to compete. Any computer failure will not affect other computers to continue packing
3. Plug in and plug-in connection with single computer to slow down the load
4. It is convenient for debugging and does not affect the normal delivery
Using the interface made by Django xadmin, the errors in the packaging can be caught and displayed in the corresponding file and running interface.
Package log page
What kind of plug-ins are needed for the basic configuration of the product can be selected through the interface shown in the figure below. For example, baidu channel should have Baidu advertising plug-ins. If this game supports photo albums and camera heads, they will also be configured here. After the configuration is completed, the package can be released. Now the average packing speed of Android is 25 seconds. Although the limit is fast, it has unstable factors, so we canceled the limit out of package. We will control the packing speed of IOS in about 60 seconds.
About tech minds
Tech minds is a series of activities of high-end technology leaders’ private meeting hosted by cloud again, which is held monthly in different cities across the country. In order to ensure the sharing effect of the private sharing meeting, the number of participants was limited to about 15. Technical leaders such as CTO, technology VP, operation and Maintenance Director and CEO with technical background from domestic excellent Internet enterprises were invited to gather together to exchange and share with each other, and jointly create a full-fledged social gathering focusing on the exchange and learning of technical topics.