Uncover the secrets of concave convex technology, basic service system and build a server-side technology center

Time:2021-12-7

preface

Bump lab has grown from the initial front-end team to today’s full-end team, which means that we pay attention not only to the technical capabilities of the front-end, but also to the capabilities of the whole end and the whole stack. In this article, from the perspective of the front-end team, we describe some difficulties we encountered in building the service-end system, the built service architecture, and how to better help business growth.

A little deja vu

First, let’s take a look at some scenes in our daily work.

  • Scenario a: in some businesses, the data interface provided by the underlying data team does not provide HTTP calls. It is necessary to find other server teams to encapsulate it. At this time, it is necessary to wait for other teams to schedule, which may cause the business to fail to go online normally;
  • Scenario B: the front-end page performance is stuck. Due to too many calling interfaces, you need to wait for other server teams to aggregate data;
  • Scenario C: we need SSR in some projects, and the front and back ends need to reuse a unified set of templates;
  • Scenario D: we have incubated some projects internally, which need interface services and need to wait for the support of other server teams.

Behind these scenarios, we urgently need to establish a server-side R & D team to undertake some business service development and better help the team develop in the future.

forming

In terms of team formation, “internal selection” and “external recruitment” are mainly adopted. In terms of team development, we have mainly experienced three stages.

embryonic form

In the initial stage, nodejs is selected as the server programming language. There are two main considerations:

  1. Most of the students in the team are familiar with JavaScript, can quickly learn nodejs, and the starting cost is low;
  2. It has natural advantages in SSR, and the front and rear ends can share some codes.

At this stage, we quickly incubated a series of systems and platforms, such as mock platform, front-end monitoring platform, bottom-up platform, etc. Our main objectives are to enrich the front-end R & D system, improve the development and cooperation efficiency of the front and rear ends, and precipitate some nodejs middleware.

grow up

After some precipitation in service development efficiency, performance, stability and security, we began to think about how to standardize service development and support business growth more efficiently.

At this stage, we not only output a series of knowledge systems such as “R & D specification”, “R & D process” and “development framework”, but also build relevant R & D platforms such as “deployment platform” and “general management platform”. In terms of business development, we use nodejs to implement “Tiangou” game service, openresty to implement “data aggregation service”, SSR on some channels, etc.

Empowerment

In the design stage, we precipitated a large number of general services, such as services related to “page”, “picture” and “compilation”, and some services were enabled to other teams. For example, we enabled services related to page intelligent design to products such as “Jianghu platform” and “smart shop”.

Within the company, most of the server team technology stacks are mainly Java, and the JSF protocol developed by the group is used for calling between services. The main technology stack of our team still focuses on nodejs, which provides HTTP calls to other teams. It is quite different from Java in access mode, current limit, code reminder and so on, and can not make good use of a large number of Middleware in the group.

At this stage, we introduced the Java technology stack to form a technical system with “nodejs + java” as the main server language. For some domain services, we provide a java version of JSF service implementation to facilitate communication and cooperation among third-party teams.

Architecture

After several years of precipitation, our team has built a preliminary architecture in the server field.

The main goal of the construction of the server-side R & D system is to improve the lower limit of the team code, improve development efficiency, improve service delivery quality and promote the common growth of the team.

system

Build a set of server-side R & D system, mainly focusing on 8 aspects, including R & D specifications, R & D platform, R & D process, document construction, team management, monitoring alarm, middleware management and infrastructure.

platform

In terms of the overall service architecture, our daily development adopts the hierarchical structure mode, pulling out the general service logic as much as possible, outputting more building blocks and reducing our development and maintenance costs.

The following is a brief description of our exploration and practice in the field of server in recent years from the aspects of “business support” and “technology construction”.

Business support

Business is the foundation of the team. Only in the rapid growth of business can we continuously verify and optimize our entire service system and ensure the reliability of the overall service.

Linglong service

Linglong’s full name is Linglong intelligent design platform, which provides one-stop online design services: one click matting, free matting, commodity belt, size change, commodity main drawing design, online advertising banner design, store home page design, activity page design, page design, interactive marketing design, applet design, motion picture video design, video advertising design, commodity main drawing video design Poster design, official account design, name card design, DM leaflet design, logistics surface design, Yi Bao Bao design, poster design, etc. It provides a large number of exquisite templates and free materials, free design, and an enterprise zone. It is a platform dedicated to becoming a design partner for businesses.

Linglong intelligent page design is a platform that gathers various senior e-commerce industry designers in the industry to provide one-stop professional intelligent pages and small program design services. The whole architecture service is lightweight and modular, which makes it more convenient to expand the business scenario of the special area. The service architecture is as follows:

服务架构图

The overall architecture is divided into four parts: Web application layer, interface layer, service layer and data layer. In this way, the split can achieve unified entry. In terms of deployment, single point deployment makes publishing more convenient. Independent deployment reduces the impact of module update on the overall service:

  • Web application layer: including Linglong and other platform applications;
  • Interface layer: provide gateway service, and the request of application layer is forwarded to each module through the gateway and authority verification;
  • Service layer: it is mainly divided into the following four parts:
    • Service communication: MQ is used for asynchronous communication, and HTTP call is used for RPC communication;
    • Business modules: that is, the core logic of services. They are divided into different modules according to functional logic. Most functions are processed independently in the module to achieve the effect of high cohesion and low coupling;
    • Basic services: support the basic functions of business modules and control users and permissions uniformly;
    • Service Management: used for service assistance to improve the stability, robustness and flexibility of services.
  • Data layer: the service mainly uses mongodb and redis. The former is the main storage and the latter is used for data caching.

The project is developed by Typescript, following the unified interface specification, ensuring the unity of style of access and participation. Modular design enables service operation and maintenance and iteration to be relaxed. It supports the expansion of the area and scenes in function, and makes the business unlimited.

Data aggregation service

In e-commerce businesses, such as channels and promotional activities, the data of commodity groups and advertising groups will often be used. There will be more redundant fields in the general interface, especially when querying services in batch, the whole response package will be relatively large.

We use openresty to implement graphql service. The data is loaded on demand, which can effectively reduce the packet size; Automatic data disclosure can ensure service availability, especially when the underlying services respond slowly during the promotion period.

o2api

Technology construction

The necessary infrastructure and technology exploration is to better help the business and team face the future.

The following describes some of our infrastructure on the server side around “Talos deployment platform” and “general management platform”.

Talos deployment platform

Talos deployment platform is developed based on the internal jdos platform. It mainly provides application resource management and deployment functions, solves the problems of difficult deployment, low development efficiency and high service operation and maintenance cost, and makes R & D students focus more on development.

The main structure diagram is as follows:
Talos 框架图

We will briefly introduce the platform from the two aspects of “resource management” and “application deployment”.

resource management

Talos 项目分组

The project grouping function is convenient for developers to manage and view the application, traffic, resource occupation, etc. under the grouping.

Talos 静态网站部署方式

Static website deployment supports the deployment of static website applications to the same back-end application. When the browser accesses, it matches the corresponding files according to the domain name or pre routing, so as to save resources and improve resource utilization.

In addition, there are other functions:

  • Talos gateway is provided to facilitate service forwarding and mounting;
  • Provide mongodb visual panel function to facilitate the development students to view the online database, and provide read-only, read-write and other permissions;
  • Provide the whole process monitoring function, including application creation, deployment, container adjustment, etc. when the CPU, disk, load, etc. exceed the threshold during operation, an alarm will also be given;
  • Other functions include container quantity adjustment, online restriction during large promotion, notification, etc.
Application deployment

It supports multi environment deployment and can set testing, pre issuance, production and other environments. Each environment has its own configuration file and configuration attribute fields. It supports one click deployment, rollback, offline and other operations. The deployment interface is shown in the following figure:

Talos 应用部署

Support deployment of different project types, such as nodejs, static sites, custom deployment, etc.

Talos 不中断部署

Support uninterrupted deployment, use jdos’s rolling update interface to control process switching, and divide the application container into two batches of rolling updates to ensure that the application will not be interrupted during the update process.

General management platform

In the development process, it is often necessary to hard code some data, and most of the data needs to be updated from time to time in future maintenance and operation. In the past, we were often occupied by these trivial modified data, which reduced the coding efficiency. There are two ways to solve this problem. The first is to store the change data in the database and develop the corresponding server interface for crud. In this way, the resources we need include database, server development and gateway domain name. It seems that the gain is not worth the loss. The second is to dynamically configure forms in the platform, define data structures, and then enter data. At the same time, the platform provides a unified crud gateway API. Under the above background, the general content management platform came into being. Let’s see what practical functions are provided!

The general management platform provides form and data creation, which meets most of the configuration management functions; At the same time, it provides permission management function for product / operation colleagues to update data and get rid of the trouble of allowing development students to modify data / version release.

通用管理平台大纲

Data form – data structure definition

Why have a form instead of storing JSON data directly? Imagine that in the mongodb database, if you can’t use the ORM framework such as mongoose to define data, you can only understand the meaning of table structure design by reading data or separated documents. That must be very painful.

创建表单

The functions of forms are mainly divided into form field design, user authority management and form identification editing. The field design provides a schema design similar to the relational database. Users can create the type and default value of the field corresponding to the table structure, and even verify the data through regular expressions.

表单管理

Data – data storage

The functions provided by the data module include data entry, data synchronization (different environments), version management and API link acquisition.

  • Data entry: when entering data, check according to the field rules of the form to prevent data inconsistency in the same form;
  • Data synchronization: the platform provides two environments, advance and formal. Users can fully synchronize and partially synchronize the data of the two environments;
  • Version management: like most content management platforms, this function is provided to prevent user misoperation or restore old version data;
  • API link: after entering data, you can access the entered data through this link.

通用管理平台帮助文档

The above describes the function points of the general management platform. In practice, there are a large number of application access, including Linglong, Jerry, taro, quark and other excellent projects.

epilogue

So far, our accumulation and precipitation in the service end field is only the tip of the iceberg, which needs further exploration and precipitation. In the future, we will focus more on service building blocks and output more reusable and enabling building blocks to escort business growth.

Concave convex secret series article address

1. Past and future of bump Laboratory

2. Uncover concave convex technology · Linglong intelligent design platform · digital intelligence of dream by Dream Design

3. Uncover concave convex technology · Deco intelligent code · start the revolution of production and research efficiency

4. Uncover concave convex technology · Linglong page visualization · the road of growth and transformation

5. Uncover concave convex technology · quark design assets · create full matrix high-quality materials

6. Uncover concave convex technology · tide R & D platform · layout R & D new infrastructure

7. Uncover concave convex technology · taro · from cross end to open cross end frame

8. Uncover the secrets of concave convex technology, basic service system and build a service end technology hub


Welcome to the lab blog:aotu.io

Or focus on the official account of bump Laboratory (AOTULabs).

欢迎关注凹凸实验室公众号

Recommended Today

Heavyweight Tencent cloud open source industry’s first etcd one-stop governance platform kstone

​ Kstone open source At the kubecon China Conference held by CNCF cloud native foundation on December 9, 2021,Tencent cloud container tke team released the open source project of kstone etcd governance platform. KstoneIt was initiated by the TKE team of Tencent cloud containerCloud native one-stop etcd governance project based on kubernetes。 The project originates […]