Still using go micro service? Come and try go zero, it’s beyond your imagination!

Time:2021-6-14

Github

https://github.com/tal-tech/go-zero

Rapid construction of high parallel development micro service

0. Why is it difficult to do well in micro service?

In order to do a good job in micro service, we need to understand and master many knowledge points

  • Basic function level

    1. Concurrency Control & current limiting to prevent services from being destroyed by burst traffic
    2. Service registration and service discovery ensure the dynamic detection of increased and decreased nodes
    3. Load balancing needs to distribute traffic according to the capacity of nodes
    4. Time out control to avoid doing useless work for the time-out request
    5. Fuse design, fast failure, ensure the recovery ability of the fault node
  • High level function

    1. Request authentication to ensure that each user can only access their own data
    2. Link tracking is used to understand the whole system and quickly locate specific request problems
    3. Log for data collection and problem location
    4. Observability, no metric, no optimization

For each of these points, we need to spend a long time on its principle and implementation. For our back-end developers, it is very difficult to master and implement these knowledge points into the business system, but we can rely on the framework that has been verified by a large amount of traffic.Go zero microservice frameworkThat’s why we were born.

In addition, we always adhere to theTools and documentationThe concept of the new concept. We want to reduce the mental burden of developers as much as possible, put all our energy into the code that generates business value, and reduce the writing of duplicate code, so we developedgoctlTools.

Next, I will demonstrate through short chain micro servicego-zeroQuickly create the process of microservice, go through it, you will find: the original writing of microservice is so simple!

1. What is short chain service?

Short chain service is to convert a long URL into a short URL string by program calculation.

This short chain service is written to demonstrate the whole process of go Zero building a complete micro service. The algorithm and implementation details are simplified as much as possible, so it is not a high-level short chain service.

2. Short chain microservice architecture

Still using go micro service? Come and try go zero, it's beyond your imagination!

  • It’s only used hereTransform RPCA microservice is not to say that API gateway can only call one microservice, just to demonstrate how API gateway can call RPC microservices
  • In a real project, each microservice should use its own database as much as possible, and the data boundary should be clear

3. List of code generation for each layer of goctl

All the function modules with green background are generated automatically and activated on demand. The red module needs to be written by itself, that is to add dependency and write business specific logic. The schematic diagrams of each layer are as follows:

  • API Gateway

Still using go micro service? Come and try go zero, it's beyond your imagination!

  • RPC

Still using go micro service? Come and try go zero, it's beyond your imagination!

  • model

Still using go micro service? Come and try go zero, it's beyond your imagination!

Let’s go through the whole process of quickly building microservices, let’sGo!?‍♂️

4. Preparation

  • Install etcd, mysql, redis
  • Install goctl tool

    GO111MODULE=on GOPROXY=https://goproxy.cn/,direct go get -u github.com/tal-tech/go-zero/tools/goctl
  • Create working directoryshorturl
  • stayshorturlExecute under directorygo mod init shorturlinitializationgo.mod

5. Write API gateway code

  • Generated by goctlapi/shorturl.apiAnd edit, for the sake of brevity, remove the beginning of the fileinfoThe code is as follows:

    type (
        expandReq struct {
            shorten string `form:"shorten"`
        }
    
        expandResp struct {
            url string `json:"url"`
        }
    )
    
    type (
        shortenReq struct {
            url string `form:"url"`
        }
    
        shortenResp struct {
            shorten string `json:"shorten"`
        }
    )
    
    service shorturl-api {
        @server(
            handler: ShortenHandler
        )
        get /shorten(shortenReq) returns(shortenResp)
    
        @server(
            handler: ExpandHandler
        )
        get /expand(expandReq) returns(expandResp)
    }

    The usage of type is the same as that of go. Service is used to define API requests such as get / post / head / delete

    • service shorturl-api {This line defines the service name
    • @serverPart is used to define the properties used by the server
    • handlerThe name of server handler is defined
    • get /shorten(shortenReq) returns(shortenResp)The route, request parameters and return parameters of get method are defined
  • Using goctl to generate API gateway code

    goctl api go -api shorturl.api -dir .

    The structure of the generated file is as follows:

    .
    ├── api
    │   ├── etc
    │     │     └ -- shorturl-api.yaml // configuration file
    │   ├── internal
    │   │   ├── config
    │     │     │     └ -- config.go // define configuration
    │   │   ├── handler
    │     │     │     Expandhandler.go // implement expandhandler
    │     │     │     Route.go // define route processing
    │     │     │     └ -- shortenhandler.go // implement shortenhandler
    │   │   ├── logic
    │     │     │     Design -- expandlogic.go // implement expandlogic
    │     │     │     └ -- shortenlogic.go // implement shortenlogic
    │   │   ├── svc
    │     │     │     └ -- servicecontext.go // define servicecontext
    │   │   └── types
    │     │         └ -- types.go // define request and return structure
    │   ├── shorturl.api
    │     └ -- shorturl.go // main entry definition
    ├── go.mod
    └── go.sum
  • Start API gateway service and listen on port 8888 by default

    go run shorturl.go -f etc/shorturl-api.yaml
  • Testing API gateway service

    curl -i "http://localhost:8888/shorten?url=http://www.xiaoheiban.cn"

    Return as follows:

    HTTP/1.1 200 OK
    Content-Type: application/json
    Date: Thu, 27 Aug 2020 14:31:39 GMT
    Content-Length: 15
    
    {"shortUrl":""}

We can see that our API gateway did nothing but returned a null value. Next, we will implement the business logic in the RPC service

  • It can be modifiedinternal/svc/servicecontext.goTo pass Service Dependencies (if needed)
  • The implementation logic can be modifiedinternal/logicCorresponding files under
  • Can passgoctlGenerate API call codes of various client languages
  • Here, you can generate the client code through goctl for the client students to develop in parallel. It supports multiple languages. See the document for details

6. Write transform RPC service

  • stayrpc/transformWrite under the directorytransform.protofile

    Proto file template can be generated by command

    goctl rpc template -o transform.proto

    The contents of the revised document are as follows:

    syntax = "proto3";
    
    package transform;
    
    message expandReq {
        string shorten = 1;
    }
    
    message expandResp {
        string url = 1;
    }
    
    message shortenReq {
        string url = 1;
    }
    
    message shortenResp {
        string shorten = 1;
    }
    
    service transformer {
        rpc expand(expandReq) returns(expandResp);
        rpc shorten(shortenReq) returns(shortenResp);
    }
  • usegoctlGenerate RPC code, in therpc/transformExecute command under directory

    goctl rpc proto -src transform.proto

    The file structure is as follows:

    rpc/transform
    ├── etc
    │     └ -- transform.yaml // configuration file
    ├── internal
    │   ├── config
    │     │     └ -- config.go // configuration definition
    │   ├── logic
    │     │     This is where the expandlogic.go // expand business logic is implemented
    │     │     └ - shortenlogic.go // shorten business logic is implemented here
    │   ├── server
    │     │     └ -- transformerserver.go // call entry, no need to modify
    │   └── svc
    │         └ -- servicecontext.go // define servicecontext and pass dependencies
    ├── pb
    │   └── transform.pb.go
    Transform.go // RPC service main function
    ├── transform.proto
    └── transformer
        Transformer. Go // provides external calling methods, which need not be modified
        ├── transformer_ Mock.go // mock method for testing
        └ -- structure definition of types.go // request / response

    It can run directly as follows:

    $ go run transform.go -f etc/transform.yaml
    Starting rpc server at 127.0.0.1:8080...

    etc/transform.yamlFile can be modified listening port configuration

7. Modify API gateway code to call transform RPC service

  • Modify configuration fileshorturl-api.yaml, add the following

    Transform:
      Etcd:
        Hosts:
          localhost:2379
        Key: transform.rpc

    Note: the MD support of this website is not friendly, localhost:2379 The original is- localhost:2379

  • modifyinternal/config/config.goAs follows, add the transform service dependency

    type Config struct {
        rest.RestConf
        Transform rpcx.rpcclientconf // manual code
    }
  • modifyinternal/svc/servicecontext.go, as follows:

    type ServiceContext struct {
        Config    config.Config
        Transformer. Transformer // manual code
    }
    
    func NewServiceContext(c config.Config) *ServiceContext {
        return &ServiceContext{
            Config:    c,
        Transformer: transformer. Newtransformer (rpcx. Mustnewclient (C. transform)), // manual code
        }
    }

    Passing dependencies between different business logics through servicecontext

  • modifyinternal/logic/expandlogic.goInsideExpandThe methods are as follows

    func (l *ExpandLogic) Expand(req types.ExpandReq) (*types.ExpandResp, error) {
      //Manual code start
        resp, err := l.svcCtx.Transformer.Expand(l.ctx, &transformer.ExpandReq{
            Shorten: req.Shorten,
        })
        if err != nil {
            return nil, err
        }
    
        return &types.ExpandResp{
            Url: resp.Url,
        }, nil
      //End of manual code
    }

By callingtransformerOfExpandMethod to realize short chain recovery to URL

  • modifyinternal/logic/shortenlogic.go, as follows:

    func (l *ShortenLogic) Shorten(req types.ShortenReq) (*types.ShortenResp, error) {
      //Manual code start
        resp, err := l.svcCtx.Transformer.Shorten(l.ctx, &transformer.ShortenReq{
            Url: req.Url,
        })
        if err != nil {
            return nil, err
        }
    
        return &types.ShortenResp{
            Shorten: resp.Shorten,
        }, nil
      //End of manual code
    }

By callingtransformerOfShortenMethod to realize the transformation from URL to short chain

So far, the API gateway has been modified. Although there are a lot of code pasted, only a small part of it has been modified in the interim. In order to understand the context easily, I pasted the complete code, and then deal with crud + cache

8. Define database table structure and generate crud + cache code

  • Short URLrpc/transform/modelcatalog:mkdir -p rpc/transform/model
  • Write SQL file to create shorturl table in RPC / transform / model directoryshorturl.sql, as follows:

    CREATE TABLE `shorturl`
    (
      `shorten` varchar(255) NOT NULL COMMENT 'shorten key',
      `url` varchar(255) NOT NULL COMMENT 'original url',
      PRIMARY KEY(`shorten`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
  • Create dB and table

    create database gozero;
    source shorturl.sql;
  • stayrpc/transform/modelExecute the following command under the directory to generate crud + cache code,-cIndicates useredis cache

    goctl model mysql ddl -c -src shorturl.sql -dir .

    It can also be useddatasourceCommand substitutionddlTo specify that the database link is generated directly from the schema

    The generated file structure is as follows:

    rpc/transform/model
    ├── shorturl.sql
    Java -- shorturlmodel. Go // crud + cache code
    └ -- vars.go // define constants and variables

9. Modify the shorten / expand RPC code to call the crud + cache code

  • modifyrpc/transform/etc/transform.yaml, add the following:

    DataSource: root:@tcp(localhost:3306)/gozero
    Table: shorturl
    Cache:
      Host: localhost:6379

    be careful:Host: localhost:6379The original is- Host: localhost:6379In order to avoid the unfriendly problem of MD support
    Multiple redis can be used as cache to support redis single point or redis cluster

  • modifyrpc/transform/internal/config.go, as follows:

    type Config struct {
        rpcx.RpcServerConf
        Datasource string // manual code
        Table string // manual code
        Cache cache.cacheconf // manual code
    }

    Added MySQL and redis cache configuration

  • modifyrpc/transform/internal/svc/servicecontext.go, as follows:

    type ServiceContext struct {
        c     config.Config
      Model * model.shorturlmodel // manual code
    }
    
    func NewServiceContext(c config.Config) *ServiceContext {
        return &ServiceContext{
            c:             c,
            Model: model. Newshorturlmodel (sqlx. Newmysql (C. datasource), C. cache, C. table), // manual code
        }
    }
  • modifyrpc/transform/internal/logic/expandlogic.go, as follows:

    func (l *ExpandLogic) Expand(in *expand.ExpandReq) (*expand.ExpandResp, error) {
        //Manual code start
        res, err := l.svcCtx.Model.FindOne(in.Shorten)
        if err != nil {
            return nil, err
        }
    
        return &transform.ExpandResp{
            Url: res.Url,
        }, nil
        //End of manual code
    }
  • modifyrpc/shorten/internal/logic/shortenlogic.go, as follows:

    func (l *ShortenLogic) Shorten(in *shorten.ShortenReq) (*shorten.ShortenResp, error) {
      //Manual code start, generate short links
        key := hash.Md5Hex([]byte(in.Url))[:6]
        _, err := l.svcCtx.Model.Insert(model.Shorturl{
            Shorten: key,
            Url:     in.Url,
        })
        if err != nil {
            return nil, err
        }
    
        return &transform.ShortenResp{
            Shorten: key,
        }, nil
      //End of manual code
    }

    At this point, the code modification is completed, and I added notes to the manual modification code

10. Full call demonstration

  • Short API call

    curl -i "http://localhost:8888/shorten?url=http://www.xiaoheiban.cn"

    Return as follows:

    HTTP/1.1 200 OK
    Content-Type: application/json
    Date: Sat, 29 Aug 2020 10:49:49 GMT
    Content-Length: 21
    
    {"shorten":"f35b2a"}
  • Expand API call

    curl -i "http://localhost:8888/expand?shorten=f35b2a"

    Return as follows:

    HTTP/1.1 200 OK
    Content-Type: application/json
    Date: Sat, 29 Aug 2020 10:51:53 GMT
    Content-Length: 34
    
    {"url":"http://www.xiaoheiban.cn"}

11. Benchmark

Because writing depends on the writing speed of MySQL, it is equivalent to pressing mysql. Therefore, the pressure test only tests the expand interface, which is equivalent to reading from MySQL and using the cache. In short.lua, 100 hot keys are randomly obtained from DB to generate the pressure test request

Still using go micro service? Come and try go zero, it's beyond your imagination!

It can be seen that I can achieve 30000 + QPS on my MacBook Pro.

12. Complete code

https://github.com/tal-tech/go-zero/tree/master/example/shorturl

12. Summary

We have always stressed thatTools and documentation

Go zero is not only a framework, but also a technical system based on framework + tools, which simplifies and standardizes the whole microservice construction.

While keeping it simple, we try our best to encapsulate the complexity of microservice governance into the framework, which greatly reduces the mental burden of developers and makes business development fast.

The code generated by go zero + goctl contains various components of microservice governance, including concurrency control, adaptive fusing, adaptive load shedding, automatic cache control, etc., which can be easily deployed to carry huge traffic.

If you have any good idea to improve the efficiency of the project, you are welcome to communicate at any time!