Opentracing and Jaeger in a real Go Microservices

Time:2020-8-17

background

Microservices have greatly changed the mode of software development and delivery. Single application is divided into multiple microservices, and the complexity of a single service is greatly reduced. The dependence between libraries is also transformed into the dependency between services. The problem is that the granularity of deployment becomes more and more fine, and many services bring great pressure on operation and maintenance. Fortunately, we have kubernetes, which can solve most of the operation and maintenance problems.

With the increase in the number of services and the complexity of the internal call chain, it is difficult to achieve “see the whole picture” only by means of log and performance monitoring. In the process of troubleshooting or performance analysis, it is no different from being blind and feeling the elephant. Distributed tracing can help developers intuitively analyze the request link, quickly locate the performance bottleneck, and gradually optimize the dependence between services. It also helps developers better understand the whole distributed system from a more macro perspective.

The distributed tracking system is divided into three parts: data acquisition, data persistence and data display. Data collection refers to embedding points in the code, setting the stage to be reported in the request, and setting which superior stage the current record belongs to. Data persistence refers to storing the reported data on disk. For example, Jaeger supports a variety of storage back ends, and Cassandra or elasticsearch can be selected. Data display is the front-end queries the request phase associated with it according to the trace ID, and presents it on the interface.
Opentracing and Jaeger in a real Go Microservices

Microservice communication architecture

Opentracing

History of development

As early as 2005, Google deployed a set of distributed tracking system, dapper, and published a paper “dapper, a large scale distributed systems tracking infrastructure”, which described the design and implementation of the distributed tracking system, which can be regarded as the originator of distributed tracking field. Then there are open source implementations inspired by this, such as Zipkin, appdash of sourcegraph, hawkurar APM of red hat, Jaeger of Uber, etc. However, the distributed tracking schemes are incompatible with each other, which leads to the birth of opentracing. Opentracing is a library, which defines a set of common data reporting interface, which requires each distributed tracking system to implement this interface. In this way, the application only needs to connect with opentracing, and does not need to care about what kind of distributed tracking system is used in the back-end. Therefore, developers can switch the distributed tracking system seamlessly, and it is possible to add support for distributed tracking in the common code base.

At present, the mainstream distributed tracking implementations have basically supported opentracing, including Jaeger, Zipkin, appdash, etc. for details, please refer to the official document supported tracer implementation.

data model

This part is written very clearly in the opentracing specification. The key parts are only translated. For details, please refer to the original document the opentracing semantic specification.

Causal relationships between Spans in a single Trace

        [Span A]  ←←←(the root span)
            |
     +------+------+
     |             |
 [Span B]      [Span C] ←←←(Span C is a `ChildOf` Span A)
     |             |
 [Span D]      +---+-------+
               |           |
           [Span E]    [Span F] >>> [Span G] >>> [Span H]
                                       ↑
                                       ↑
                                       ↑
                         (Span G `FollowsFrom` Span F)

Trace is a call chain, and each call chain consists of multiple spans. The word meaning of span is scope, which can be understood as a processing stage. The relationship between span and span is called reference. In the figure above, there are 8 stages labeled a-h.

Temporal relationships between Spans in a single Trace

––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–> time

 [Span A···················································]
   [Span B··············································]
      [Span D··········································]
    [Span C········································]
         [Span E·······]        [Span F··] [Span G··] [Span H··]

The diagram above shows the call chain in chronological order.

Each span contains the following states:

  • Operation name
  • Starting time
  • End time
  • A set of kV values as span Tags
  • Span logs
  • Span context, which contains trace ID and span ID
  • References

A span can have two referential relationships: childof and followsfrom. Childof is used to represent the parent-child relationship, that is, another stage occurs in a certain stage. It is the most common phase relationship. Typical scenarios include calling RPC interface, executing SQL, and writing data. Followsfrom represents a following relationship, which means that another stage occurs after a certain stage, and is used to describe the sequential execution relationship.

ChildOf relationship means that the rootSpan has a logical dependency on the child span before rootSpan can complete its operation. Another standard reference type in OpenTracing is FollowsFrom, which means the rootSpan is the ancestor in the DAG, but it does not depend on the completion of the child span, for example if the child represents a best-effort, fire-and-forget cache write.

Concepts and terminology

Traces

A trace represents a potential, distributed system with parallel data or parallel execution trajectories (potentially distributed, parallel). A trace can be considered as a DAG with multiple spans.

Spans

A span represents the logical running unit with start time and execution time. The logical causal relationship between span is established by nesting or sequential arrangement.

Operation Names

Each span has an operation name, which is simple and readable. (for example, the name of an RPC method, a function name, or a subtask or stage in a large calculation). The operation name of span should be an abstract and general identification, which can be clear and statistically significant. For more specific description of subtypes, please use tags
For example, suppose a span that gets account information has the following possible names:

| Operation Name | Guidance |
|:—————|:——–|
| get | Too general |
| get_account/792 | Too specific |
| get_account | Good, and account_id=792 would make a nice Span tag |

References between Spans

A Span may reference zero or more other SpanContexts that are causally related. OpenTracing presently defines two types of references: ChildOf and FollowsFrom. Both reference types specifically model direct causal relationships between a child Span and a parent Span. In the future, OpenTracing may also support reference types for Spans with non-causal relationships (e.g., Spans that are batched together, Spans that are stuck in the same queue, etc).

ChildOf references: A Span may be the ChildOf a parent Span. In a ChildOf reference, the parent Span depends on the child Span in some capacity. All of the following would constitute ChildOf relationships:

  • A Span representing the server side of an RPC may be the ChildOf a Span representing the client side of that RPC
  • A Span representing a SQL insert may be the ChildOf a Span representing an ORM save method
  • Many Spans doing concurrent (perhaps distributed) work may all individually be the ChildOf a single parent Span that merges the results for all children that return within a deadline

These could all be valid timing diagrams for children that are the ChildOf a parent.

    [-Parent Span---------]
         [-Child Span----]

    [-Parent Span--------------]
         [-Child Span A----]
          [-Child Span B----]
        [-Child Span C----]
         [-Child Span D---------------]
         [-Child Span E----]

FollowsFrom references: Some parent Spans do not depend in any way on the result of their child Spans. In these cases, we say merely that the child Span FollowsFrom the parent Span in a causal sense. There are many distinct FollowsFrom reference sub-categories, and in future versions of OpenTracing they may be distinguished more formally.

These can all be valid timing diagrams for children that “FollowFrom” a parent.

    [-Parent Span-]  [-Child Span-]


    [-Parent Span--]
     [-Child Span-]


    [-Parent Span-]
                [-Child Span-]

Logs

Each span can perform multiple log operations. Each log operation requires a time name with a timestamp and an optional storage structure of any size.
Some common use cases of logging operations and key values of related log events are defined in the standard. Please refer to the data conventions guidelines for reference.

Tags

Each span can have multiple key value pairs( key:value )In the form of tags, tags have no time stamp and support simple annotation and supplement to span.
As in the case of logs, tracer can pay special attention to tags of key value pairs known in application specific scenarios. For more information, refer to the data conventions guidelines.

SpanContext

Each span must provide a method to access the spancontext. Spancontext represents the state passed across the process boundary to the lower span. (for example, include < trace_ id, span_ ID, sampled > tuple), and is used to encapsulate bag (see below for an explanation of bag). Spancontext is used when crossing process boundaries and creating boundaries in trace graphs. (childof relationship or other relationship, refer to span relationship).

Baggage

Bag is a collection of key value pairs (spancontext) stored in spancontext. It will be transmitted globally in all spans on a tracking link, including the spancontexts corresponding to these spans. In this case, “Baggage” will be transmitted along with trace, so he gets the name (Baggage can be simultaneous interpreting the luggage transported with the trace running process). In view of the need of full stack opentracing integration, bag realizes powerful functions by transparently transmitting data of any application program. For example, a bag element can be added to the mobile terminal of the end user, which is passed to the storage layer through the distributed tracking system, and then the call stack is constructed in reverse, which consumes a lot of SQL query statements in the positioning process.

Bag has powerful functions, but also has a lot of consumption. Due to the global transmission of packet, if the number of packets is too large or too many elements are included, it will reduce the throughput of the system or increase the latency of RPC.

Baggage vs. Span Tags

  • Bag transfers data across processes (along with business system calls) globally. Tags of span are not transmitted because they are not inherited by child span.
  • The tag of span can be used to record business-related data and store it in the tracking system. When opentracing is implemented, you can choose whether to store non business data in bag, which is not required by opentracing standard.

Inject and Extract

In other words, it is possible to obtain the data from the carrier (e.g. by adding a header to the carrier). In this way, spancontexts can cross process boundaries and provide enough information to establish inter process relationships (thus enabling continuous tracing across processes).

Global and No-op Tracers

Each platform’s opentracing API library (opentracing go, opentracing Java, etc.) must implement an empty tracer. The implementation of no OP tracer must be error free and has no side effects. In this way, when the business side does not specify the collector service, storage, and initialize the global tracer, but RPC components, ORM components or other components add probes. In this way, the global default is no OP tracer instance, which will not have any impact on the business.

jaeger

framework

Jaeger can be deployed either as all-in-one binary, where all Jaeger backend components run in a single process, or as a scalable distributed system, discussed below. There two main deployment options:

Collectors are writing directly to storage.
Collectors are writing to Kafka as a preliminary buffer.

Opentracing and Jaeger in a real Go Microservices

Illustration of direct-to-storage architecture

Opentracing and Jaeger in a real Go Microservices

Illustration of architecture with Kafka as intermediate buffer

An instrumented service creates spans when receiving new requests and attaches context information (trace id, span id, and baggage) to outgoing requests. Only ids and baggage are propagated with requests; all other information that compose a span like operation name, logs, etc. are not propagated. Instead sampled spans are transmitted out of process asynchronously, in the background, to Jaeger Agents.

The instrumentation has very little overhead, and is designed to be always enabled in production.

Note that while all traces are generated, only a few are sampled. Sampling a trace marks the trace for further processing and storage. By default, Jaeger client samples 0.1% of traces (1 in 1000), and has the ability to retrieve sampling strategies from the agent.

Agent

The Jaeger agent is a network daemon that listens for spans sent over UDP, which it batches and sends to the collector. It is designed to be deployed to all hosts as an infrastructure component. The agent abstracts the routing and discovery of the collectors away from the client.

Collector

The Jaeger collector receives traces from Jaeger agents and runs them through a processing pipeline. Currently our pipeline validates traces, indexes them, performs any transformations, and finally stores them.

Jaeger’s storage is a pluggable component which currently supports Cassandra, Elasticsearch and Kafka.

Query

Query is a service that retrieves traces from storage and hosts a UI to display them.

Ingester

Ingester is a service that reads from Kafka topic and writes to another storage backend (Cassandra, Elasticsearch).

deploy

Agent

Jaeger client libraries expect jaeger-agent process to run locally on each host.

It can be executed directly on the host or via Docker, as follows:

## make sure to expose only the ports you use in your deployment scenario!
docker run \
  --rm \
  -p6831:6831/udp \
  -p6832:6832/udp \
  -p5778:5778/tcp \
  -p5775:5775/udp \
  jaegertracing/jaeger-agent:1.12

The agents can connect point to point to a single collector address, which could be load balanced by another infrastructure component (e.g. DNS) across multiple collectors. The agent can also be configured with a static list of collector addresses.

On Docker, a command like the following can be used:

docker run \
  --rm \
  -p5775:5775/udp \
  -p6831:6831/udp \
  -p6832:6832/udp \
  -p5778:5778/tcp \
  jaegertracing/jaeger-agent:1.12 \
  --reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250

When using gRPC, you have several options for load balancing and name resolution:

  • Single connection and no load balancing. This is the default if you specify a single host:port. (example: –reporter.grpc.host-port=jaeger-collector.jaeger-infra.svc:14250)
  • Static list of hostnames and round-robin load balancing. This is what you get with a comma-separated list of addresses. (example: reporter.grpc.host-port=jaeger-collector1:14250,jaeger-collector2:14250,jaeger-collector3:14250)
  • Dynamic DNS resolution and round-robin load balancing. To get this behaviour, prefix the address with dns:/// and gRPC will attempt to resolve the hostname using SRV records (for external load balancing), TXT records (for service configs), and A records. Refer to the gRPC Name Resolution docs and the dns_resolver.go implementation for more info. (example: –reporter.grpc.host-port=dns:///jaeger-collector.jaeger-infra.svc:14250)

Collectors

The collectors are stateless and thus many instances of jaeger-collector can be run in parallel. Collectors require almost no configuration, except for the location of Cassandra cluster, via –cassandra.keyspace and –cassandra.servers options, or the location of Elasticsearch cluster, via –es.server-urls, depending on which storage is specified. To see all command line options run

go run ./cmd/collector/main.go -h

or, if you don’t have the source code

docker run -it --rm jaegertracing/jaeger-collector:1.12 -h

Storage Backends

Collectors require a persistent storage backend. Cassandra and Elasticsearch are the primary supported storage backends.

The storage type can be passed via SPAN_STORAGE_TYPE environment variable. Valid values are cassandra, elasticsearch, kafka (only as a buffer), grpc-plugin, badger (only with all-in-one) and memory (only with all-in-one).

Elasticsearch

Supported in Jaeger since 0.6.0 Supported versions: 5.x, 6.x

Elasticsearch does not require initialization other than installing and running Elasticsearch. Once it is running, pass the correct configuration values to the Jaeger collector and query service.

Configuration

Minimal

docker run \
  -e SPAN_STORAGE_TYPE=elasticsearch \
  -e ES_SERVER_URLS=<...> \
  jaegertracing/jaeger-collector:1.12

To view the full list of configuration options, you can run the following command:

docker run \
  -e SPAN_STORAGE_TYPE=elasticsearch \
  jaegertracing/jaeger-collector:1.12 \
  --help

more info

Microservice framework access to opentracing process

A microservice framework consists of two parts, HTTP (GIN) & grpc, which provides rest externally and grpc services internally.

Microservice software framework:

Opentracing and Jaeger in a real Go Microservices

The following is the general process of microservice framework accessing opentracing.

Create a tracer for each HTTP request

tracer, closer := tracing.Init("hello-world")
defer closer.Close()
opentracing.SetGlobalTracer(tracer)

Create a span. If there are trace and span information in the HTTP header, get it from the header. Otherwise, create a new one.

spanCtx, _ := tracer.Extract(opentracing.HTTPHeaders, opentracing.HTTPHeadersCarrier(r.Header))
span := tracer.StartSpan("format", ext.RPCServerOption(spanCtx))

defer span.Finish()

//When span is written into context, CTX needs to be passed in internal call of function, or CTX needs to be passed between span.
ctx := opentracing.ContextWithSpan(context.Background(), span)

In process of HTTP / grpc service function

span, _ := opentracing.StartSpanFromContext(ctx, "formatString")
defer span.Finish()


//Cross process calls, such as calling a rest API, need to inject the span information into the HTTP header.
// tracing.InjectToHeaders(ctx, "GET", url, req.Header)
func InjectToHeaders(ctx context.Context, method string, url string, header http.Header) {
    span := opentracing.SpanFromContext(ctx)
    if span != nil {
        ext.SpanKindRPCClient.Set(span)
        ext.HTTPUrl.Set(span, url)
        ext.HTTPMethod.Set(span, "GET")
        span.Tracer().Inject(
            span.Context(),
            opentracing.HTTPHeaders,
            opentracing.HTTPHeadersCarrier(header),
        )
    }
}

span.LogFields(
        log.String("event", "string-format"),
        log.String("value", helloStr),
)

How to bury it

Gin framework

Router buried point

On each HTTP routing method that needs to track requests, add the tracing.NewSpan “Function.

import ".../go_common/tracing"

...

authorized := r.Group("/v1")
authorized.Use(handlers.TokenCheck, handlers.MustLogin())
{
    authorized.GET("/user/:id", handlers.GetUserInfo)
    authorized.GET("/user", handlers.GetUserInfoByToken)
    authorized.PUT("/user/:id", tracing.NewSpan("put /user/:id", "handlers.Setting", false), handlers.Setting)
}

Parameter description

NewSpan(service string, operationName string, abortOnErrors bool, opts …opentracing.StartSpanOption)

service generally fill with the endpoint of api.
operationName can be filled with HandleFunc’s name.

Buried point of handler function

func Setting(c *gin.Context) {
    ...
    //To obtain span; from gin context, you must bury a point!
    span, found := tracing.GetSpan(c)

    //Add tag and log
    if found == true && span != nil {
        span.SetTag("req", req)
        span.LogFields(
            log.Object("uid", uid),
        )
    }

    // opentracing.ContextWithSpan , bind span to context; in handler function, this place must be buried.
    ctx, cancel := context.WithTimeout(opentracing.ContextWithSpan(context.Background(), span), time.Second*3)
    defer cancel()

    //Call by grpc; this one doesn't need special treatment
    auth := passportpb.Authentication{
        LoginToken: c.GetHeader("Qsc-Peduli-Token"),
    }
    cli, _ := passportpb.Dial(ctx, grpc.WithPerRPCCredentials(&auth))
    reply, err := cli.Setting(ctx, req)

    //Directly call by local RPC method; this does not need special handling
    ctx = metadata.AppendToOutgoingContext(ctx, "logintoken", c.GetHeader("Qsc-Peduli-Token"))
    reply, err := rpc.Srv.Setting(ctx, req)

    ...
}

GRPC

Grpc CLIENT SDK

import "github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing"

...

// Dial grpc server
func (c *Client) Dial(serviceName string, opts ...grpc.DialOption) (*grpc.ClientConn, error) {
    
    ...
    
    unaryInterceptor := grpc_middleware.ChainUnaryClient(
        grpc_opentracing.UnaryClientInterceptor(),
    )

    c.Dialopts = append(c.Dialopts, grpc.WithUnaryInterceptor(unaryInterceptor))

    conn, err := grpc.Dial(serviceName, c.Dialopts...)
    if err != nil {
        return nil, fmt.Errorf("Failed to dial %s: %v", serviceName, err)
    }
    return conn, nil
}

Grpc server SDK

import "github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing"

...

func NewServer(serviceName, addr string) *Server {
    var opts []grpc.ServerOption
    opts = append(opts, grpc_middleware.WithUnaryServerChain(
        grpc_opentracing.UnaryServerInterceptor(),
    ))

    srv := grpc.NewServer(opts...)
    return &Server{
        serviceName: serviceName,
        addr:        addr,
        grpcServer:  srv,
    }
}

Buried point of RPC function

func (s *Service) Setting(ctx context.Context, req *passportpb.UserSettingRequest) (*passportpb.UserSettingReply, error) {
    //If it is not a grpc call, that is, the local RPC function call mode, the span is extracted from the context.
    if !s.meta.IsGrpcRequest(ctx) {
    span, _ := opentracing.StartSpanFromContext(ctx, "rpc.srv.Setting")
    defer span.Finish()
    }

    //If there is a request for other grpc functions in the RPC function, it can be called normally, because there are trace and span information in the request context of grpc, so it can be propagated directly without additional operation.
    reqVerify := new(passportpb.VerifyRequest)
    reqVerify.UID = req.UserID
    cli, _ := passportpb.Dial(ctx)
    replyV, _ := cli.Verify(ctx, reqVerify)
}

Jaeger UI final effect

Opentracing and Jaeger in a real Go Microservices

reference resources

jaegertracing

opentracing-tutorial

grpc-opentracing

gin-opentracing

go-opentracing-guides

Chinese version of opentracking document