The lightweight distributed log tracking tool can be accessed in ten minutes, and there is no difficulty in log tracking from now on

Time:2021-9-14

preface

With the popularity of microservices, many companies have disassembled the system into many microservices according to business boundaries, when troubleshooting and checking logs. Because the service link runs through many micro service nodes, it will be difficult to locate the log of a request and the log of upstream and downstream services.

At this time, many children’s shoes will start to consider using distributed tracking systems such as skywalking and pinpoint. They are based on the opentracking specification, are usually non-invasive, and have a relatively friendly management interface to query the link span.

However, it takes a certain time period to build a distributed tracking system, get familiar with and promote the system to the whole company, and it involves the storage cost of link span nodes. Is it full or partial acquisition? For full collection, take skywalking storage as an example. At least 5 nodes are required to build an ES cluster. This requires an increase in server costs. Moreover, if there are many microservice nodes, it is normal to generate tens of gigabytes and hundreds of gigabytes of data a day. If you want to save for a long time, you also need to increase the cost of server disk.

Therefore, the open source project introduced this time is tlog. Its background stems from the desire to make a lightweight log tracking solution, covering the current popular log framework and RPC framework, so that users can access it at the least cost to solve the pain point of log tracking in the microservice system.

Of course, distributed tracking system is the final solution. If your company has adopted distributed tracking system, tlog is not applicable.

Tlog provides the simplest way to solve the problem of log tracking. It does not collect logs and does not need additional storage space. It just automatically labels your business log and automatically generates traceid throughout the whole link of your microservice. And provide upstream and downstream node information. It is suitable for small and medium-sized enterprises and companies that want to quickly solve the problem of log tracking.

Therefore, I have adapted three log frameworks for tlog to support automatic detection and adaptation. Support three RPC frameworks: Dubbo, dubbox and spring cloud. More importantly, your project may not need to access tlog for ten minutes:)

Tlog project and features

Project address:https://gitee.com/bryan31/TLog

The lightweight distributed log tracking tool can be accessed in ten minutes, and there is no difficulty in log tracking from now on

Tlog has the following features:

  • Complete lightweight microservice log tracking by labeling logs
  • No intrusive design for service code, easy to use, and can be accessed in 10 minutes
  • It supports three common log frameworks: log4j, log4j2 and logback, and provides automatic detection to complete adaptation
  • Supports three RPC frameworks: Dubbo, dubbox and springcloud
  • Supports the configuration of custom templates for log labels
  • Almost no performance loss

How to access quickly

Tlog supports the automatic assembly of springboot. In the springboot environment, you can access it in the following two steps!

rely on

<dependency>
  <groupId>com.yomahub</groupId>
  <artifactId>tlog-all-spring-boot-starter</artifactId>
  <version>1.0.0</version>
</dependency>

At present, the jar package has been uploaded to the central warehouse and can be directly relied on

Log frame adaptation

Just add a line of code to your startup class, which can automatically detect and enhance the log framework used by your project. At present, it supports three log frameworks: log4j, log4j2 and logback.

@SpringBootApplication
public class Runner {

    Static {aspectlogenhance. Enhance();} // perform log enhancement and automatically determine the log framework

    public static void main(String[] args) {
        SpringApplication.run(Runner.class, args);
    }
}

It should be noted that because this is implemented with javassit, bytecode enhancement needs to be performed before the JVM loads the classes corresponding to the log framework. So the static block is used here. However, in the following two cases, the automatic adaptation of one line may not take effect:

1. Add the log definition to the startup class of springboot / spring. This situation will not take effect because the log framework has been loaded by classload before loading the static block.

2. You started with external containers such as Tomcat / JBoss / Jetty (no impact on the spring boot built-in container)

For these two cases, tlog also provides separate adaptation of the three log frameworks. Just replace the corresponding layout class in the log configuration file, which will also be introduced below

Adaptation of RPC framework

In the springboot environment, tlog will automatically detect the RPC framework you use and adapt it automatically.

After completing the above two steps, you are OK. The final effect is as follows (take Dubbo + log4j as an example):

Consumer code:

The lightweight distributed log tracking tool can be accessed in ten minutes, and there is no difficulty in log tracking from now on

Log printing:

2020-09-16 18:12:56748 [warn] [tlog] regenerate traceid [7161457983341056] > > com.yomahub.tlog.web.tlogwebinterceptor: 39
2020-09-16 18:12:56,763 [INFO] <7161457983341056> logback-dubbox-consumer:invoke method sayHello,name=jack  >> com.yomahub.tlog.example.dubbox.controller.DemoController:22
2020-09-16 18:12:56763 [info] < 7161457983341056 > test log AAAA > > com.yomahub.tlog.example.dubbox.controller.democontroller: 23
2020-09-16 18:12:56763 [info] < 7161457983341056 > test log BBBB > > com.yomahub.tlog.example.dubbox.controller.democontroller: 24

Provider code:

The lightweight distributed log tracking tool can be accessed in ten minutes, and there is no difficulty in log tracking from now on

Log printing:

2020-09-16 18:12:56,854 [INFO] <7161457983341056> logback-dubbox-provider:invoke method sayHello,name=jack  >> com.yomahub.tlog.example.dubbo.service.impl.DemoServiceImpl:15
2020-09-16 18:12:56854 [info] < 7161457983341056 > test log CCCC > > com.yomahub.tlog.example.dubbo.service.impl.demoserviceimpl: 16
2020-09-16 18:12:56854 [info] < 7161457983341056 > test log dddd > > com.yomahub.tlog.example.dubbo.service.impl.demoserviceimpl: 17

It can be seen that after simple access, each request between micro services has a globally unique traceid running through it, which can take effect for all log outputs. Therefore, it becomes easier to locate the log chain of a request.

Change the format of log labels

Tlog allows users to customize the format of log labels. By default, tlog only prints traceid in the template of < $traceid >. Of course, you can customize its template. Other tag headers can also be added

You only need to define the following in the application.properties of springboot, change the format of the label, and you can print according to the template you define

tlog.pattern=[$preApp][$preIp][$traceId]

$preApp: name of upstream micro service node

$preIp: IP address of upstream microservice

$traceId: globally unique trace ID

Separate adaptation of log framework

If your automatic log detection fails or you use an external container, you need to modify the log framework configuration in your project. The modification method is also very simple.

Log4j profile enhancements

Just putlayoutJust replace the implementation class

The log4j templates of each company are similar. Only XML examples are given here

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration>
    <appender name="stdout" class="org.apache.log4j.ConsoleAppender">
        <!-- Replace with aspectlog4jpatternlayout -- >
        <layout class="com.yomahub.tlog.core.enhance.log4j.AspectLog4jPatternLayout">
            <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss,SSS} [%p] %m  >> %c:%L%n"/>
        </layout>
    </appender>
    <appender name="fileout" class="org.apache.log4j.DailyRollingFileAppender">
        <param name="File" value="./logs/test.log"/>
        <!-- Replace with aspectlog4jpatternlayout -- >
        <layout class="com.yomahub.tlog.core.enhance.log4j.AspectLog4jPatternLayout">
            <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss,SSS} [%p] %m  >> %c:%L%n"/>
        </layout>
    </appender>
    <root>
        <priority value="info" />
        <appender-ref ref="stdout"/>
        <appender-ref ref="fileout"/>
    </root>
</log4j:configuration>

Profile enhancements for logback

changeencoderOr replace the implementation classlayoutThe implementation class is OK

The following is an example of XML:

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
    <property name="APP_NAME" value="logtest"/>
    <property name="LOG_HOME" value="./logs" />
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <!-- Replace with aspectlogbackencoder -- >
        <encoder class="com.yomahub.tlog.core.enhance.logback.AspectLogbackEncoder">
              <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
        </encoder>
    </appender>
    <appender name="FILE"  class="ch.qos.logback.core.rolling.RollingFileAppender">
        <File>${LOG_HOME}/${APP_NAME}.log</File>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <FileNamePattern>${LOG_HOME}/${APP_NAME}.log.%d{yyyy-MM-dd}.%i.log</FileNamePattern>
            <MaxHistory>30</MaxHistory>
            <maxFileSize>1000MB</maxFileSize>
        </rollingPolicy>
        <!-- Replace with aspectlogbackencoder -- >
        <encoder class="com.yomahub.tlog.core.enhance.logback.AspectLogbackEncoder">
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
        </encoder>
    </appender>

    <!--  Log output level -- >
    <root level="INFO">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="FILE" />
    </root>
</configuration>

Profile enhancements for log4j2

Since log4j2 is implemented in the form of plug-ins, log4j2 has the function of automatically scanning plug-ins. Therefore, no changes need to be made to the configuration file to take effect.

Post language

Tlog is essentially a logging framework, but it has the log tracking function supporting RPC framework. It took me about a week to write the framework and strive to achieve the lightest log tracking solution so that users can access it at the least cost. Tlog not only supports springboot, but also supports non springboot projects. For specific configuration methods, please see the document on the project home page.

I also wrote a detailed example project for tlog, covering almost all scenarios involved. The specific example project address is also on the project home page.

The lightweight distributed log tracking tool can be accessed in ten minutes, and there is no difficulty in log tracking from now on

If you also encounter log tracking difficulties in the project, you might as well try tlog. I hope this open source framework can help you. Open source is not easy. If you like, please help star. Your support is the biggest driving force for me to move forward. If you have any questions, please contact me.

Focus on the author

“Yuanren tribe” is an original technology sharing number. I hope you can pay attention to me. I will publish a practical original technology article every week to accompany you and grow up with you. Pay attention to official account replytlogIf I can join the group chat, I will patiently answer every question you have in use, and I will maintain and iterate the project for a long time.

The lightweight distributed log tracking tool can be accessed in ten minutes, and there is no difficulty in log tracking from now on