Log troubleshooting difficult? Distributed log link tracking to help you

Time:2019-12-6

Background

The most common way to develop and troubleshoot system problems is to check system logs, which are commonly used in distributed environmentsELKTo collect logs in a unified way, but it’s still troublesome to use log location when the concurrency is large. Because a large number of logs of other users / other threads are output together, it’s difficult to filter out all the related logs of the specified request and the logs corresponding to the downstream threads / services.

 

II. Solutions

  • One for each requestUnique identificationTo trace all links and display them in the log without modifying the original printing method (code is not intruded)
  • Using logbackMDCAdd to the mechanism log templatetraceIdIdentification, value taking method is%X{traceId}

MDC (mapped diagnostic context) is a function provided by log4j and logback, which is convenient for logging under multi-threaded conditions. MDC can be regarded as a map bound to the current thread, where you can add key value pairs. The content contained in MDC can be accessed by code executed in the same thread. The child thread of the current thread inherits the contents of the MDC in its parent thread. When you need to log, you only need to get the required information from MDC. The content of MDC is saved by the program at an appropriate time. For a web application, the data is usually saved at the beginning when the request is processed.

 

III. scheme realization

BecauseMDCInternal use isThreadLocalTherefore, only this thread can be effective. Sub threads and downstream servicesMDCSo the main difficulty of the scheme is to solveValue passing problem, mainly including the following parts:

  • In API gatewayMDCHow to transfer data to downstream services
  • How the service receives the data and continues to deliver when it calls other remote services
  • How to transfer (thread pool) to a child thread in case of asynchrony

3.1. Modify log template

Logback profile template format add identity%X{traceId}
Log troubleshooting difficult? Distributed log link tracking to help you

 

3.2. Add filter to gateway

generatetraceIdAnd deliver it to the downstream service through the header

@Component
public class TraceFilter extends ZuulFilter {
    @Autowired
    private TraceProperties traceProperties;

    @Override
    public String filterType() {
        return FilterConstants.PRE_TYPE;
    }

    @Override
    public int filterOrder() {
        return FORM_BODY_WRAPPER_FILTER_ORDER - 1;
    }

    @Override
    public boolean shouldFilter() {
        //Control whether to turn on the filter according to the configuration
        return traceProperties.getEnable();
    }

    @Override
    public Object run() {
        //Link tracking ID
        String traceId = IdUtil.fastSimpleUUID();
        MDC.put(CommonConstant.LOG_TRACE_ID, traceId);
        RequestContext ctx = RequestContext.getCurrentContext();
        ctx.addZuulRequestHeader(CommonConstant.TRACE_ID_HEADER, traceId);
        return null;
    }
}

 

3.3. Add spring interceptor to downstream services

Receive and savetraceIdValue
Interceptor

public class TraceInterceptor implements HandlerInterceptor {
    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
        String traceId = request.getHeader(CommonConstant.TRACE_ID_HEADER);
        if (StrUtil.isNotEmpty(traceId)) {
            MDC.put(CommonConstant.LOG_TRACE_ID, traceId);
        }
        return true;
    }
}

Register interceptor

public class DefaultWebMvcConfig extends WebMvcConfigurationSupport {
  @Override
  protected void addInterceptors(InterceptorRegistry registry) {
    //Log link tracking interceptor
    registry.addInterceptor(new TraceInterceptor()).addPathPatterns("/**");

    super.addInterceptors(registry);
  }
}

 

3.4. Add feign interceptor to downstream services

Continue totraceIdValue passed to downstream service

public class FeignInterceptorConfig {
    @Bean
    public RequestInterceptor requestInterceptor() {
        RequestInterceptor requestInterceptor = template -> {
            //Delivery log traceid
            String traceId = MDC.get(CommonConstant.LOG_TRACE_ID);
            if (StrUtil.isNotEmpty(traceId)) {
                template.header(CommonConstant.TRACE_ID_HEADER, traceId);
            }
        };
        return requestInterceptor;
    }
}

 

3.5. Solve the problem of parent-child thread delivery

It mainly uses thread pool (asynchronous, parallel processing) for business, andspringOneself also have@AsyncAnnotation to use thread pool requires two steps to solve this problem

3.5.1. Rewrite logback’sLogbackMDCAdapter

Due to logback’sMDCTo achieve internal useThreadLocalThe child thread cannot be passed, so it needs to be rewritten and replaced with Ali’sTransmittableThreadLocal

TransmittableThreadLocalAlibaba is open source for solution“Pass ThreadLocal when using components such as thread pool that will cache threads”Inheritablethreadlocal extension of the problem. If you want to transfer the transmittable ThreadLocal thread pool to the main thread, you need to cooperate withTtlRunnableandTtlCallableUse.

Ttlmdcadapter class

package org.slf4j;

import com.alibaba.ttl.TransmittableThreadLocal;
import org.slf4j.spi.MDCAdapter;

public class TtlMDCAdapter implements MDCAdapter {
    /**
     *Here's the key
     */
    private final ThreadLocal<Map<String, String>> copyOnInheritThreadLocal = new TransmittableThreadLocal<>();

    private static TtlMDCAdapter mtcMDCAdapter;

    static {
        mtcMDCAdapter = new TtlMDCAdapter();
        MDC.mdcAdapter = mtcMDCAdapter;
    }

    public static MDCAdapter getInstance() {
        return mtcMDCAdapter;
    }

Other codes andch.qos.logback.classic.util.LogbackMDCAdapterAgain, just callcopyOnInheritThreadLocalvariable

 
Ttlmdcadapterinitializer classUsed to load your ownmdcAdapterRealization

public class TtlMDCAdapterInitializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
    @Override
    public void initialize(ConfigurableApplicationContext applicationContext) {
        //Load the instance of ttlmdcadapter
        TtlMDCAdapter.getInstance();
    }
}

 

3.5.2. Implementation of extended thread pool

increaseTtlRunnableandTtlCallableExtended implementationTTL

public class CustomThreadPoolTaskExecutor extends ThreadPoolTaskExecutor {
    @Override
    public void execute(Runnable runnable) {
        Runnable ttlRunnable = TtlRunnable.get(runnable);
        super.execute(ttlRunnable);
    }

    @Override
    public <T> Future<T> submit(Callable<T> task) {
        Callable ttlCallable = TtlCallable.get(task);
        return super.submit(ttlCallable);
    }

    @Override
    public Future<?> submit(Runnable task) {
        Runnable ttlRunnable = TtlRunnable.get(task);
        return super.submit(ttlRunnable);
    }

    @Override
    public ListenableFuture<?> submitListenable(Runnable task) {
        Runnable ttlRunnable = TtlRunnable.get(task);
        return super.submitListenable(ttlRunnable);
    }

    @Override
    public <T> ListenableFuture<T> submitListenable(Callable<T> task) {
        Callable ttlCallable = TtlCallable.get(task);
        return super.submitListenable(ttlCallable);
    }
}

 

IV. scenario test

4.1. Test code is as follows

Log troubleshooting difficult? Distributed log link tracking to help you
 

4.2. Log printed by API gateway

Gateway generationtraceIdThe value is13d9800c8c7944c78a06ce28c36de670
Log troubleshooting difficult? Distributed log link tracking to help you
 

4.3. Log printed when requesting to jump to file service

DisplayedtraceIdAs with the gateway, the exception scenario is specially simulated here
Log troubleshooting difficult? Distributed log link tracking to help you
 

4.4. Elk aggregation log passedtraceIdQuery the whole link log

When there is an exception in the system, you can directly use thetraceIdQuery all log information of the request in the log center
Log troubleshooting difficult? Distributed log link tracking to help you

 

V. source code download

Attach my open source microservice framework (including the code in this article), welcome to star

https://gitee.com/zlt2000/mic…

Recommended Today

PHP realizes UnionPay business H5 payment

UnionPay business H5 payment interface document: document address 1: H5 payment interface address: 1: Alipay payment Test address: http://58.247.0.18:29015/v1/netpay/trade/h5-pay Official address: https://api-mop.chinaums.com/ 2: UnionPay payment Test address: http://58.247.0.18:29015/v1/netpay/uac/order Official address: https://api-mop.chinaums.com/ 2: Basic parameters required by the interface The interface uses get parameters. After the interface parameters are directly put into the interface address, the […]