Logging operator – elegant cloud native log management scheme (III)

Time:2022-1-19

Previously:
The article on logging operator was delayed for a long time last year. I thought there would be no further progress, but recently in my own blogKubeGemsWhen the project encountered the need to deal with the observability part of the log, it revisited it, so there was the third article in this series.

Logging OperatorIt is an open source log collection scheme under Banzai cloud under the cloud native scenario. In March 2020, with the reconstructed V3 version, the underlying logging operator has almost perfectly adapted to the log collection scenario under kubernetes mode by virtue of efficient fluent bit and plug-in rich fluent. It can be expected in the future. Last year, I accidentally found that rancher also adopted logging operator as a unified logging solution after version 2.5, which is enough to show that it is being accepted by some management platforms with kubernetes as the core and integrated internally (including Xiaobai’s kubegems).

As a sequel to the previous two articles, this article mainly talks about Xiaobai’s recent cases and feelings when using logging operator to solve user needs, so I don’t intend to spend more time describing its architecture and use. Interested students can turn to Xiaobai’s articles.

About indicators

In the process of application containerization, due to the temporary nature of the container file system, developers are always faced with the dilemma of dropping their own log files and outputting stdout. When R & D hands over the right of application log management to the platform, it means that what the platform needs to do is far more complex than one-to-one collection of applications. Among the many requirements, one day, SRE asked: “we can see the real-time rate of log collection in Alibaba cloud. We need to customize quality monitoring indicators for this.”. This question also reminds me that when we are doing private cloud, standing outside the platform and observing the inside of the log collection pipeline has always been in the blind area of lack of information. Fortunately, both fluent bit and fluent d have independent Prometheus plug-ins to expose internal indicators. However, after using the logging operator, its indicator collection relies on the Prometheus operator, and the architecture is clear enough. Therefore, we can also have more options to meet the needs of R & D.

Logging operator - elegant cloud native log management scheme (III)

First, when defining logging, we can enable fluent bit (d) to start the collection of Prometheus

spec:
  fluentdSpec:
    metrics:
      serviceMonitor: true
      serviceMonitorConfig:
        Honorlabels: true // opening honorlabels is mainly to keep the original label of the component and prevent the label from being overwritten.
  fluentbitSpec:
    metrics:
      serviceMonitor: true

Here you can see that the logging operator mainly relies onServiceMonitorFor service discovery at the collection end, the Prometheus operator needs to be run inside the cluster to support the CRD. If the resource type is not changed within the cluster, you can also use Prometheus’s own service discovery mechanism to complete the discovery and collection of indicators.

However, only the indicator entry of the collection end is declared here. By default, it only includes the basic internal operation status of fluent bit (d). If you want to further monitor the log rate, you need to launch flunetd. In the early years, when Google’s gke still used fluent as a log collector, it occasionally looked at it casually(purposeful plagiarism)A Prometheus plug-in configuration aroused my interest

<filter **>
  @type prometheus
  <metric>
    type counter
    name logging_entry_count
    desc Total number of log entries generated by either application containers or system components
  </metric>
  <labels>
    container: $.kubernetes.container_name
    pod: $.kubernetes.pod_name
  </labels>
</filter>

This rule will match all logs entering fluent and enter Prometheus filter for counting. And the statistical indicators arelogging_entry_countThe label is named according to some metadata information in the log as an indicator, which is used to distinguish between different containers.

Since the kubernetes metadata of the log needs to be parsed, fluent is needed herekubernetes-metadata-filterPlug in to extract container metadata. In the logging operator, the metadata of kubernetes is parsed in fluent bit. There is no need to add the plug-in in fluent D

Although Google gke now also changes the log collector to fluent bit, the above configuration is not “outdated” in the logging operator. With a lesson from the past, we can introduce the Prometheus plug-in into the tenant’s log collector (flow / clusterflow) to analyze the log rate. The simplest practice is as follows:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: default
  namespace: demo
spec:
  - prometheus:
      labels:
        container: $.kubernetes.container_name
        namespace: $.kubernetes.namespace_name
        node: $.kubernetes.host
        pod: $.kubernetes.pod_name
      metrics:
      - desc: Total number of log entries generated by either application containers
          or system components
        name: logging_entry_count
        type: counter
  globalOutputRefs:
  - containers-console
  match:
  - select:
      labels:
        what.you.want: collect

After the above indicators are stored in Prometheus, we can find out the application rate of the log collector under the current cluster through this statement

sum by (pod) (rate(logging_entry_count[1m]))

At this time, if the cloud platform is based on multi tenant and multi environment architecture, you can even aggregate the log rate according to the tenant environment and tenant level.

Logging operator - elegant cloud native log management scheme (III)

The above is only the collection and monitoring of the overall log rate. If we need to count the specific contents in the log or the bytes of the log, we need to combine them with other plug-ins. At present, the plugins supported by the logging operator are far less abundant than those supported by fluent D, but we can refer to the official documents to write the required plugins for integration into the operator.Logging operator developers manual

The logging operator has its own set of rules for internal monitoring and alarm of the logging component, which can be enabled in the CR

spec:
  fluentbitSpec:
    metrics:
      prometheusRules: true
  fluentdSpec:
    metrics:
      prometheusRules: true

thereprometheusRulesIt is also a resource managed by Prometheus operator. If there is no such resource type in the cluster, you can manually configure rules for Prometheus

Back to the original question,If it is necessary to take the log collection rate as the quantitative index of the application, uselogging_entry_countJust.

About sampling

In most cases, the log architecture should not adopt some uncontrollable strategies for business logs, resulting in incomplete application logs, such as sampling. Obviously, I don’t recommend you to enable this function in the existing architecture. However, sometimes, or when some magicians can’t effectively control the program’s “boundless power” and crazy output, the platform can sample such playful applications. After allEnsure the availability of the entire log channelIs the first priority of the platform.

In terms of log sampling, the logging operator adoptsThrottleThe plug-in speed limiter is summarized in one sentence. The plug-in introduces the leaky bucket algorithm for each pipeline entering the filter log to discard the logs exceeding the rate limit.

apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: default
  namespace: demo
spec:
  - throttle:
      group_bucket_limit: 3000
      group_bucket_period_s: 10
      group_key: kubernetes.pod_name
  globalOutputRefs:
  - containers-console
  match:
  - select:
      labels:
        what.you.want: collect
  • group_ Key: the aggregation key of log sampling. Usually, we aggregate according to the pod name, or directly fill in kubenretes Other value aggregation of metadata is also OK
  • group_ bucket_ period_ s: Sampling time range, 60s by default
  • group_ bucket_ Limit: maximum capacity of log bucket during sampling

The sampling rate of the log is determined by the formulagroup_bucket_limit / group_bucket_period_sCalculation, whengroup_keyIf the log rate in exceeds the value, it will be lost to subsequent logs.

Because throttle does not adopt the token bucket algorithm, it will not have burst to deal with the collection of burst logs.

About log disk dropping

As mentioned earlier, for all container based applications, the best practice for logging is to direct logs tostdoutstderrHowever, not all “magicians” will follow this agreement. Log file dropping is still the choice of most R & D at present. Although in theory, the standard (error) output of the container also resets the log stream set to/var/log/containers But it is still limited by uncontrollable factors caused by runtime configuration or other hard disk reasons.

For the scenario of log disk dropping, there is no unified solution in the industry, but there are actually two implementation methods:

  • Sidecar scheme

    This scheme is to run the log collector in the pod together with the application container, and collect the logs by volume sharing the log path. The common way is to develop a separate controller for kubernetes and adoptMutatingWebhookIn the pod startup phase, the sidecar information is injected.

    The sidecar scheme has the advantage that the sidecar configuration for each application is relatively independent, but the disadvantage is that in addition to occupying too many resources,The update iteration of the collector needs to follow the life cycle of the application, not very elegant for continuous maintenance.

  • Node agent scheme

    This scenario uses the log collector toDaemonSetThe method runs in each node, and then centralized collection is carried out at the operating system level. Usually, this scheme requires the development of the platform, and a certain path strategy needs to be made to fix ithostpathThe Vulume of is mounted to the container for use when the application log is dropped. In addition, we know that all kubernetes default storage types or third-party CSI compliant storage will mount volumes to/var/lib/kubelet/pods/<pod_id>/volumes/kubernetes.io~<type>/<pvc_id>/mountDirectory, so a more elegant solution for node agent is to give node agent the right to call kubernetes API and let it know the path mapped to the host of the collected container log.

    The advantage of the node agent scheme is that the configuration is centrally managed, and the collector is decoupled from the application container without affecting each other. The disadvantage is that the collector has the risk of insufficient throughput.

You can see that neither of the above two schemes has anything to do with the logging operator. It is true that the community does not have an effective scheme for this scenario, but according to its ideas, we canLog file to standard (error) output streamTo deal with this problem in disguise.

use tailLet’s give an intuitive example to illustrate the above scheme.

...
containers:
- args:
  - -F
  - /path/to/your/log/file.log
  command:
  - tail
  image: busybox
  name: stream-log-file-[name]
  volumeMounts:
  - mountPath: /path/to/your/log
    name: mounted-log
...

althoughtailIt is an extremely simple and crude method, and can not solve the problems such as log rotation, but it does provide a new scheme for logging operator in the scenario of log disk dropping. Although it looks the same as sidecar, the biggest difference is that this scheme can be seamlessly compatible with the existing log pipeline of the logging operator, and the logs can still be processed in the flow stage after being collected.

summary

From the perspective of automatic operation and maintenance, the logging operator has effectively solved the problems of complex log architecture and difficult application log collection in the kubernetes scenario, although the current support for disk dropping logs is not comprehensive enough. However, with the gradual growth of access users, there may be better solutions to the current problems in the future. However, at present, it is indeed one of the best cloud native log architectures.


Pay attention to the official account, Yun Sheng Sheng Xiao Bai, reply to “entering group” and join Loki learning group