Deploying log system elfk on k8s

Time:2020-10-16

Log system elfk

preface

According to the company’s existing business flow and technology stack, we decided to choose the log system solution as follows: elasticsearch (ES) + logstash (LO) + filebeat (FI) + kibana (KI). Es chooses to use the ES provided by aliyun, Lo & fi chooses to deploy itself, and Ki is sent by alicloud. Because it takes a certain time to apply for ECS, we choose to deploy it in the test production environment for a while. We make complaints about testing and producing a common set of k8s and hosting aliyun. It takes one day to complete the deployment of elfk on kubernetes (deploy it first, and then optimize what needs to be done later).

Introduction to components

ES is a real-time, distributed and extensible search engine, which allows full-text and structured search. It is usually used to index and search a large number of log data, and can also be used to search many different types of articles.

The main advantage of Lo is its flexibility, mainly because it has many plug-ins, detailed documentation and straightforward configuration format, so that it can be applied in a variety of scenarios. We can basically find a lot of resources on the Internet and can deal with almost any problem.

As a member of the beats family, FI is a lightweight log transport tool. Its existence makes up for the shortcomings of lo. As a lightweight log transport tool, FI can push logs to the central lo.

Ki is an analysis and visualization platform, which can browse and visualize the top log data stored in ES cluster, and build dashboards. Ki is a professional log display application, which combines es operation and integrates most es API.

Data acquisition flow chart

img

Log flow: logs_ data—> fi —> lo —> es—> ki。

logs_ The data is collected by FI and output to lo. After filtering and modifying by lo, the data is transferred to es database. Ki reads es database for analysis.

deploy

According to the actual cluster situation of our company, this document deployment will completely restore the deployment of the log system.

Install kubectl on the local Mac and connect to aliyun hosting k8s

Install the same version of kubectl as the hosted k8s on the client (any local virtual machine)

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.14.8/bin/linux/amd64/kubectl   
chmod +x ./kubectl 
mv ./kubectl /usr/local/bin/kubectl  
Copy the kubeconfig of the k8s hosted by alicloud to the $home /. Kube / config directory, and pay attention to user permissions
Deploy elfk

Apply for a namespace (usually one for each project).

# cat kube-logging.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: loging

Deploy es. Find a similar list of resources online, according to their own needs for appropriate modification, operation, error according to the log to modify.

# cat elasticsearch.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-class
  namespace: loging
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
# Supported policies: Delete, Retain
reclaimPolicy: Delete
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: datadir1
  namespace: logging
  labels:
    type: local
spec:
  storageClassName: local-class
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/data/data1"
--- 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: loging
spec:
  serviceName: elasticsearch
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: elasticsearch:7.3.1
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: "discovery.type"
            value: "single-node"
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-class"
      resources:
        requests:
          storage: 5Gi
---
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: loging
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

Deploy ki. According to the data acquisition flow chart, Ki is combined with ES, and the configuration is relatively simple.

# cat kibana.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kibana
  namespace: loging
  labels:
    k8s-app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana
  template:
    metadata:
      labels:
        k8s-app: kibana
    spec:
      containers:
      - name: kibana
        image: kibana:7.3.1
        resources:
          limits:
            cpu: 1
            memory: 500Mi
          requests:
            cpu: 0.5
            memory: 200Mi
        env:
          - name: ELASTICSEARCH_HOSTS
#Note that value is the services of ES, because es is a stateful and headless service, so it is not only the name of the pod that is connected
            value: http://elasticsearch:9200   
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: loging
spec:
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana

Configure the ingress controller. Because our company uses nginx ingress of k8s hosted by alicloud, and it is configured with forced conversion of HTTPS. So kibana ingress should also be configured as HTTPS.

# openssl genrsa -out tls.key 2048
# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=kibana.test.realibox.com
# kubectl create secret tls kibana-ingress-secret --cert=tls.crt --key=tls.key

Kibana ingress is configured as follows. Two kinds are provided, one is HTTPS and the other is http.

https:
# cat kibana-ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana
  namespace: loging
spec:
  tls:
  - hosts:
    - kibana.test.realibox.com
    secretName: kibana-ingress-secret
  rules:
  - host: kibana.test.realibox.com
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana
          servicePort: 5601

http:
# cat kibana-ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana
  namespace: loging
spec:
  rules:
  - host: kibana.test.realibox.com
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana
          servicePort: 5601

Deploy lo. Because the role of Lo is to filter the logs collected by FI, it needs to do different processing according to different logs, so it may need to be changed frequently and decoupled. Therefore, mount in the form of configmap.

# cat logstash.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: logstash
  namespace: loging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
      - name: logstash
        image: elastic/logstash:7.3.1
        volumeMounts:
        - name: config
          mountPath: /opt/logstash/config/containers.conf
          subPath: containers.conf
        command:
        - "/bin/sh"
        - "-c"
        - "/opt/logstash/bin/logstash -f /opt/logstash/config/containers.conf"
      volumes:
      - name: config
        configMap:
          name: logstash-k8s-config
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: logstash
  name: logstash
  namespace: loging
spec:
  ports:
    - port: 8080       
      targetPort: 8080
  selector:
    app: logstash
  type: ClusterIP

# cat logstash-config.yaml 
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: logstash
  name: logstash
  namespace: loging
spec:
  ports:
    - port: 8080       
      targetPort: 8080
  selector:
    app: logstash
  type: ClusterIP
---

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-k8s-config
  namespace: loging
data:
  containers.conf: |
    input {
      beats {
        Port = > 8080 filebeat connection port
      }
    }
    output {
      elasticsearch {
        hosts => ["elas ticsearch:9200 "] service of
        index => "logstash-%{+YYYY.MM.dd}"
      }
    }
Note: modifying configmap is equivalent to modifying the image. The resource list must be reapplied to take effect. According to the data acquisition flow chart, the data of Lo flows from fi to es.

Deploy fi. The main function of FI is to collect the log, and then give the data to lo.

# cat filebeat.yaml 
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: loging
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false
    # To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      hints.enabled: true
    output.logstash:
      hosts: ['${LOGSTASH_ HOST:logstash }:${LOGSTASH_ PORT:8080 }'], flow to lo
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: loging
  labels:
    app: filebeat
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: loging
  labels:
    app: filebeat
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: elastic/filebeat:7.3.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        Env: injection variable
        - name: LOGSTASH_HOST
          value: logstash
        - name: LOGSTASH_PORT
          value: "8080"
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: inputs
          mountPath: /usr/share/filebeat/inputs.d
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: inputs
        configMap:
          defaultMode: 0600
          name: filebeat-inputs
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: loging
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: loging
  labels:
    app: filebeat
---

So far, ES + Lo + fi + ki is deployed on k8s for simple verification.

verification

View SVC, pod, ingress information

# kubectl get svc,pods,ingress -n loging
NAME                    TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)             AGE
service/elasticsearch   ClusterIP   None                      9200/TCP,9300/TCP   151m
service/kibana          ClusterIP   xxx.168.239.2xx           5601/TCP            20h
service/logstash        ClusterIP   xxx.168.38.1xx           8080/TCP            122m

NAME                            READY   STATUS    RESTARTS   AGE
pod/elasticsearch-0             1/1     Running   0          151m
pod/filebeat-24zl7              1/1     Running   0          118m
pod/filebeat-4w7b6              1/1     Running   0          118m
pod/filebeat-m5kv4              1/1     Running   0          118m
pod/filebeat-t6x4t              1/1     Running   0          118m
pod/kibana-689f4bd647-7jrqd     1/1     Running   0          20h
pod/logstash-76bc9b5f95-qtngp   1/1     Running   0          122m

NAME                        HOSTS                       ADDRESS        PORTS     AGE
ingress.extensions/kibana   kibana.test.realibox.com   xxx.xx.xx.xxx   80, 443   19h
Web configuration

Configuration index


find

At this point, it is a simple completion. The follow-up needs to be optimized continuously, but that’s the future.

Problem summary

This should be regarded as the first time to deploy the application in the test & production environment, and it is a day when I am not familiar with the system. I have encountered many problems and need to be summarized.

  1. How to investigate a technology stack;
  2. How to select the scheme;
  3. Because there are almost no similar solutions on the Internet (I don’t know how other companies do it, anyway, there is no effective online reference). We need to summarize and try according to different documents;
  4. The label of a component should be consistent as far as possible;
  5. How to check whether the company has made port restrictions and HTTPS casts;
  6. When you encounter it, you must look at the log. This is very important. Log can solve most problems;
  7. A person how to integrate will also ignore some points, their first try and then consult friends, common progress.
  8. The project will be launched first and then others. At present, it is like this. You can do something with 20% confidence. 80 percent of it doesn’t make sense.
  9. The focus of self-study is theory, and the company can learn operation.

Recommended Today

nuxt.js Imitating wechat app communication chat | Vue + nuxt chat | imitating wechat interface

Project overview be based on vue.js + nuxt.js +Chat room imitating wechat app interface developed by vuex + webpack + node + vant and other technologiesNuxtchatroom project。 It realizes the functions of card type drag and slide, message / expression sending, picture / video preview, red packet / circle of friends and so on. Technology […]