K8s and HPA — event driven auto scaling based on kubernetes

Time:2020-1-7

Keda enables fine-grained automatic scaling of event driven kubernetes workloads (including zero to Zero Auto scaling). Keda acts as the kubernetes metrics server, allowing users to define auto scaling rules using a dedicated kubernetes custom resource definition.

Keda can run on the cloud and edge, can be locally integrated with kubernetes components (such as horizo ntal pod autoscaler), and has no external dependency.

Working principle

Keda plays two key roles in kubernetes. First, it acts as an agent to activate and deactivate the deployment to scale from zero to zero without events. Second, it acts as a kubernetes metric server, exposing rich event data (such as queue length or flow lag) to a horizontal pod scaler to drive horizontal scaling. It is then up to the deployment to decide whether to use events directly from the source. This preserves rich event integration and makes gestures such as completing or discarding queue messages immediately available.

K8s and HPA -- event driven auto scaling based on kubernetes

Event sources and scalers

Keda has many “scalers” that can either detect whether a deployment should be activated or deactivated, or provide custom metrics for a specific event source. Today, scaler support is available for:

  • AWS CloudWatch
  • AWS Simple Queue Service
  • Azure Event Hub†
  • Azure Service Bus Queues and Topics
  • Azure Storage Queues
  • GCP PubSub
  • Kafka
  • Liiklus
  • Prometheus
  • RabbitMQ
  • Redis Lists

Of course, other event sources are increasing, as follows:

In planning

  • Azure IoT Hub#214
  • Azure Storage Blobs#154
  • Azure Cosmos DB#232
  • Azure Monitor
  • Azure Durable Functions

Pending planning

  • AWS Kinesis
  • Kubernetes Events
  • MongoDB
  • CockroachDB
  • MQTT

Scaledobject custom resource definition

In order to synchronize the deployment with the event source, a scaledobject custom resource needs to be deployed. Scaledobjects contains information about the deployment to be extended, metadata for the event source (for example, connection string key, queue name), polling interval, and cooling time. Scaledobject will generate the corresponding auto extend resource (HPA definition) to extend the deployment. When scaledobjects is deleted, the corresponding HPA definition is cleared.

For example:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-scaledobject
  namespace: default
  labels:
    deploymentName: azure-functions-deployment
spec:
  scaleTargetRef:
    deploymentName: azure-functions-deployment
  pollingInterval: 30
  triggers:
  - type: kafka
    metadata:
      # Required
      brokerList: localhost:9092
      consumerGroup: my-group       # Make sure that this consumer group name is the same one as the one that is consuming topics
      topic: test-topic
      lagThreshold: "50"

deploy

You can deploy using helm or yaml. With yaml deployment, you can perform the following operations:

kubectl apply -f KedaScaleController.yaml

Kedascalecontroller.yaml is as follows:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: scaledobjects.keda.k8s.io
spec:
  group: keda.k8s.io
  version: v1alpha1
  names:
    kind: ScaledObject
    singular: scaledobject
    plural: scaledobjects
    shortNames:
      - sco
    categories:
      - keda
  scope: Namespaced
  additionalPrinterColumns:
    - name: Deployment
      type: string
      JSONPath: .spec.scaleTargetRef.deploymentName
    - name: Triggers
      type: string
      JSONPath: .spec.triggers[*].type
    - name: Age
      type: date
      JSONPath: .metadata.creationTimestamp
  validation:
    openAPIV3Schema:
      properties:
        spec:
          required: [triggers]
          type: object
          properties:
            scaleType:
              type: string
              enum: [deployment, job]
            pollingInterval:
              type: integer
            cooldownPeriod:
              type: integer
            minReplicaCount:
              type: integer
            maxReplicaCount:
              type: integer
            scaleTargetRef:
              required: [deploymentName]
              type: object
              properties:
                deploymentName:
                  type: string
                containerName:
                  type: string
            triggers:
              type: array
              items:
                type: object
                required: [type, metadata]
                properties:
                  type:
                    type: string
                  authenticationRef:
                    type: object
                    properties:
                      name:
                        type: string
                  metadata:
                    type: object
                    additionalProperties:
                      type: string
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: triggerauthentications.keda.k8s.io
spec:
  group: keda.k8s.io
  version: v1alpha1
  names:
    kind: TriggerAuthentication
    singular: triggerauthentication
    plural: triggerauthentications
    shortNames:
      - ta
      - triggerauth
    categories:
      - keda
  scope: Namespaced
---
apiVersion: v1
kind: Namespace
metadata:
  name: keda
---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: keda-operator
  namespace: keda
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: keda-operator-service-account-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: keda-operator
    namespace: keda
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: keda:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: keda-operator
    namespace: keda
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: keda-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
  - kind: ServiceAccount
    name: keda-operator
    namespace: keda
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: keda-operator
  name: keda-operator
  namespace: keda
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keda-operator
  template:
    metadata:
      labels:
        app: keda-operator
      name: keda-operator
    spec:
      serviceAccountName: keda-operator
      containers:
        - name: keda-operator
          image: kedacore/keda:latest
          args:
            - /adapter
            - --secure-port=6443
            - --logtostderr=true
            - --v=2
          ports:
            - containerPort: 6443
              name: https
            - containerPort: 8080
              name: http
          volumeMounts:
            - mountPath: /tmp
              name: temp-vol
      volumes:
        - name: temp-vol
          emptyDir: {}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics-resource-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-metrics-resource-reader
subjects:
  - kind: ServiceAccount
    name: keda-operator
    namespace: keda
---
apiVersion: v1
kind: Service
metadata:
  name: keda-operator
  namespace: keda
spec:
  ports:
    - name: https
      port: 443
      targetPort: 6443
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app: keda-operator
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  service:
    name: keda-operator
    namespace: keda
  group: external.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-metrics-resource-reader
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
      - pods
      - services
      - external
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: keda-hpa-controller-custom-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: horizontal-pod-autoscaler
    namespace: kube-system

Code architecture interpretation

The key code is in PKG folder, as shown below:
K8s and HPA -- event driven auto scaling based on kubernetes

  • Adapter and provider mainly implement an adapter of custom metrics. Refer to github.com/kubernetes-innovator/custom-metrics-api server for the basic specification. If it is external metrics, it is mainly to implement getexternalmetric and listallexternalmetrics.
  • Both APIs and client are generated by k8s scaffold. APIs mainly stores the definition of CRD — scaledobject object, and the client is Keda client and informer. Those who have extended k8s through CRD should be familiar with this.
  • The controller is a k8s controller for scaledobject. In the development of k8s, after the CRD is created, the corresponding controller must be written, and the actual operation must be made for the add, update and delete events of CRD.
  • Signals is relatively simple, encapsulating context.context.
  • Kubernetes is relatively simple. The general idea is to create kdea client and Kube client for the controller to use according to config.
  • The handler is more critical. Basically, the sync logic in the controller and the interface provided by metrics server are implemented here.
  • Scalers. Is the implementation of different event sources. So if we want to add our own event source, we can implement it here.

For example, when using the client — kubectl or client go to deploy a scaledobject CRD for deployment a. Want to HPA based on kafaka’s message backlog. Then the controller will listen to the creation of CRD, and operate on the new action. Specifically, create an HPA object according to the specific content of CRD, and the spec content of CRD will be converted into HPA. At this time, the HPA of the official k8s will read the number of messages of the topic specified by Kafka through Kafka scaler in the scalers, and then the HPA controller will decide whether to expand or not.

conclusion

Keda is currently in the experimental phase stage. Microsoft and red hat want the community to participate.

Keda didn’t realize its own HPA, but in fact, it still works in the community. It just generates HPA objects based on CRD content, but this metrics is external metrics. Keda mainly integrates various event sources.