Kubernetes two step verification: dynamic access control using serverless


Author: coding – Wang Wei

1. Background

If we pay special attention to kubernetes cluster security, we may want to implement these requirements:

  • How to realize the two-step authentication of kubernetes cluster, in addition to the cluster credentials, also need to provide one-time token verification?
  • How to verify whether the deployed image is safe and compliant, so that only the docker image of the company’s internal image warehouse can be deployed?
  • How to dynamically inject sidecar into each deployment to meet specific security or business requirements?
  • How to implement cluster level imagepullsecrets and automatically inject imagepullsecrets into the new namespace when creating a new namespace?

This paper takes the implementation of kubernetes two-step verification as an exampleKubernetes AdmissionDynamic access control, with the help ofServerlessImplement aTwo step validationSo that readers canDynamic access controlandServerlessHave a deeper understanding.

1.2 implementation effect

Token two-step verification failed, deployment is not allowed

Token two-step verification is successful and deployment is allowed

2. What is advertisement

Permission is the step after the user executes kubectl authentication and before persisting resources to etcd. Kubernetes realizes the supplement of user-defined business logic by calling webhook in order to decouple this part of logic. The above processes are all within the life cycle of the user executing kuberctl and waiting for the API server to synchronously return the results.

① and ② marked in the above figure are the workflow of the intervention of the advertisement. We will find these features:

  1. Permission works after the cluster authentication is passed
  2. There are two kinds of advertisements:MutatingandValidating
  3. These two specific implementation methods are based onWebhookRealized
  4. The operation object of the permission can be the currently deployed user, yaml content, etc

2.2 Admission Mutating

MutatingBefore the resources are persisted to etcd, the real meaning is that,MutatingThe controller can modify the deployed resource files, such as dynamically adding labels to a specific pod and dynamically injectingsidecarEtc.
Careful readers will find that,Admission MutatingIt is used in many products, such asIstioIt is used to dynamically inject sidecar into each containerEnvoyContainer to achieve traffic hijacking and management.

2.3 Admission Validating

ValidatingIt is easy to understand, that is, “verification”, which is in theMutatingAfter that, we can implement the custom verification logic in this phase. In this paper, we use it to implement a simple two-step verification mechanism.

3. What is the advertisement webhook

Admission WebhookIn fact, it isMutating ControllersandValidating ControllersIn other words, we need to provide an external webhook endpoint for kubernetes cluster. When the API server executes the corresponding process, it will call our predefined webhook to implement our predefined business logic, and change or verify the yaml file by returning the specified data structure.

4. Hands on practice

4.1 cluster conditions

According to official documents, the preconditions are as follows:

  • Kubernetes cluster version is at least v1.16
  • The mutatingadmissionwebhook and validatingadmissionwebhook controllers are enabled

If you are not sure, you can query it with the following command:

kubectl get pods kube-apiserver -n kube-system -o yaml | grep MutatingAdmissionWebhook,ValidatingAdmissionWebhook

If you are using a managed cluster, use the following command to query:

kubectl api-versions | grep admission

If it appearsadmissionregistration.k8s.io/v1beta1Explain the cluster support and proceed to the next step.

4.2 other conditions

  • Open coding Devops
  • Clone code repository admission webhook- example.git And push it to your own coding git warehouse
  • Prepare a Tencent cloud account

4.3 deploy Tencent serverless service

  1. Log in to coding, configure the serverless identity authorization, record the credential ID (similar to: b68948cb-2ad9-4b67-8a49-ad7ba910ed92), and use it later

  2. Clone code repositoryadmission-webhook-example

git clone https://e.coding.net/wangweicoding/admission-webhook-example.git
  1. Modify the files in the root directory
  • The Jenkins file in the root directory replaces the credential ID at the cursor with the credential ID obtained in the previous step
  • Modify theVPC_IDandSUBNET_ID, these two items can be found in the “private network” of Tencent cloud console; if there is no private network and subnet, you can create a new one by yourself, and pay attention to selecting “Guangzhou” as the region

  • When you’re done, push the code to your ownCODING GitCode warehouse
  1. Use “blank template” to create a build plan and select “use Jenkins file of code repository”

  2. Run the build plan and deploy the serverless service

    After running, click the “output endpoint” stage to view the output URL (similar to: https://service-faeax9cy-1301578102.gz.apigw.tencentcs.com/release/index ), which is the URL of the server less service.Record for use in the next phase

So far, Tencent cloud serverless service has been deployed.

4.4 kubernetes cluster deployment validation webhook

Because the approval webhook only allows the HTTPS protocol and needs to provide the certificate information, we need to generate it in advance. The code warehouse has provided the script. After running, the cluster certificate can be configured.

$ ./deployment/webhook-create-signed-cert.sh

creating certs in tmpdir /var/folders/mt/965plkfs62v6wqx2839qthz40000gq/T/tmp.i1imELSt
Generating RSA private key, 2048 bit long modulus (2 primes)
e is 65537 (0x010001)
certificatesigningrequest.certificates.k8s.io/admission-webhook-example-svc.default created
NAME                                    AGE   REQUESTOR   CONDITION
admission-webhook-example-svc.default   1s    admin       Pending
certificatesigningrequest.certificates.k8s.io/admission-webhook-example-svc.default approved
secret/admission-webhook-example-certs configured

modifydeployment/deployment.yamlFile, willserverlessURLReplace with the one recorded in the previous phaseEndpoint(similar to: https://service-faeax9cy-1301578102.gz.apigw.tencentcs.com/release/index )

After the certificate is successfully created, deploy deployment and services

$ kubectl create -f deployment/deployment.yaml
deployment.apps "admission-webhook-example-deployment" created

$ kubectl create -f deployment/service.yaml
service "admission-webhook-example-svc" created

At this point, the service we use to receive validation requests has been deployed and finally configuredValidatingWebhookConfiguration, run the following command:

cat ./deployment/validatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > ./deployment/validatingwebhook-ca-bundle.yaml

After execution, you can seevalidatingwebhook-ca-bundle.yamlOfcaBundleField has been replaced.

Script running depends on JQ (shell read JSON tool). If you have not installed it, please move to: https://www.ibm.com/developerworks/cn/linux/1612_ chengg_ Jq/ index.html

MAC system can be installed directly by using: brew install JQ.

Next, wedefaultThe namespace is labeled because of ourValidatingWebhookConfigurationUsednamespaceSelectorOnly a two-step validation is performed for the namespace containing specific labels.

$ kubectl label namespace default admission-webhook-example=enabled
namespace "default" labeled

Finally, createValidatingWebhookConfiguration

$ kubectl create -f deployment/validatingwebhook-ca-bundle.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io "validation-webhook-example-cfg" created

In this way, once thedefaultThe resource is created in the namespace. The deployed service will intercept the request and conduct secondary verification.

4.5 try two-step verification

So far, we have successfully deployed the demo with two steps of verification, and the overall architecture diagram now becomes:

Now, we can try to deploy

$ kubectl apply -f deployment/sleep.yaml
Error from server: error when creating "deployment"/ sleep.yaml ": admission webhook "required- labels.coding.net "Denied the request: token error, deployment not allowed

When creating the serverless service, we pre configured four groups of tokens to the database, namely 1111, 2222, 3333 and 4444, so we can modify themsleep.yaml, will be annotatedmetadata.annotations.tokenRevised as1111, try to deploy again

$ kubectl apply -f deployment/sleep.yaml
deployment.apps/sleep created

The deployment is successful. If the token is reused, it cannot be verified. So far, the two-step verification based on serverless has been completed.

5. Source code analysis

5.1 what does our deployment do

After executing kubectl apply, the API server will forward the request to our deployed pod. The core code is in the root directory of the project, mainly includingmain.goandwebhook.go

main.go It mainly started an HTTP service and read the certificate and serverless endpoint we created from the command line

// main.go

flag.IntVar(&parameters.port, "port", 443, "Webhook server port.")
flag.StringVar(&parameters.certFile, "tlsCertFile", "/etc/webhook/certs/cert.pem", "File containing the x509 Certificate for HTTPS.")
flag.StringVar(&parameters.keyFile, "tlsKeyFile", "/etc/webhook/certs/key.pem", "File containing the x509 private key to --tlsCertFile.")
flag.StringVar(&parameters.serverlessURL, "serverlessURL", "https://example.com", "serverless endpoint URL.")

webhook.go It mainly forwards requests sent by API server. We willvalidateRewrite and forward all requests to the serverless endpoint.

// webhook.go

glog.Infof("parameters.serverlessURL is %v", whsvr.parameters.serverlessURL)
res, _ := Post(whsvr.parameters.serverlessURL, req)
//Initialize request variable structure
jsonData := make(map[string]string)
//Call the parsing of JSON package to parse the request body
_ = json.NewDecoder(res.Body).Decode(&jsonData)
glog.Infof("res is %v", jsonData)
allowed := false
reason := &metav1.Status{
    Reason: "token error, deployment not allowed,",
if jsonData["allow"] == "true" {
    allowed = true
return &v1beta1.AdmissionResponse{
    Allowed: allowed,
    Result:  reason,

After the pod forwards the request to our serverless function, it makes the business logic to determine whether admission is allowed. The pod then reformats the results of serverless and returns it to the API server.

5.2 what does serverless do?

The serverless service we deployed mainly includes four parts:

  • API Gateway
  • Cloud function
  • Postgresql
  • VPC

We used coding Devops to deploy the above serverless services in Tencent cloud. The core code of jenkinsfile is as follows:

Stage ('deploy serverless service '){
  steps {
    withCredentials([string(credentialsId:"b68948cb-2ad9-4b67-8a49-ad7ba910ed92", variable:'tencent_serverless')]) {
      sh 'echo "${tencent_serverless}" > .tmp'
      sh '''
        SecretId=$(cat .tmp | jq -r .SecretId)
        SecretKey=$(cat .tmp | jq -r .SecretKey)
        token=$(cat .tmp | jq -r .token)
        AppId=$(cat .tmp | jq -r .AppId)
        echo "TENCENT_SECRET_ID=${SecretId}" >> ./serverless/.env
        echo "TENCENT_SECRET_KEY=${SecretKey}" >> ./serverless/.env
        echo "TENCENT_APP_ID=${AppId}" >> ./serverless/.env
        echo "TENCENT_TOKEN=${token}" >> ./serverless/.env
      sh 'cd serverless && cat .env'
      sh 'cd serverless && npm run bootstrap && sls deploy --all | tee log.log'
      sh 'rm ./serverless/.env'
    Echo 'deployment complete'
Stage ('output endpoint '){
  steps {
    sh 'cd serverless && cat log.log | grep apigw.tencentcs.com'

Here we mainly use temporary credentials, as well as the server less SDK serverless.yml Deployment.

API gateway is responsible for providing external network access

# ./serverless/api/ serverless.yml  API gateway deployment file

  - apigw:
      name: k8sAdmission
          - http
          - https
        description: Based on Tencent Cloud Serverless, it provides dynamic access control for K8S
        environment: release
          - path: /index
            method: POST

PostgreSQL is responsible for storing predefined tokens

# ./serverless/db/ serverless.yml   Database deployment file

org: k8sAdmission
app: k8sAdmission-db
stage: dev

component: postgresql
name: fullstackDB

  region: ${env:REGION}
  zone: ${env:ZONE}
  dBInstanceName: ${name}
    vpcId: ${output:${stage}:${app}:serverlessVpc.vpcId}
    subnetId: ${output:${stage}:${app}:serverlessVpc.subnetId}
  extranetAccess: false

VPC realizes interworking between cloud function and PostgreSQL network

# ./serverless/vpc/ serverless.yml  VPC deployment file

org: k8sAdmission
app: k8sAdmission-db
stage: dev

component: vpc # (required) name of the component. In that case, it's vpc.
name: serverlessVpc # (required) name of your vpc component instance.

  region: ${env:REGION}
  zone: ${env:ZONE}
  vpcName: serverless
  subnetName: serverless

Cloud function is responsible for access logic judgmenthandler: api_service.main_handlerIn other words, the entry function of cloud function ismain_handlerWhen an external request comes, it will be executedmain_handlerfunction

# ./serverless/api/ serverless.yml  Cloud function deployment file

org: k8sAdmission
Component: SCF ා (required) refers to the name of component. Currently, Tencent SCF component is used
Name: k8s ා (required) the instance name created by the component
App: k8sadmission DB ා (optional) the SCF application name
Stage: dev ා (optional) is used to distinguish environment information. The default value is dev

  src: ./
  name: ${name}
  Description: k8s dynamic access control based on Tencent cloud serverless
  handler: api_ service.main_ Handler ා entry function
  Runtime: Python 3.6 ා runtime environment of cloud functions. In addition to nodejs10.15, the optional values are: Python 2.7, python 3.6, nodejs6.10, nodejs8.9, PHP5, php7, golang1, java8.
  region: ${env:REGION}
    vpcId: ${output:${stage}:${app}:serverlessVpc.vpcId}
    subnetId: ${output:${stage}:${app}:serverlessVpc.subnetId}
  timeout: 10
      PG_CONNECT_STRING: ${output:${stage}:${app}:fullstackDB.private.connectionString}
      PG_DN_NAME: ${output:${stage}:${app}:fullstackDB.private.dbname}
    - apigw:
        name: k8sAdmission
            - http
            - https
          description: Based on Tencent Cloud Serverless, it provides dynamic access control for K8S
          environment: release
            - path: /index
              method: POST

Cloud function key code

We will create the tokens table on the first trigger (request) and insert four sets of predefined tokens into the table. And check that we are executing the kubectl apply yaml fileAnnotationsWhether the tokens carried in is legal, and compare the token with the token stored in the PostgreSQL database.

# ./serverless/api/api_ service.py  Cloud function business logic

def main_handler(event,content):
    logger.info('start main_handler')
    logger.info('got event{}'.format(event))
    logger.info('got content{}'.format(content))
    #Connect to database
    print('Start Serverlsess DB SDK function')

    conn = psycopg2.connect(DB_HOST)
    print("Opened database successfully")

    cur = conn.cursor()
           (ID INT PRIMARY KEY     NOT NULL,
           tokens TEXT    NOT NULL);''')

    cur.execute("select * from TOKENS")
    myresult = cur.fetchall()
    for row in myresult:
       print("ID = " + str(row[0]))
       print("tokens = " + row[1])

    if not bool(cur.rowcount):
        print("insert default tokens")
        cur.execute("INSERT INTO TOKENS (ID,tokens) \
            VALUES (1, '1111')")
        cur.execute("INSERT INTO TOKENS (ID,tokens) \
            VALUES (2, '2222')")
        cur.execute("INSERT INTO TOKENS (ID,tokens) \
            VALUES (3, '3333')")
        cur.execute("INSERT INTO TOKENS (ID,tokens) \
            VALUES (4, '4444')")

    json_dict = json.loads(event["body"])
    if json_dict["object"]["metadata"]["annotations"]["token"] == "":
        return {"errorCode":0,"errorMsg":"","allow":"false"}
    cur.execute("SELECT * FROM TOKENS where tokens=%s",[json_dict["object"]["metadata"]["annotations"]["token"]])
    myresult = cur.fetchall()
    allow = "false"
    if len(myresult) > 0:
        allow = "true"
        query_id = myresult[0][0]
        cur.execute("DELETE FROM TOKENS where ID=%s",[query_id])
    return {"errorCode":0,"errorMsg":json_dict["object"]["metadata"]["annotations"]["token"],"allow":allow}

If the token exists in the database, delete the token used this time from the database and return JSON toWe deploy the pod within the cluster


Pod reassembles the information according to the results returned by serverless, and returns the following JSON to Kubernetes API Server

        "Reason": "token error, deployment not allowed,",

Among them,AllowedThe field is this timekubectl applyWhether access is critical or not,ReasonThe information will be presented as a result.

Some students may ask, why do we need to call the serverless service through our deployed pod? Can’t API server request serverless endpoint directly? The answer is no, because the webhook URL requested by API server requires two-way TLS authentication. We need to create TLS certificate signed by kubernetes CA to ensure the security of communication between webhook and API server. Therefore, we adopt this method to achieve this.

6. Conclusion

So far, we have implemented a simple kubernetes two-step verification. If you want to implement more logic, such as judging image compliance and refusing to deploy images from non company internal warehouses, you can implement them in the serverless cloud function.

In production practice, for example, token in this example belongs to the dynamic yaml product type deployment. We can combine it withCoding continuous deploymentTo provide dynamic parameter binding for artifact files.

If you want to realize the dynamic injection of sidecar into the deployment, you can use the mutating webhook to monitor the deployed deployment and inject the sidecar dynamic patch that needs to be injected.

If you want to implement cluster level imagepullsecrets, a feasible idea is to use mutating webhook to listen and create namespaces, and automatically patch existing imagepullsecrets into new namespaces.

To implement mutating webhook, please pay attention to the webhook.go Document’smutateFunction, the principle is similar to validating webhook, the difference is that it is mainly implemented through patch.

Kubernetes admissionThe process of kubectl is decoupled by the way of webhook, so that our own business logic can be dynamically added to the process of user executing kubectl to returning the results. The two-step verification in this paper is just a simple demo. For more in-depth understanding, you can browse the “resources” link.

7. References

  • In-depth introduction to Kubernetes admission webhooks
  • Extensible permission enters beta phase
  • Dynamic access control
  • Tencent Serverless
  • This project fork source code

Recommended Today

Vue monitors whether the DOM element scrolls up or down in demo to achieve ceiling effect.

Go directly to the page dmeo <template> <div> <van-pull-refresh v-model=”isLoading” @refresh=”onRefresh” > <ul ref=”msgList”> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> <li>11111111</li> </ul> </van-pull-refresh> </div> </template> <script> //I use the vant component globally here, so I can use the drop-down effect directly //Van pull refresh export […]