Kubernetes certificate with unclear distinction

Kubedm generated a bunch of certificates, which are not so magical, to dig deep into its underwear.

[email protected]:/etc/kubernetes/pki# tree
.
|-- apiserver.crt
|-- apiserver-etcd-client.crt
|-- apiserver-etcd-client.key
|-- apiserver.key
|-- apiserver-kubelet-client.crt
|-- apiserver-kubelet-client.key
|-- ca.crt
|-- ca.key
|-- etcd
|   |-- ca.crt
|   |-- ca.key
|   |-- healthcheck-client.crt
|   |-- healthcheck-client.key
|   |-- peer.crt
|   |-- peer.key
|   |-- server.crt
|   `-- server.key
|-- front-proxy-ca.crt
|-- front-proxy-ca.key
|-- front-proxy-client.crt
|-- front-proxy-client.key
|-- sa.key
`-- sa.pub

1 directory, 22 files

Starting from RSA

To deeply understand the role of certificate, we need to understand some principles and have some basic knowledge, such as what is asymmetric encryption, what is public key, private key, digital signature and so on. Start with RSA algorithm.

Asymmetric encryption generates a key pair, as shown above sa.key sa.pub It’s a key pair, one for encryption and one for decryption.

Plaintext + public key = > ciphertext

Ciphertext + private key = > plaintext

If there is no private key, it is difficult to decrypt the ciphertext.

If you don’t want to pay attention, you can skip the following principle part:

Suppose we want to encrypt a word Caesar, first turn it into a string of numbers, such as ASCII code x = 067097101115097114, which is the clear code we need to encrypt. Now let’s encrypt X.

1. Find two large prime numbers P and Q, calculate their product n = P * q, and let m = (P – 1) (Q – 1)
2. Find a number e such that E and m have no common divisor except 1
3. Find a number d satisfying e multiplied by D divided by m, and E * D mod M = 1

Now e is the public key, which can be disclosed to anyone for encryption

D is the private key, used for decryption, must keep their own

The n that connects the public key and private key is public. The reason why this can be made public is that it is very easy to calculate n according to p q, but it is very difficult to decompose n into two large prime numbers p q, so it is difficult to crack the existing computer computing power

Now encryption:

Pow (x, e) mod n = y, y is ciphertext, now without D (private key), the immortal can’t calculate x (plaintext)

decrypt:

Pow (y, d) mod n = x, X is plaintext, and plaintext comes out.

Mathematics is not very magical, now can be considered sa.key = D sa.pub = E

digital signature

Suppose you write a letter to your boss, the content is “boss I admire you”, and then ask colleagues to send the letter to your boss. How can you make sure that the letter is written by you, and how can you prevent colleagues from changing the letter to “boss you are an sb” in the process of sending the letter?

You can do this. First, you generate a key pair, give the public key to the boss, then make a hash digest of the content of the letter, and then encrypt the digest with the private key. The result is the signature

In this way, after the boss gets the letter, he decrypts it with the public key and finds that the hash value is consistent with the hash value of the letter. In this way, he determines that the letter is written by you

So digital signature is an application of encryption technology. The difference between digital signature and fully encrypted information is that the information here is public. Your colleagues can see that you flatter your boss.

digital certificate

Root certificate and certificate

Usually, we need to apply for a certificate from an “authority” when we configure the HTTPS service.

The process is as follows:

1. The website creates a key pair that provides public key and organizational and personal information to authorities
2. Certificate issued by authority
3. Friends browsing the web use the root certificate public key of the authority to decrypt the signature, compare the abstract, and determine the legitimacy
4. The client verifies the valid time of domain name information, etc. (browsers basically have CA public keys of major authorities built in)

This certificate contains the following contents:

1. Applicant’s public key
2. Applicant organization and personal information
3. CA information of issuing agency, valid time, serial number, etc
4. Signature of the above information

Root certificate is also known as self signed certificate, which is the certificate issued by oneself. Ca (certificate authority) is called certificate authority, and the CA certificate in k8s is the root certificate.

Kubernetes certificate

With the above foundation, let’s start…

First classification:

Key pair: sa.key sa.pub
Root certificate: ca.crt etcd/ca
Private key: ca.key etc.
Other certificates

First of all, other certificates are issued by Ca root certificate. Kubernetes and etcd use different ca. the important point is whether the certificate is used for client verification or server verification. Let’s look at it one by one

Service account key pair sa.key sa.pub

It can be used by Kube controller manager sa.key The token is signed by the master node through the public key sa.pub Verify the signature
For example, Kube proxy runs in the form of pod. In pod, service account and Kube apisever are directly used for authentication. In this case, there is no need to create a certificate for Kube proxy, and token verification is directly used

Root certificate

pki/ca.crt
pki/ca.key

K8s cluster certification authority

Apiserver certificate

pki/apiserver.crt
pki/apiserver.key

Kubelet certificate

pki/apiserver-kubelet-client.crt
pki/apiserver-kubelet-client.key

If kubelet wants to actively access Kube apisever, Kube apisever also needs to initiate requests to kubelet,
Therefore, both sides need to have their own root certificate, and the server certificate and client certificate issued by using the root certificate. In Kube apisever, the server certificate used for HTTPS access and the client certificate with CN user name information are generally specified
In the startup configuration of kubelet, only the CA root certificate is specified, but the server certificate for HTTPS access is not specified. When generating the server certificate, the server address or host name is usually specified,
Kube apisever changes relatively infrequently, so the IP or host name / domain name used as Kube apisever can be pre assigned at the beginning of cluster creation,
However, due to the frequent changes of kubelet deployed on node due to the change of cluster size, it is impossible to predict all IP information of node, so the server certificate is not explicitly specified on kubelet,
Instead, only the CA root certificate is specified, and kubelet automatically generates the server certificate according to the local host information and saves it in the configured cert dir folder

Aggregation certificate

Proxy root certificate:

pki/front-proxy-ca.crt
pki/front-proxy-ca.key

Client certificate issued by proxy root certificate:

pki/front-proxy-client.crt
pki/front-proxy-client.key

For example, when using kubectl proxy to access, Kube apisever uses this certificate to verify whether the client certificate is issued by itself.

Etcd root certificate

pki/etcd/ca.crt
pki/etcd/ca.key

Peer certificate for communication between etcd nodes

Issued by root certificate

pki/etcd/peer.crt
pki/etcd/peer.key

Client certificate of liveness probe in pod

pki/etcd/healthcheck-client.crt
pki/etcd/healthcheck-client.key

You can view the yaml live detection configuration:

Liveness:       exec [/bin/sh -ec ETCDCTL_API=3 etcdctl \
  --endpoints=https://[127.0.0.1]:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key get foo] \
  delay=15s timeout=15s period=10s #success=1 #failure=8

API server’s certificate for accessing etcd

pki/apiserver-etcd-client.crt
pki/apiserver-etcd-client.key

Note the difference between the client certificate and the server certificate. The server certificate usually checks the address, domain name, etc.

code implementation

Kubedm wrote the certificate time as one year, which is a sad story. As a result, sealos had to separate the logic of certificate generation to allow the installation to support arbitrary expiration time.

Let’s have an in-depth experience of kubedm certificate generation according to the source code. It may be a bit tired to directly look at the kubedm code. It’s easier to understand the core code separated from the sealos / cert directory.

In order to highlight the core logic, some error handling details are deleted from the code. You can read them if you are interested github.com/fanux/sealos/cert Source code

Key pair generation

// create sa.key sa.pub for service Account
func GenerateServiceAccountKeyPaire(dir string) error {
    key, err := NewPrivateKey(x509.RSA)
    pub := key.Public()
    err = WriteKey(dir, "sa", key)
    return WritePublicKey(dir, "sa", pub)
}

Generate the private key. The keytype here is x509. RSA

func NewPrivateKey(keyType x509.PublicKeyAlgorithm) (crypto.Signer, error) {
    if keyType == x509.ECDSA {
        return ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
    }
    return rsa.GenerateKey(rand.Reader, rsaKeySize)
}

Generate CA certificate

Will return ca.crt (self signed certificate) ca.key (private key)

func NewCaCertAndKey(cfg Config) (*x509.Certificate, crypto.Signer, error) {
    key, err := NewPrivateKey(x509.UnknownPublicKeyAlgorithm)
    cert, err := NewSelfSignedCACert(key, cfg.CommonName, cfg.Organization, cfg.Year)
    return cert, key, nil
}

According to the private key, a self signed certificate is generated. NotAfter is the expiration time of the certificate. We add a variable instead of writing it dead

// NewSelfSignedCACert creates a CA certificate
func NewSelfSignedCACert(key crypto.Signer, commonName string, organization []string, year time.Duration) (*x509.Certificate, error) {
    now := time.Now()
    tmpl := x509.Certificate{
        SerialNumber: new(big.Int).SetInt64(0),
        Subject: pkix.Name{
            CommonName:   commonName,
            Organization: organization,
        },
        NotBefore:             now.UTC(),
        NotAfter:              now.Add(duration365d * year).UTC(),
        KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
        BasicConstraintsValid: true,
        IsCA:                  true,
    }

    certDERBytes, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, key.Public(), key)
    return x509.ParseCertificate(certDERBytes)
}

It is very useful to pay attention to the common name and organization fields. For example, we create a k8s user to specify which user group the user belongs to, corresponding to the above two fields.

For example, if fanux in the certificate belongs to sealyun, generating a kubeconfig is equivalent to having fanux as the user. In this way, k8s only needs to verify the signature when doing authentication, and does not need to visit it
Database to do authentication, which is very conducive to the horizontal expansion of apiserver.

Generate other certificates

The key pair is generated by itself, and then the root certificate information will be brought with it when signing the certificate

func NewCaCertAndKeyFromRoot(cfg Config, caCert *x509.Certificate, caKey crypto.Signer) (*x509.Certificate, crypto.Signer, error) {
    key, err := NewPrivateKey(x509.UnknownPublicKeyAlgorithm)
    cert, err := NewSignedCert(cfg, key, caCert, caKey)

    return cert, key, nil
}

At this time, you must have a common name. In addition, you must specify whether to use the server or the client. Pay attention to the difference between the above and selfsign

// NewSignedCert creates a signed certificate using the given CA certificate and key
func NewSignedCert(cfg Config, key crypto.Signer, caCert *x509.Certificate, caKey crypto.Signer) (*x509.Certificate, error) {
    serial, err := rand.Int(rand.Reader, new(big.Int).SetInt64(math.MaxInt64))
    if len(cfg.CommonName) == 0 {
        return nil, errors.New("must specify a CommonName")
    }
    if len(cfg.Usages) == 0 {
        return nil, errors.New("must specify at least one ExtKeyUsage")
    }

    certTmpl := x509.Certificate{
        Subject: pkix.Name{
            CommonName:   cfg.CommonName,
            Organization: cfg.Organization,
        },
        DNSNames:     cfg.AltNames.DNSNames,
        IPAddresses:  cfg.AltNames.IPs,
        SerialNumber: serial,
        NotBefore:    caCert.NotBefore,
        NotAfter:     time.Now().Add(duration365d * cfg.Year).UTC(),
        KeyUsage:     x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
        ExtKeyUsage:  cfg.Usages,
    }
    certDERBytes, err := x509.CreateCertificate(rand.Reader, &certTmpl, caCert, key.Public(), caKey)
    return x509.ParseCertificate(certDERBytes)
}

All certificates in kubernetes

Root certificate list

var caList = []Config{
    {
        Path:         BasePath,
        BaseName:     "ca",
        CommonName:   "kubernetes",
        Organization: nil,
        Year:         100,
        AltNames:     AltNames{},
        Usages:       nil,
    },
    {
        Path:         BasePath,
        BaseName:     "front-proxy-ca",
        CommonName:   "front-proxy-ca",
        Organization: nil,
        Year:         100,
        AltNames:     AltNames{},
        Usages:       nil,
    },
    {
        Path:         EtcdBasePath,
        BaseName:     "ca",
        CommonName:   "etcd-ca",
        Organization: nil,
        Year:         100,
        AltNames:     AltNames{},
        Usages:       nil,
    },
}

List of other signing certificates

var certList = []Config{
    {
        Path:         BasePath,
        BaseName:     "apiserver",
        CAName:       "kubernetes",
        CommonName:   "kube-apiserver",
        Organization: nil,
        Year:         100,
        Altnames: altnames {// you need to add the user-defined domain name of the server IP to the actual installation
            DNSNames: []string{  
                "apiserver.cluster.local",
                "localhost",
                "master",
                "kubernetes",
                "kubernetes.default",
                "kubernetes.default.svc",
            },
            IPs: []net.IP{
                {127,0,0,1},
            },
        },
        Usages: [] x509. Extkeyusage {x509. Extkeyusageserverauth}, // used for server verification
    },
    {
        Path:         BasePath,
        BaseName:     "apiserver-kubelet-client",
        CAName:       "kubernetes",
        CommonName:   "kube-apiserver-kubelet-client",
        Organization: []string{"system:masters"},
        Year:         100,
        AltNames:     AltNames{},
        Usages:       []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
    },
    {
        Path:         BasePath,
        BaseName:     "front-proxy-client",
        CAName:       "front-proxy-ca",
        CommonName:   "front-proxy-client",
        Organization: nil,
        Year:         100,
        AltNames:     AltNames{},
        Usages:       []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
    },
    {
        Path:         BasePath,
        BaseName:     "apiserver-etcd-client",
        CAName:       "etcd-ca",
        CommonName:   "kube-apiserver-etcd-client",
        Organization: []string{"system:masters"},
        Year:         100,
        AltNames:     AltNames{},
        Usages:       []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
    },
    {
        Path:         EtcdBasePath,
        BaseName:     "server",
        CAName:       "etcd-ca",
        CommonName: "etcd", // kubedm etcd server certificate common name uses the node name, which also needs to be changed when calling
        Organization: nil,
        Year:         100,
        Altnames: altnames {}, // when calling, you need to add node name, node IP, etc
        Usages:       []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth},
    },
    {
        Path:         EtcdBasePath,
        BaseName:     "peer",
        CAName:       "etcd-ca",
        CommonName: "etcd peer", // same as etcd server
        Organization: nil,
        Year:         100,
        Altnames: altnames {}, // same as etcd server
        Usages:       []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth},
    },
    {
        Path:         EtcdBasePath,
        BaseName:     "healthcheck-client",
        CAName:       "etcd-ca",
        CommonName:   "kube-etcd-healthcheck-client",
        Organization: []string{"system:masters"},
        Year:         100,
        AltNames:     AltNames{},
        Usages:       []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
    },
}

It is very important to note that the IP and domain name should be added when installing the certificate verified by the server, and the common name of etcd should also be set to node name.

Look at the last generated certificate information:

apiserver:

[[email protected] pki]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout
Certificate:
...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Mar 31 09:18:06 2020 GMT
            Not After : Mar  8 09:18:06 2119 GMT
        Subject: CN=kube-apiserver
...
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Subject Alternative Name: 
                DNS:iz2ze4ry74x8bh3cweeg69z, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver.cluster.local, DNS:apiserver.cluster.local, IP Address:10.96.0.1, IP Address:172.16.9.192, IP Address:127.0.0.1, IP Address:172.16.9.192, IP Address:172.16.9.193, IP Address:172.16.9.194, IP Address:10.103.97.2
    Signature Algorithm: sha256WithRSAEncryption

etcd server:

[[email protected] pki]# openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1930981199811083392 (0x1acc392ba2b27c80)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=etcd-ca
        Validity
            Not Before: Mar 31 09:18:07 2020 GMT
            Not After : Mar  8 09:18:07 2119 GMT
        Subject: CN=iz2ze4ry74x8bh3cweeg69z
...
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name: 
                DNS:iz2ze4ry74x8bh3cweeg69z, DNS:localhost, IP Address:172.16.9.192, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1
    Signature Algorithm: sha256WithRSAEncryption

Generate user certificate and kubeconfig

Now there is an intern fanux coming to the company. He also wants to use k8s. He is not sure to give him the kubeconfig of admin. What should he do?
With the above foundation, I will further teach you how to assign a separate kubeconfig to fanux

1. Load root certificate, and private key from disk
2. Generate the certificate of fanux. Common name is fanux
3. Encoding to PEM format
4. Write kubeconfig, write disk

func GenerateKubeconfig(conf Config) error{
    certs, err := cert.CertsFromFile(conf.CACrtFile)
    caCert := certs[0]
    cert := EncodeCertPEM(caCert)
    caKey,err := TryLoadKeyFromDisk(conf.CAKeyFile)
    //Here conf.User It's fanux, conf.Groups It can be multiple user groups
    clientCert,clientKey,err := NewCertAndKey(caCert,caKey,conf.User,conf.Groups,conf.DNSNames,conf.IPAddresses)
    encodedClientKey,err := keyutil.MarshalPrivateKeyToPEM(clientKey)
    encodedClientCert := EncodeCertPEM(clientCert)
    //Building triple information of kubeconfig
    config := &api.Config{
        Clusters: map[string]*api.Cluster{
            conf.ClusterName: {
                Server:  conf.Apiserver , // the cluster address is as follows: https://apiserver.cluster.local :6443
                Certificateauthority data: Cert, // root certificate in PEM format, used for HTTPS
            },
        },
        Contexts: map[string]*api.Context{
            CTX: {// triple information, user name fanux, cluster name above, and namespace are not written here
                Cluster:  conf.ClusterName, 
                AuthInfo: conf.User,
            },
        },
        AuthInfos:      map[string]* api.AuthInfo {// user information, so it's useless to change the user in kubeconfig directly, because k8s only uses the name in the certificate
            conf.User:&api.AuthInfo{
                Clientcertificatedata: encoded clientcert, // user certificate in PEM format
                Clientkeydata: encodedclientkey, // user private key in PEM format
            },
        },
        Currentcontext: CTX, // current context, kubeconfig can support multi-user and multi cluster
    }

    err = clientcmd.WriteToFile(*config, conf.OutPut)
    return nil
}

User certificate and private key are generated. Just like the above signature certificate, user is fanux and group is user group

func NewCertAndKey(caCert *x509.Certificate, caKey crypto.Signer, user string, groups []string, DNSNames []string,IPAddresses []net.IP) (*x509.Certificate, crypto.Signer, error) {
    key,err := rsa.GenerateKey(rand.Reader, 2048)
    serial, err := rand.Int(rand.Reader, new(big.Int).SetInt64(math.MaxInt64))

    certTmpl := x509.Certificate{
        Subject: pkix.Name{
            CommonName:   user,
            Organization: groups,
        },
        DNSNames:     DNSNames,
        IPAddresses:  IPAddresses,
        SerialNumber: serial,
        NotBefore:    caCert.NotBefore,
        NotAfter:     time.Now().Add(time.Hour * 24 * 365 * 99).UTC(),
        KeyUsage:     x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
        ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
    }
    certDERBytes, err := x509.CreateCertificate(rand.Reader, &certTmpl, caCert, key.Public(), caKey)
    cert,err := x509.ParseCertificate(certDERBytes)
    return cert,key,nil
}

Then the kubeconfig of the little partner is generated. At this time, there are no permissions:

kubectl --kubeconfig ./kube/config get pod
Error from server (Forbidden): pods is forbidden: User "fanux" cannot list resource "pods" in API group ...

Finally, it’s OK to play RBAC. Here you can directly bind an administrator’s permission

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: user-admin-test
subjects:
- kind: User
  name: "fanux" # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin  # using admin role
  apiGroup: rbac.authorization.k8s.io

summary

Certificate and k8s authentication principle is very useful in cluster installation and development of multi tenant container platform. I hope this article can give you a comprehensive understanding.