KubeCube user management and authentication


KubeCube (https://kubecube.io) is a lightweight enterprise-level container platform recently open-sourced by Netease Shufan, which provides enterprises with visual management of kubernetes resources and unified multi-cluster multi-tenant management functions. The KubeCube community will explain the design features and technical implementation of KubeCube through a series of technical articles to help developers and users understand and get started with KubeCube faster. This article is the third one, focusing on the implementation of user management and identity authentication in KubeCube.

User Management

All Kubernetes clusters have two types of users: service accounts managed by Kubernetes and normal users.

Kubernetes assumes that normal users are managed by a cluster-agnostic service in one of the following ways:

* Administrator responsible for distributing private keys

* User databases like Keystone or Google Accounts

* A file containing a list of usernames and passwords

For this reason, Kubernetes does not contain objects to represent common user accounts. Ordinary user information cannot be added to the cluster through API calls.

According to the official Kubernetes statement, Kubernetes itself does not directly provide user management features, does not support common user objects, and does not store any information about common users. If you need to create a user, you need to create a private key and certificate for the user, and use the certificate for identity authentication. Moreover, since no user information is stored, the cluster administrator cannot centrally manage users and has no awareness of other users. Therefore, KubeCube first redefines the concept of user, that is, it provides the resource type of User, stores user information, manages users, and facilitates subsequent identity authentication and permission verification.

apiVersion: user.kubecube.io/v1
kind: User
    name: login account, unique user ID, user-defined, non-repeatable, non-modifiable
    password: password, required, the system will encrypt the password with md5 and salt and save it
    displayName: username
    email: email
    phone: phone
    language: language: en/ch
    loginType: User login method: normal/ldap/github/...
    state: user status: normal/forbidden
    lastLoginTime: last login time
    lastLoginIp: last login IP

Users can be manually created by the administrator on the front-end page, or automatically created by the system when logging in for the first time using external authentication. Therefore, users can be divided into ordinary registered users of the system and third-party authorized login users in terms of registration methods. However, for these two creation methods, the corresponding User cr is created in the management and control cluster. Then Warden’s resource synchronization manager will synchronize the cr from the control cluster to the computing cluster, so as to facilitate subsequent multi-cluster unified authentication.

In this way, on the user management page, you only need to query the User resources in the management and control cluster to realize centralized management of users. Moreover, it is easy to add users, query users, and modify user meta information.


In KubeCube, local authentication and external authentication are supported. Local authentication means that a common user is created in KubeCube, and the user then uses the username and password registered at the time of creation to log in and authenticate. External authentication refers to accessing KubeCube by authenticating the user’s identity through a third-party authentication platform without creating a user. The implementation of these two authentication methods will be introduced respectively below.

local authentication

In KubeCube, user authentication is mainly done through JWT (JSON Web Token).

When using local authentication to log in, the user needs to enter the user name and password. KubeCube will query the User cr in the cluster according to the user name and compare the passwords. If the User is queried and the passwords are consistent, the login is considered successful. After KubeCube updates the user’s login status, it will generate a JWT based on the user name and splice it into a Bearer Token, store it in the cookie and return it.

KubeCube user management and authentication

The code after verifying the username and password successfully when the user logs in is as follows:

  // generate token and return
    authJwtImpl := jwt.GetAuthJwtImpl()
    token, err := authJwtImpl.GenerateToken(&v1beta1.UserInfo{Username: name})
    if err != nil {
        response.FailReturn(c, errcode.AuthenticateError)
    bearerToken := jwt.BearerTokenPrefix + " " + token
    c.SetCookie(constants.AuthorizationHeader, bearerToken, int(authJwtImpl.TokenExpireDuration), "/", "", false, true)

    user.Spec.Password = ""
    response.SuccessReturn(c, user)

After the user successfully logs in, for each subsequent request, the front-end will bring the JWT to the request through the cookie, and the back-end authentication middleware will verify the JWT. If it is valid, a new token will be generated and returned, and the above process will be cycled. In this way, even though the default valid time of the JWT generated in KubeCube is 1 hour, as long as the user continues to visit, the JWT will be continuously refreshed so that the user is always logged in.

KubeCube user management and authentication

Part of the code of the authentication middleware is as follows:

func Auth() gin.HandlerFunc {
    return func(c *gin.Context) {
        if !withinWhiteList(c.Request.URL, c.Request.Method, whiteList) {
            authJwtImpl := jwt.GetAuthJwtImpl()
      userToken, err := token.GetTokenFromReq(c.Request)
      if err != nil {
        response.FailReturn(c, errcode.AuthenticateError)

      newToken, respInfo := authJwtImpl.RefreshToken(userToken)
      if respInfo != nil {
        response.FailReturn(c, errcode.AuthenticateError)

      v := jwt.BearerTokenPrefix + " " + newToken

      c.Request.Header.Set(constants.AuthorizationHeader, v)
      c.SetCookie(constants.AuthorizationHeader, v, int(authJwtImpl.TokenExpireDuration), "/", "", false, true)

external authentication

The implementation of external authentication is currently mainly divided into three types, namely general authentication, LDAP authentication and OAuth2 authentication.

Universal Certification

To make it easier for users to connect to their own authentication system, KubeCube supports a general authentication method. Users can enable the general authentication method and configure the address of the authentication system, so that users will go to their own authentication system for authentication every time they access KubeCube. After the authentication is passed, the user name of the user needs to be returned to KubeCube, and KubeCube will still generate a corresponding Bearer Token based on the user name and put it in the header for subsequent permission verification. The main logic code is implemented as follows:

func Auth() gin.HandlerFunc {
    return func(c *gin.Context) {
        if !withinWhiteList(c.Request.URL, c.Request.Method, whiteList) {
            authJwtImpl := jwt.GetAuthJwtImpl()
            if generic.Config.GenericAuthIsEnable {
                h := generic.GetProvider()
                user, err := h.Authenticate(c.Request.Header)
                if err != nil || user == nil {
                    clog.Error("generic auth error: %v", err)
                    response.FailReturn(c, errcode.AuthenticateError)
                newToken, error := authJwtImpl.GenerateToken(&v1beta1.UserInfo{Username: user.GetUserName()})
                if error != nil {
                    response.FailReturn(c, errcode.AuthenticateError)
                b := jwt.BearerTokenPrefix + " " + newToken
                c.Request.Header.Set(constants.AuthorizationHeader, b)

LDAP authentication

  1. When the user selects the LDAP login method, the user enters the user name and password. First, it checks whether the user exists in the cluster, and whether the user is in the “disabled” state. If it does not exist or exists and is in a normal state, start LDAP authentication
  2. As an LDAP client, KubeCube obtains the user’s username and password, and sends an administrator binding request message to the LDAP server with the administrator DN and administrator password as parameters to obtain query permissions.
  3. After receiving the administrator binding request message, the LDAP server verifies whether the administrator DN and administrator password are correct. If the administrator DN and administrator password are correct, an administrator binding response message of successful binding will be sent to KubeCube.
  4. After KubeCube receives the binding response message, it uses the username input by the user to construct a filter condition, and sends a user DN query request message to the LDAP server. For example: the construction filter condition is CN=User2.
  5. After receiving the user DN query request message, the LDAP server searches the user DN according to the query starting point, query range, and filter conditions in the message. If the query is successful, a successful query response message will be sent to KubeCube. There can be one or more user DNs obtained from the query. If not one user is obtained, it is considered that the user name or password is wrong, and the authentication fails.
  6. KubeCube sends a user binding request message to the LDAP server based on the user DN obtained from the query and the password entered by the user as parameters.
  7. After receiving the user binding request message, the LDAP server checks whether the password entered by the user is correct.
  • If the password entered by the user is correct, a binding response message indicating successful binding will be sent to KubeCube.
  • If the password entered by the user is incorrect, a binding failure response message will be sent to KubeCube. KubeCube takes the next queried user DN as a parameter, and continues to send binding requests to the LDAP server until a DN is successfully bound. If all user DNs fail to bind, KubeCube will notify the user that the authentication failed.

After successful authentication, the logic is the same as that of local authentication: if the user does not exist in the cluster, create User cr according to the username; and generate the corresponding Bearer Token based on the username and store it in the cookie, which will be carried in the next request for identification user ID.

KubeCube user management and authentication

OAuth2 authentication

In KubeCube, OAuth2 authentication adopts the authorization code mode, because this mode is the authorization mode with the most complete functions and the strictest process. The usual authentication process of OAuth2 is:

  1. The user accesses the client, which directs the former to the authentication server.
  2. The user chooses whether to authorize the client.
  3. Assuming that the user grants authorization, the authentication server directs the user to the “redirection URI” (redirection URI) specified by the client in advance, and attaches an authorization code at the same time.
  4. The client receives the authorization code, attaches the previous “redirect URI”, and requests a token from the authentication server. This step is done on the server in the background of the client and is not visible to the user.
  5. The authentication server checks the authorization code and the redirection URI, and after confirming that they are correct, sends an access token (access token) and a refresh token (refresh token) to the client.

In the implementation of KubeCube, take GitHub login as an example:

The user selects GitHub authentication login when logging in, and the front end forwards the request to GitHub;
GitHub asks the user to agree to authorize KubeCube;
If the user agrees, GitHub will redirect back to KubeCube (/oauth/redirect) and send back an authorization code (code);
KubeCube uses the authorization code to request a token (access_token) from GitHub;
GitHub return token (access_token);
KubeCube uses the token (access_token) to request user information data from GitHub;
Query the cluster, if the user does not exist, create User cr according to the user information;
Generate a Bearer Token for accessing the cluster according to the user name, and return authentication success;
The front end stores the Bearer Token in a cookie and carries it in the next request.
OpenAPI certification
Based on the above design scheme, it can be easily inferred that the implementation of OpenAPI authentication is also completed through JWT. User is bound to each group of AK and SK, and the corresponding User is queried through AK and SK, and then a Bearer Token is generated and returned through the User.Name. In the next request, the user needs to carry the Token in the cookie or header, and the KubeCube authentication middleware can parse out the user’s identity through the Token to complete the authentication.

cluster authentication
After the middleware completes the identity authentication, the refresh token will be refreshed, but if you directly carry the token in the request header to request kube-apiserver to complete the cluster authentication, you need to modify the authentication backend of kube-apiserver when deploying KubeCube, that is, modify kube-apiserver Configuration. This will cause intrusion to the native kubernetes cluster, greatly increasing the deployment cost and operation and maintenance cost of KubeCube. Therefore, we need to build another module to help complete cluster authentication – auth-proxy.

When a user accesses KubeCube to request kubernetes resources, after entering the transparent transmission interface through the authentication middleware, it will go to the auth-proxy module; auth-proxy resolves the Bearer Token in the request into the corresponding User; and then uses the User impersonation In this way, the request proxy is sent to the kube-apiserver, that is, the “admin” user is disguised as the current user to request the kube-apiserver, thereby “skipping” authentication and facilitating subsequent authentication.

KubeCube’s user management system is mainly implemented based on User CRD; the authentication system supports local and external authentication methods. Local authentication is based on JWT. External authentication also needs to create a User crd in the cluster after the third-party authentication platform passes the authentication. Follow-up user management, authority binding, etc. For cluster authentication, the Impersonation method “skip authentication” provided by Kubernetes is mainly used. The overall design and implementation are relatively simple, adhering to the lightweight design concept of KubeCube.

For more information see:

KubeCube official website:https://www.kubecube.io/

KubeCube source code:https://github.com/kubecube-i…

In-depth interpretation of KubeCube multi-cluster management

KubeCube multi-level tenant model

KubeCube open source: six features that simplify the implementation of Kubernetes

NetEase Shufan more open source projects

About the author: Jiahui, senior engineer of NetEase Shufan, core member of KubeCube community