K8s and storage — Interpretation of flexvolume



The great thing about k8s is its scalability. In the storage field, it supports flex volume and CSI. Today I’ll focus on flexvolume. Flexvolume is a storage plug-in extension mode supported by kubernetes v1.8 +. Similar to CNI plug-ins, it requires external plug-ins to put the binaries in a pre configured path (such as / usr / libexec / kubernetes / kubelet plugins / volume / exec /), and install all the required dependencies in the system. As you can imagine, this is an out tree extension mode, and there is no need to add a new storage plug-in to change the source code of k8s.

Flexvolume interface

Some interfaces are provided by the government. When we implement custom storage plug-ins, we need to implement some interfaces. The reason why we say some of them is mainly based on our own needs. For example, when I implemented dynamic hostpath (host path contains podid), I only implemented init and mount and unmount interfaces.

Flexvolume interfaces include

  • Init: called when kubelet / Kube controller manager initializes a storage plug-in, the plug-in needs to return whether it needs attach and detach operations
  • Attach: mount the storage volume to the node
  • Detach: unmount the storage volume from the node
  • Wait for attach: wait for the attach operation to succeed (the timeout is 10 minutes)
  • Isattached: check if the storage volume is mounted
  • Mount device: mount the device to the specified directory for subsequent bind mount use
  • Unmountdevice: unmount the device
  • Mount: mount the storage volume to the specified directory
  • Umount: unmount the storage volume

Driver output

Flexvolume wants the driver to return to the operating state in the following format

    "status": "<Success/Failure/Not supported>",
    "message": "<Reason for success/failure>",
    "device": "<Path to the device attached. This field is valid only for attach & waitforattach call-outs>"
    "volumeName": "<Cluster wide unique name of the volume. Valid only for getvolumename call-out>"
    "attached": <True/False (Return true if volume is attached on the node. Valid only for isattached call-out)>
    "capabilities": <Only included as part of the Init response>
        "attach": <True/False (Return true if the driver implements attach and detach)>

Relationship with kubelet

The user-defined storage plug-ins mentioned above are placed in / usr / libexec / kubernetes / kubelet plugins / volume / exec /, and the specific directory is / usr / libexec / kubernetes / kubelet plugins / volume / exec / < vendor ~ driver > / < driver >.
After the user-defined plug-in is used in the spec of pod, the format is < vendor ~ driver > / < driver >, in fact, kubelet calls the corresponding plug-in according to the interface specification.
Therefore, we should pay attention to the following points:

  • During the debug process, the flexvolume log needs to be in the kubelet.log View in
  • The deployment method is generally daemonset. This ensures that there will be a custom plug-in for each node.

Since k8s1.8, it supports dynamic plug-in discovery, and flexvolume supports dynamic detection of driver. When the system is running, the driver can be installed, upgraded / demoted and unloaded, instead of requiring the driver to exist or having to restart kubelet or controller manager during system initialization.

Official demo

The official provides some demos for you to refer to.

How to solve the problem that kubelet often fails to find flexvolume

In the process of actual combat, when using custom plug-ins, we often have the following alarms:

Unable to mount volumes for pod "hermes-devices-push-myd-79f4577cc9-dqx9v_cbs(a0bc2b86-d140-11e9-9a8e-fa163e4660a0)": timeout expired waiting for volumes to attach or mount for pod "cbs"/"hermes-devices-push-myd-79f4577cc9-dqx9v". list of unmounted volumes=[log-dir]. list of unattached volumes=[log-dir default-token-kf6sr shareit-aksk-aws-conf shareit-aksk-huawei-conf]

Our custom plug-in is used in log dir.

After checking the kubelet log, we found that many kubelet appearedno volume plugin matchedError messages for. Combined with the relevant issue of k8s and analysis of kubelet code, it is found that this is actually a bug of kubelet. Unfortunately, this bug has not been solved.

Of course, every time the corresponding plug-in cannot be found, restart kubelet.

However, it is not a long-term plan after all. By studying the code, in fact, the mechanism of dynamically discovering plug-ins is implemented through inotify. It monitors the file change events in the / usr / libexec / kubernetes / kubelet plugins / volume / exec / directory, and makes corresponding processing. So sometimes when we can’t change the kubelet code, we can add the function of regular refresh when implementing custom plug-ins. For example, let’s change the last access time and modification time of the corresponding driver file.

Recommended Today

Build HTTP service with C + + Mongoose

Mongoose source code address:https://github.com/cesanta/mo… Mongoose user manual:https://www.cesanta.com/devel… Mngoose set up HTTP service #include <string> #include “mongoose.h” using namespace std; static const char *s_http_port = “8000”; static void ev_handler(mg_connection *nc, int ev, void *ev_data) { struct http_message *hm = (struct http_message *) ev_data; if (ev == MG_EV_HTTP_REQUEST) { std::string uri; if (hm->uri.p && hm->uri.p[0] == ‘/’) […]