Container image security in the cloud native era (Part 1)


Hello, I’m Zhang Jintao.

Kubernetes, as the cornerstone of the cloud, has brought us great convenience. More and more companies have also applied kubernetes to the production environment. However, while enjoying the convenience it brings, we also need to pay attention to some potential safety hazards.

In this article, I will focus on youContainer image securityRelevant contents.

Usually, we mentionContainer image security, mainly refers to the following two aspects:

  • The security of the image’s own content;
  • Security of image distribution process;

Security of mirroring its own content

To talk about the security of the image’s own content, we need to know what the image is and what its content is.

What is mirroring

We takedebianTake the image as an example, pull the latest image, save it as a tar file, and then unzip it:

➜  ~ mkdir -p debian-image
➜  ~ docker pull debian
Using default tag: latest
latest: Pulling from library/debian
647acf3d48c2: Pull complete 
Digest: sha256:e8c184b56a94db0947a9d51ec68f42ef5584442f20547fa3bd8cbd00203b2e7a
Status: Downloaded newer image for debian:latest
➜  ~ docker image save -o debian-image/debian.tar debian
➜  ~ ls debian-image 
➜  ~ tar -C debian-image -xf debian-image/debian.tar 
➜  ~ tree -I debian.tar debian-image 
├── 827e5611389abf13dad1057e92f163b771febc0bcdb19fa2d634a7eb0641e0cc.json
├── b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── manifest.json
└── repositories

1 directory, 6 files

After unpacking, we see that it is a pile of JSON files andlayer.tarThe combination of documents, we analyze them againlayer.tarUnzip:

➜  ~ tar -C debian-image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda -xf debian-image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda/layer.tar
➜  ~ tree -I 'layer.tar|json|VERSION'  -L 1 debian-image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda
├── bin
├── boot
├── dev
├── etc
├── home
├── lib
├── lib64
├── media
├── mnt
├── opt
├── proc
├── root
├── run
├── sbin
├── srv
├── sys
├── tmp
├── usr
└── var

19 directories, 0 files

The unzipped directory structure must be familiar to you. Yes, this isrootfsDirectory structure.

If we use some application images built by ourselves, you will also find the corresponding files of the application after several times of decompression.

How to ensure the security of the image’s own content

As we saw earlier, the container image isrootfsAnd applications, as well as a combination of some configuration files. Therefore, to ensure the security of its own content, it is mainly considered from the following aspects:


Corresponding to our actual situation,rootfsIt is usually provided by the basic (system) image we use, or it can also be considered when we build the image,DockerfileofFROMField provided by the configured image.

In this regard, to achieve security, we need to:

  • Use images from trusted sources, such as those officially maintained by docker;
  • Continuously scan and upgrade the vulnerability of the basic image;
  • You can also consider usingDistrolessMirror image, which can also be protected from attack to a certain extent;

application program

The application is actually provided by ourselves. In order to achieve security in this regard, we need to:

  • Continuously scan software vulnerabilities;
  • Update the dependent timely;
  • Consider transitioning from SDL (security development lifecycle) to devsecops;

configuration file

The configuration files contained in the image are provided by the image building tool. Generally, as long as we ensure that the image building tool used has not been tampered with or left any loopholes, there will basically be no major problems here.

On the whole, we can use similar methods directlyTrivyperhapsAnchore EngineAnd other image vulnerability scanning tools to help us ensure the security of image content. In addition, some image warehouses, such as harbor, have built-in image security scanning tools, or can be useddocker scanCommand to perform a security scan of the image.

Image distribution security

How are images distributed

Let’s first look at how container images are built and deployed into our kubernetes environment.

Container image security in the cloud native era (Part 1)

Fig. 1 is a brief process diagram of container image from creation to release and deployment

After writing the code, the developer pushes the code to the code warehouse. This triggers the CI to build. In this process, the image will be built and pushed to the image warehouse.

In the CD phase, the images in the image warehouse will be used and deployed to the target kubernetes cluster.

So how do attackers attack in this process?

Security issues in image distribution

Container image security in the cloud native era (Part 1)

Figure 2, image distribution deployment security example

As shown in the figure, in the link of image distribution and deployment, the upstream is the image warehouse and the downstream is the kubernetes cluster. For the mirror warehouse, even for the self built environment of the intranet, since our concept has changed from border based security to zero trust security, we will take the public warehouse as an example.

Attackers can hijack and replace with malicious images by some means, including directly attacking the image warehouse.

To ensure the security source and integrity of the image deployed in the kubernetes cluster, there are actually two main links:

  • Sign the image when building the image;
  • Verify the signature during image distribution and deployment; (continued in the next article)

Let’s take a look at them separately.

Mirrored labels and summaries

We usually have two choices when using container mirroring:

  • Labels, such asalpine:3.14.3
  • Summary, e.g[email protected]:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2

In most scenarios, we will use the tag directly because it is more readable. However, the mirrored content may change over time, because we may use the same label for images of different content, the most common is:latestLabel. Every time a new version is released, the image of the new version will continue to be used:latestTag, but the application version has been upgraded to the latest.

The main disadvantage of using summary is its poor readability. However, the summary of each image is unique. The summary is the hash value of sha256 of the image content. Therefore, we can ensure the uniqueness of the image through the summary.

You can see the label and summary information directly through the following example:

➜  ~ docker pull alpine:3.14.3                                                                          
3.14.3: Pulling from library/alpine
Digest: sha256:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2
Status: Image is up to date for alpine:3.14.3
➜  ~ docker image inspect alpine:3.14.3 | jq -r '.[] | {RepoTags: .RepoTags, RepoDigests: .RepoDigests}'
  "RepoTags": [
  "RepoDigests": [
    "[email protected]:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2"

So how to ensure the correctness / security of the image? This is the main problem solved by image signature.

Mirror signature solution

Digital signature is a well-known method to maintain the integrity of any data transmitted on the network. We have several general schemes for container image signature.

Docker Content Trust (DCT)

When transmitting general files, you may have had similar experiences, such as incomplete downloaded files due to network reasons; Or encounter the attack of middleman, resulting in the tampering and replacement of files.

In fact, images may encounter similar problems during distribution, which is the focus of our discussion here, that is, the main problem solved by docker content trust (DCT).

Docker content trust uses digital signatures and allows the client or runtime to verify the integrity and publisher of specific image labels. For use, that isdocker trustRelated functions provided by the command. Note: docker CE 17.12 and above is required.

As mentioned earlier, image records can have some labels in the following format:


Take the label as an example, DCT will be associated with a part of the label. Each image warehouse has a set of keys, which are used by the image publisher to sign the image label. (the image publisher can decide which labels to sign) the image warehouse can contain multiple images with signed and unsigned labels at the same time.

It should be noted here that if the image publisher pushes the signed latest image first and then the unsigned latest image, the latter image will not affect the content of the previous image (different from the place covered by the label above).

Container image security in the cloud native era (Part 1)

Figure 4: example of DCT image signature (the figure outlines the authentication process of logging in to the image warehouse)

In production, we can enable DCT to ensure that all images used are signed. If DCT is enabled, only trusted images (signed and verifiable images) can be pulled, run, or built.

Enabling DCT is a bit like applying “filter” to the image warehouse, that is, you can only see the signed image label, but not the unsigned image label. If the client does not have DCT enabled, it can see all the images.

Here, let’s take a quick look at the working process of DCT

Its trust in the mirror label is managed by using the signature key. When we first start DCT and use it, we will create a keyset. A key set consists of the following types of keys:

  • An offline key, which is the root of the image label DCT(It is difficult to recover the lost root key
  • The repository or tag key that signs the label tag key
  • Server managed keys, such as timestamp keys

Container image security in the cloud native era (Part 1)

Figure 5, example of mirror signature key

We just mentioned that the client uses DCT, which is ourdocker trustCommand, which is based on notary v1. By default, DCT is disabled in docker client. Settings are required to enableDOCKER_CONTENT_TRUST=1Environment variables.

The effect is as follows:

➜  ~ DOCKER_CONTENT_TRUST=1 docker pull alpine:3.12
Pull (1 of 1): alpine:[email protected]:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a[email protected]:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a: Pulling from library/alpine
188c0c94c7c5: Already exists        
Digest: sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a
Status: Downloaded newer image for [email protected]:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a
Tagging [email protected]:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a as alpine:3.12

Notary v1

We mentioned earlier that DCT is implemented based on notary V1, but this is not the focus of this article, so here is only a brief introduction to notary v1. Notary project address:…

Container image security in the cloud native era (Part 1)

Figure 6. Interaction flow of notary client, server and signature

Process 1 – identity authentication. Any connection without a token will be redirected to the authorization server (docker registry V2 identity authentication);

Process 2 – the client will log in to the authorization server through HTTPS authentication to obtain the token;

Process 3 – when the client uploads new metadata files, the server will check whether they have conflicts according to the previous versions, and verify the signature, verification and validity of the uploaded metadata;

Process 4 – once all uploaded metadata has been verified, the server will generate time stamps (and possibly snapshots), and then send them to sign for signature;

Process 5 – sign retrieves the encrypted private key from its database, decrypts the key, signs with them, and sends it back to the server;

Procedure 6 – the server stores the metadata uploaded by the client and generated by the server in the tuf library. The generated timestamp and snapshot metadata prove that the metadata uploaded by the client is the latest version of the trusted collection. After that, the server will notify the client that the upload is successful;

Procedure 7 – the client can now download the latest metadata from the server immediately. When the timestamp expires, the server will traverse the whole sequence, generate a new timestamp, request a sign signature, and store the newly signed timestamp in the database. Then, it sends the new timestamp together with other stored metadata to the requesting client;

As this project is a security project, although it is of great use, it is not active as a whole. The V2 version is now under development. Interested partners are welcome to join us.

Sigstore and cosign

Here is another project from the Linux foundation, called sigstore, which is mainly to provide some standard libraries / tools for better signature and verification. Of course, sigstore has gathered open source projects including cosign, fulcio and rekor, involving image signature verification and supply chain.

Container image security in the cloud native era (Part 1)

Figure 7. Introduction to sigstore

Cosign is one of the tools of sigstore, which is used to create, store and verify container image signatures in OCI registry. Cosign v1. 0 has been released in the second half of this year. Whether it can be stably used in the production environment remains to be tested. So far, cosign has released v1 Version 3.1. For detailed changes, please refer to its release note:…

Let’s take a look at how it performs mirror signature

➜  cosign cosign generate-key-pair                                                                                              
Enter password for private key: 
Enter password for private key again: 
Private key written to cosign.key
Public key written to
➜  cosign cosign sign --key cosign.key
Enter password for private key: %   
➜  cosign cosign verify --key 

Verification for --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.

[{"critical":{"identity":{"docker-reference":""},"image":{"docker-manifest-digest":"sha256:768845efa2a32bc5c5d83a6f7ec668b98f5db46585dd1918afc9695a9e653d2d"},"type":"cosign container image signature"},"optional":null}]

It seems relatively simple.


The above is about the content security of the image itself and the image signature verification in the image distribution security.

In the next article, I will introduce how to verify the signature during image distribution and deployment, and how to protect kubernetes cluster from attacks from unsigned or untrusted source images. Please look forward to it!

Welcome to subscribe to my article official account [moelove]

Container image security in the cloud native era (Part 1)