Are you right about container image security| IDCF


Container has become very popular in recent years. When we talk about container, we have to mention image. If container is one of the core contents of cloud computing era, image is the soul of container. So the security of the image is particularly important.
But from the following data, we can see that the security of the image is not so optimistic.

1) The vulnerability of Linux OS is increasing year by year

From less than 300 + in 2008 to 2000 + in 2018. And in recent two years, it shows an exponential upward trend.

2) The number of high-risk vulnerabilities is increasing year by year

From less than 2000 + in 2014 to 4000 + in 2018.

3) There are common vulnerabilities in docker image

The most popular images, such as nginx, Ubuntu, Java and so on, have dozens of vulnerabilities.

4) 40% of the respondents said that they did not take any safety measures in the CI stage.

Therefore, ensuring the image security is not only an important part of ensuring the security of cloud platform, but also an important topic in the governance system of devsecops.

1、 Characteristics of mirror image

The essence of mirror image, the source of mirror image and the making of mirror image together create the following characteristics of mirror image

  • complexity

The complexity of image content (including various libraries and packages of some OS, as well as some Python or Ruby files), the complexity of source (official and personal), and the complexity of dockerfile command (the different use of add and copy, the different use of CMD and entrypoint, and the different order of command create different images) make the image itself very complex. If security factors are considered, the image itself will be very complex, That’s more complicated.

  • Convenience

It is very convenient to make an image. As long as you can write dockerfile, you can make an image, or you can make an image of an existing container. Moreover, the use of images is also very convenient. A command (docker run) can transform a static image into a dynamic container. It is this convenience that causes another feature of images, that is, the following confusion.

  • Chaos

Everyone can make and manage images, which makes the types and quantity of images extremely huge. Moreover, there are dozens or even hundreds of images for a specific function, including official, personal, shared warehouse and private warehouse. There is no effective standard to regulate the management of these images, which makes the image good and bad, and it is difficult to select.

The above characteristics also make the security of image become a complex and easy to be ignored point. However, no matter how complicated it is, there are also means to complete this complicated work, which is the image scanning to be discussed below.

2、 The position of image scanning in SDLC

Image scanning is a powerful means to ensure image security. It usually occurs in the construction phase of the software development life cycle (SDLC)

Are you right about container image security| IDCF

The existing image scanning mostly relies on the scanning function (built-in image scanning tool) provided by the image warehouse. The general process is as follows:

Are you right about container image security| IDCF

  • Developers submit code changes;
  • Trigger a construction process;
  • To build the image (docker build);
    Docker push;
  • Image scanning (using the built-in scanning tool of image warehouse);
  • Image deployment (docker run).

This process has many disadvantages

  • Scan lag

The image scanning depends on the image scanning function of the image warehouse. Only when the image is pushed to the image warehouse, the image scanning will be carried out. This kind of delayed scanning will lead to the storage of vulnerable images in the warehouse. In devseops, what we want to do is to store secure and deployable images in the image warehouse.

  • Unable to terminate CI / CD process effectively

Mature devsecops CI / CD should be: when the image is detected to be unsafe, the CI / CD process should be terminated immediately to prevent the unsafe image from being deployed to the environment. This kind of delayed image scanning, which depends on the scanning function of the image warehouse, can not do this because the scanning after image push and image deployment are carried out at the same time.

  • Waste of resources

Images need to occupy a certain amount of disk space (some images are hundreds of M or even g in size). If there are a large number of vulnerable images in the image warehouse, it will occupy a large amount of disk space and increase the cost of using the warehouse.

3、 Pre image scanning

Based on the lack of scanning lag, the front-end of image scanning is particularly important. The front-end operation of image scanning (as shown in the green dotted box in the figure below) is as follows:

Are you right about container image security| IDCF

After the image is built and before the image is pushed, once an image scanning is finished, if the result is safe, then the subsequent operation (pushing to the image warehouse and image deployment) can be carried out smoothly; If the scan result is unsafe, terminate the CI / CD process immediately and notify the corresponding personnel. This prevents unsafe images from being pushed to the image warehouse. It ensures that the images stored in the image warehouse are safe and save disk space. At the same time, the operation space of pre scan on the image will become larger, such as black and white list, RBAC (role based access control) permission control and so on.

For the front-end image scanning operation, we need to use some existing image scanning tools to help us complete this operation. These tools include Clair, who is well known to all, and also some upstarts, such as anchor, trivy, etc. there will be a special article to analyze the specific use and comparison of these tools in the later stage.

Image scanning is just to help us find and solve the image problems, which is only a treatment for the image problems. However, for the image problems, it is better to prevent the problems, that is, when building the image, we should build some images as safe as possible according to some best practices, These are the best practices of image security that we will talk about next. In this way, prevention first, prevention combined (prevention: according to the best practice to build a safe image; To ensure the security of the image, we use the method of pre image scanning embedded in CI / CD.

4、 Container image security best practices

4.1 try to choose a lightweight basic image

Each image has a basic image, which is the image specified in from in dockerfile. Generally, the basic image is a Linux distribution, such as alpine, Ubuntu, etc. When selecting the basic image for a new project, we should consider whether a full operating system (including various libraries) is needed for the operation of the project. If Alpine can meet the requirements, we should not choose Ubuntu, a relatively large operating system, as the basic image of the new project. In addition to the difference in image size, different operating systems of different magnitudes are more important. The more file systems the operating system contains, the larger the attack surface and the more vulnerabilities it contains.

With anchor openjdk:8-jdk-alpine and openjdk:8-jdk Scan, the results show that openjdk:8-jdk There are 128 vulnerabilities, and openjdk:8-jdk-alpine Only 48. and openjdk:8-jdk-alpine The image size of is only openjdk:8-jdk Less than a third of the total.

Are you right about container image security| IDCF

4.2 add a non root user to run the container with the smallest authorized user

The container shares the kernel with the host. If the container is started as root user, it means that the container has root authority to operate the host. This will increase the attack area of the host and the potential threat coefficient. Therefore, you should specify a specific user who is not root, and then use this specific user to start the container. If the image is constructed by dockerfile, you can add the following code in dockerfile to add a specific user in a specific group and start the container with this specific user:

FROM your_base_image
RUN groupadd -r  devsecops && useradd -r  -g devops devsecops
USER devsecops
......(your other dockerfile steps)

4.3 don’t expose sensitive information in the image

Do not store sensitive information such as token, password and SSH key in the image. Once the image is pushed to the public image warehouse, the above sensitive information will be leaked. The result will be disastrous. If you need to use some sensitive information in the image, you can use the secret function provided by docker, or through multi-stage construction. Specific use can refer to the official website of docker.

4.4 do not install unnecessary packages

Many images use “apt get update & & apt get install XXX” or “Yum update & & Yum install XXX” to install software packages, but remember: only install the packages related to the running of the application, not the unnecessary packages. For example, install VIM in the database image. The more packages are installed, the higher the complexity and dependence of the image, which not only leads to the larger volume of the container image, but also makes the attack surface of the image larger and the security lower.

When ubuntu:16.04 When using the “apt get update & apt get install VIM telnet curl – Y” command as the basic image, the number of vulnerabilities and the size of the image are not the same. The detailed comparison is as follows.

Are you right about container image security| IDCF

4.5 scan dockerfile

You can scan dockerfile with dockerfile scanning tools (such as hadolint) to find out the nonstandard writing or some build commands that may cause security problems in dockerfile. For example, use hadolint to scan the following dockerfile:

FROM alpine
RUN apk add busybox-extras

You can see the following output results:

/dev/stdin:1 DL3006 Always tag the version of an image explicitly
/dev/stdin:2 DL3018 Pin versions in apk add. Instead of `apk add <package>` use `apk add <package>=<version>`
/dev/stdin:2 DL3019 Use the `--no-cache` switch to avoid the need to use `--update` and remove `/var/cache/apk/*` when done installing packages

The results show that tag should be specified in the basic image (dl3006 prompt), and the version of busybox extras should be specified in the package installation (APK add) (dl3018 prompt).

4.6 do not select the image with unknown source and no maintenance

When selecting the basic image, don’t select the image whose source is unknown (I don’t know the author of the image, I don’t know the contents of the dockerfile, I don’t know the contents of the basic image), stop maintenance (the last update was a few months or even years ago, how many problems accumulated during this period, God knows). This is because it is not possible to determine whether the content contained in the image is secure. Let’s not just use the number of times the image is pulled and the number of stars as indicators to decide whether to use the image, because this number of times can be brushed up. Be sure to select official images with clear dockerfile content and frequent updates as far as possible. If there is no qualified image, you can try to make your own image.

The above points are only part of the means to build a secure and standardized image. This paper only lists some important and common ones. Other means such as using the image signature to prevent the image from being tampered, building a trusted image warehouse, and using mechanisms such as RBAC and blacklist to control the pull and push of the image to further ensure the image security. Limited to space, this paper no longer further in-depth discussion, interested, can be in-depth study.

5、 Summary

There is no absolute security, there is no security once and for all, only the never-ending action. Image scanning is only a tool to help us find security problems, and then to solve the problem purposefully, which belongs to a treatment method; Therefore, we should form a good habit: from the perspective of safety, combined with the above best practices, to build a safe and standardized mirror. In order to further ensure the security of the image, the way of “prevention first, prevention combined” is adopted.

Source: CIO talk

Author: Ma Jinghe

Ma Jinghe, who is a little horse elder brother, has done LTE 4G network protocol development, and then turned to DevOps. She preached for Cloud Native DevSecOps, love to study docker, kubernetes, istio and other Cloud Native related technologies, and was willing to share, running the official account of the istio. Devops workshop was held in Dalian Devops community activities.

reference material

Are you right about container image security| IDCF

At 8 p.m. every Thursday in May, there is a special quality and test show. The official account can be obtained by “quality”.

  • 0506 how to maximize software testing effectiveness by Zhu Shaomin
  • 0513 data driven testing by Chen Qi
  • 0520 Chen Ji “yes, going to QA is the most effective way to improve quality!”
  • 0527 “continuous test of Devops practice” by Shi Huibin