clair-cicd

Clair, released by CoreOS in Nov '16, is a very effective tool for statically analyzing docker images to determine which known vulnerabilities exist in the images. Integrating Clair into a CI/CD pipeline:

1. is complex (believe this is partly a documentation challenge)
2. creates performance problems (building the Clair required Postgres database of vulnerabilities is slow)
3. in and of itself is insufficient from a risk assessment point of view because once vulnerabilities are identified there's a lack of prescriptive guidance on how to act on the identified vulnerabilities

This repo was created to address the above challenges.

Background

The roots of this repo center around the following beliefs:

• when inserted into a CI/CD pipeline, Clair can be a very effective foundation for automating the risk assessment of Docker image vulnerabilities
• services should be run in Docker containers and thus CI/CD pipelines should be focused on the automated generation, assessment and ultimately deployment of Docker images
• understanding and assessing the risk profile of services is important ie. security is important
• Docker images should not be pushed to a Docker registry until their risk profile is understood (this is an important one)
• CI/CD pipelines should to be fast. how fast? ideally < 5 minutes between code commit and automated deployment begins
• there should be a clear division of responsibilities between those who create a docker image (service engineer) and those who determine the risk of vulnerabilities in a docker image (security analyst)
• the risk assessment process should generate evidence which can be used to understand the risk assessment decision

Key Participants

• service engineer - responsible for implementing a service that is packaged in a docker image
• security analyst - responsible for defining whitelists which are consumed by clair_cicd to influence Docker image risk assessment decisions

How to Use

Getting Started

To start using clair-cicd, a service engineer inserts a single line of code into a service's CI pipeline. The single line of code runs the shell script assess-image-risk.sh. Part of the CI pipeline's responsibility is to build the docker image and then push that docker image to a docker registry. The single line of clair-cicd code should appear after the docker image is built and tested but before the docker image is pushed to a docker registry.

In this simple case, assess-image-risk.sh returns a zero exit status if the docker image contains no known vulnerabilities above a medium severity. If the docker image contains any known vulnerabilities with a severity higher than medium, assess-image-risk.sh returns a non-zero exit status and the build fails ie. the build should fail before the docker image is pushed to a docker registry.

The example illustrates what's described about for the alpine:3.4 docker image.

~> curl -s -L \
  https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
  bash -s -- alpine:3.4
~> echo $?
0
~>

Understanding the Risk Assessment Decision

To understand how assess-image-risk.sh is making its risk assessment decision try using the -v flag.

~> curl -s -L \
  https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
  bash -s -- -v alpine:3.4
2020-01-12 16:43:35 pulling clair database image 'simonsdave/clair-cicd-database:latest'
2020-01-12 16:44:17 successfully pulled clair database image
2020-01-12 16:44:17 starting clair database container 'clair-db-c1dbb5f93ae98755'
2020-01-12 16:44:23 waiting for database server in container 'clair-db-c1dbb5f93ae98755' to start ...........................
2020-01-12 16:44:54 successfully started clair database container
2020-01-12 16:44:54 clair configuration in '/var/folders/7x/rr443kj575s8zz54jrbrp4jc0000gn/T/tmp.ElAlhGNl'
2020-01-12 16:44:59 pulling clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:45:13 successfully pulled clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:45:13 starting clair container 'clair-e9573ae537134fa0'
2020-01-12 16:45:15 successfully started clair container 'clair-e9573ae537134fa0'
2020-01-12 16:45:15 saving docker image 'alpine:3.4' to '/tmp/tmp.IaNHCH'
2020-01-12 16:45:16 successfully saved docker image 'alpine:3.4'
2020-01-12 16:45:16 starting to create clair layers
2020-01-12 16:45:16 creating clair layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:45:16 successfully created clair layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:45:16 done creating clair layers
2020-01-12 16:45:16 starting to get vulnerabilities for clair layers
2020-01-12 16:45:16 saving vulnerabilities to directory '/tmp/tmp.MDncHN'
2020-01-12 16:45:16 getting vulnerabilities for layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:45:16 successfully got vulnerabilities for layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:45:16 done getting vulnerabilities for clair layers
2020-01-12 21:45:17 INFO io:89 Looking for vulnerabilities in directory '/tmp/tmp.MDncHN'
2020-01-12 21:45:17 INFO io:95 Found 1 files with vulnerabilities in directory '/tmp/tmp.MDncHN'
2020-01-12 21:45:17 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.MDncHN/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'
2020-01-12 21:45:17 INFO io:122 Found 0 vulnerabilities in '/tmp/tmp.MDncHN/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'
2020-01-12 21:45:17 INFO io:133 Found 0 vulnerabilities in 1 files in directory '/tmp/tmp.MDncHN'
2020-01-12 21:45:17 INFO assessor:19 Assessment starts
2020-01-12 21:45:17 INFO assessor:26 Assessment ends - pass
~> echo $?
0
~>

Adding a Vulnerability Whitelist

In the above examples a default vulnerability whitelist was used. When specified as a JSON doc, this whitelist would be:

{
  "ignoreSevertiesAtOrBelow": "medium"
}

By default, assess-image-risk.sh returns a non-zero exit status if any vulnerabilities are identified in the image with a severity higher than medium. The medium is derived from the default vulnerability whitelist.

The example below illustrate how to specify a vulnerability whitelist and with a severity other than medium. Note the use of the json:// prefix to indicate this is an inline whitelist.

~> curl -s -L \
  https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
  bash -s -- -v --whitelist 'json://{"ignoreSevertiesAtOrBelow": "negligible"}' ubuntu:18.04
2020-01-12 16:46:56 pulling clair database image 'simonsdave/clair-cicd-database:latest'
2020-01-12 16:46:58 successfully pulled clair database image
2020-01-12 16:46:58 starting clair database container 'clair-db-3b0811925f7e8bc2'
2020-01-12 16:46:59 waiting for database server in container 'clair-db-3b0811925f7e8bc2' to start .............................
2020-01-12 16:47:32 successfully started clair database container
2020-01-12 16:47:32 clair configuration in '/var/folders/7x/rr443kj575s8zz54jrbrp4jc0000gn/T/tmp.BXCs3Giy'
2020-01-12 16:47:34 pulling clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:47:36 successfully pulled clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:47:36 starting clair container 'clair-fc579c71e7daba57'
2020-01-12 16:47:38 successfully started clair container 'clair-fc579c71e7daba57'
2020-01-12 16:47:38 saving docker image 'ubuntu:18.04' to '/tmp/tmp.lPDhNd'
2020-01-12 16:47:43 successfully saved docker image 'ubuntu:18.04'
2020-01-12 16:47:43 starting to create clair layers
2020-01-12 16:47:43 creating clair layer 'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'
2020-01-12 16:47:43 successfully created clair layer 'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'
2020-01-12 16:47:43 creating clair layer '27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'
2020-01-12 16:47:44 successfully created clair layer '27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'
2020-01-12 16:47:44 creating clair layer 'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'
2020-01-12 16:47:44 successfully created clair layer 'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'
2020-01-12 16:47:44 creating clair layer '1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'
2020-01-12 16:47:44 successfully created clair layer '1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'
2020-01-12 16:47:44 done creating clair layers
2020-01-12 16:47:44 starting to get vulnerabilities for clair layers
2020-01-12 16:47:44 saving vulnerabilities to directory '/tmp/tmp.dkfgmI'
2020-01-12 16:47:44 getting vulnerabilities for layer 'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'
2020-01-12 16:47:44 successfully got vulnerabilities for layer 'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'
2020-01-12 16:47:44 getting vulnerabilities for layer '27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'
2020-01-12 16:47:44 successfully got vulnerabilities for layer '27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'
2020-01-12 16:47:44 getting vulnerabilities for layer 'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'
2020-01-12 16:47:44 successfully got vulnerabilities for layer 'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'
2020-01-12 16:47:44 getting vulnerabilities for layer '1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'
2020-01-12 16:47:44 successfully got vulnerabilities for layer '1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'
2020-01-12 16:47:44 done getting vulnerabilities for clair layers
2020-01-12 21:47:45 INFO io:89 Looking for vulnerabilities in directory '/tmp/tmp.dkfgmI'
2020-01-12 21:47:45 INFO io:95 Found 4 files with vulnerabilities in directory '/tmp/tmp.dkfgmI'
2020-01-12 21:47:45 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.dkfgmI/27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54.json'
2020-01-12 21:47:45 INFO io:122 Found 33 vulnerabilities in '/tmp/tmp.dkfgmI/27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54.json'
2020-01-12 21:47:45 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.dkfgmI/cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd.json'
2020-01-12 21:47:45 INFO io:122 Found 33 vulnerabilities in '/tmp/tmp.dkfgmI/cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd.json'
2020-01-12 21:47:45 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.dkfgmI/1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d.json'
2020-01-12 21:47:45 INFO io:122 Found 33 vulnerabilities in '/tmp/tmp.dkfgmI/1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d.json'
2020-01-12 21:47:45 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.dkfgmI/d80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e.json'
2020-01-12 21:47:45 INFO io:122 Found 33 vulnerabilities in '/tmp/tmp.dkfgmI/d80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e.json'
2020-01-12 21:47:45 INFO io:133 Found 33 vulnerabilities in 4 files in directory '/tmp/tmp.dkfgmI'
2020-01-12 21:47:45 INFO assessor:19 Assessment starts
2020-01-12 21:47:45 INFO assessor:34 Assessing vulnerability CVE-2018-11236 - start
2020-01-12 21:47:45 INFO assessor:52 Vulnerability CVE-2018-11236 @ severity medium greater than whitelist severity @ negligible - fail
2020-01-12 21:47:45 INFO assessor:36 Assessing vulnerability CVE-2018-11236 - finish
2020-01-12 21:47:45 INFO assessor:23 Assessment ends - fail
~> echo $?
1
~>

The above is an example of an inline whitelist. It's also possible to specify a whitelist in a file. The example below illustrates the usage. Note use of the file:// prefix to indicate the whitelist is contained in a file.

~> cat whitelist.json
{
  "ignoreSevertiesAtOrBelow": "medium"
}
~> curl -s -L \
  https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
  bash -s -- -v --whitelist file://whitelist.json alpine:3.4
2020-01-12 16:48:41 pulling clair database image 'simonsdave/clair-cicd-database:latest'
2020-01-12 16:48:42 successfully pulled clair database image
2020-01-12 16:48:42 starting clair database container 'clair-db-191152e37b864e4b'
2020-01-12 16:48:43 waiting for database server in container 'clair-db-191152e37b864e4b' to start .............................
2020-01-12 16:49:16 successfully started clair database container
2020-01-12 16:49:16 clair configuration in '/var/folders/7x/rr443kj575s8zz54jrbrp4jc0000gn/T/tmp.GdlBNmiG'
2020-01-12 16:49:19 pulling clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:49:20 successfully pulled clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:49:20 starting clair container 'clair-747d1c50606fba7e'
2020-01-12 16:49:21 successfully started clair container 'clair-747d1c50606fba7e'
2020-01-12 16:49:22 saving docker image 'alpine:3.4' to '/tmp/tmp.Eldkbe'
2020-01-12 16:49:23 successfully saved docker image 'alpine:3.4'
2020-01-12 16:49:23 starting to create clair layers
2020-01-12 16:49:23 creating clair layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:49:23 successfully created clair layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:49:23 done creating clair layers
2020-01-12 16:49:23 starting to get vulnerabilities for clair layers
2020-01-12 16:49:23 saving vulnerabilities to directory '/tmp/tmp.pCOhlL'
2020-01-12 16:49:23 getting vulnerabilities for layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:49:23 successfully got vulnerabilities for layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:49:23 done getting vulnerabilities for clair layers
2020-01-12 21:49:23 INFO io:89 Looking for vulnerabilities in directory '/tmp/tmp.pCOhlL'
2020-01-12 21:49:23 INFO io:95 Found 1 files with vulnerabilities in directory '/tmp/tmp.pCOhlL'
2020-01-12 21:49:23 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.pCOhlL/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'
2020-01-12 21:49:23 INFO io:122 Found 0 vulnerabilities in '/tmp/tmp.pCOhlL/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'
2020-01-12 21:49:23 INFO io:133 Found 0 vulnerabilities in 1 files in directory '/tmp/tmp.pCOhlL'
2020-01-12 21:49:23 INFO assessor:19 Assessment starts
2020-01-12 21:49:23 INFO assessor:26 Assessment ends - pass
~> echo $?
0
~>

Specific vulnerabilities can also be whitelisted. The example below illustrates this capability. If you add the -v (verbose) flag to assess-image-risk.sh you see exactly how whitelisted vulnerabilities impact the risk assessment with statements like Vulnerability CVE-2019-13627 in whitelist - pass

~> curl -s -L \
  https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
  bash -s -- --whitelist 'json://{"ignoreSevertiesAtOrBelow":"low"}' ubuntu:18.04
~> echo $?
1
~> cat whitelist.json
{
  "ignoreSevertiesAtOrBelow": "low",
  "vulnerabilities": [
    { "cveId": "CVE-2018-20839", "rationale": "reason #1" },
    { "cveId": "CVE-2019-5188", "rationale": "reason #2" },
    { "cveId": "CVE-2018-11236", "rationale": "reason #3" },
    { "cveId": "CVE-2019-13627", "rationale": "reason #4" },
    { "cveId": "CVE-2019-13050", "rationale": "reason #5" },
    { "cveId": "CVE-2018-11237", "rationale": "reason #6" },
    { "cveId": "CVE-2018-19591", "rationale": "reason #7" }
  ]
}
~> curl -s -L \
  https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
  bash -s -- --whitelist 'file://whitelist.json' ubuntu:18.04
~> echo $?
0
~>

ITO whitelists it's worth seeing issue #7 which says:

When specifying the whitelist for assess-image-risk.sh with the --whitelist command line argument, should support https:// scheme in addition to the existing json://, file:// schemes. Why is this important? Ideally whitelists should be maintained by a security analyst not a service engineer. This means that whitelists should be maintained in another repo with appropriate change management process. The json:// and file:// schemes are fine for maintaining whitelists in the same repo is service code. However, it would be better to maintain whitelists in a repo that is readonly for service engineers and editable only by security analysts who could apply appropriate change management processes are used to make changes (code reviews, feature branches, etc).

How it Works + Requirements/Assumptions

There are 3 moving pieces:

1. assess-image-risk.sh is bash script which does the heavy lifting to co-ordinate the interaction of the 2 other moving pieces
2. Clair's vulnerability database which is packaged inside the docker image simonsdave/clair-database - a CircleCI cron job is used to rebuild simonsdave/clair-database 3 days per week to ensure the vulnerability database is current
3. a set of Python and Bash risk assessment scripts packaged in the simonsdave/clair-cicd-clair docker image which is based on the docker image quay.io/coreos/clair which packages up Clair

From the samples at the start of this doc you'll see the approach of curl'ing the latest release of assess-image-risk.sh into a localy run bash shell. assess-image-risk.sh then spins up a container using the simonsdave/clair-database. Another container is then run using simonsdave/clair-cicd-clair with the simonsdave/clair-cicd-clair container being able to talk with the simonsdave/clair-database container. Once the simonsdave/clair-cicd-clair container is running, assess-image-risk.sh docker exec's this bash script which does the actual risk assessment.

Armed with the understanding of how clair-cicd works you'll appreciate that the ability to execute assess-image-risk.sh is what defines the requirements for the execution environment. assess-image-risk.sh is a bash script used to launch the risk assessment process and as such it's this script which defines the bulk of the assumptions/requirements for clair-cicd - the script uses docker, sed and openssl so all these need to be available in the environment running clair-cicd

References

• 12 May '22 - Identify known Vulnerabilities in Docker Image using Clair
• 1 Nov '20 - A Practical Introduction to Container Security
• 5 Aug '20 - How to do vulnerability management with Docker and CI/CD
• 20 May '20 - Security scanners for Python and Docker: from code to dependencies
• 29 Jan '20 - What Is DevSecOps, and How Is It Different from DevOps?
• 26 Dec '19 - CoreOS Clair — Part 2: Installation & Integration
• 4 Oct '19 - CoreOS Clair — Part 1: Container Image Scanning
• 1 Oct '18 - Baking Compliance in your CI/CD Pipeline
• 11 Sep '18 - What is DevSecOps?
• 20 Jun '18 - Where in the DevOps cycle do you do security?
• 8 May '18 - DevSecOps: 7 habits of strong security organizations
• 18 Apr '18 - The Cloudcast #343 - Container Vulnerability Scanning
• 21 Feb '18 - Automated Compliance Testing Tool Accelerates DevSecOps
• 20 Feb '18 - 6 Requirements for Achieving DevSecOps
• 22 Jan '18 - DevOps and Security: How to Overcome Cultural Challenges and Transform to True DevSecOps
• 15 Jan '18 - Why DevSecOps matters to IT leaders
• 27 Nov '17 - What is vulnerability management? Processes and software for prioritizing threats
• 23 Oct '17 - The Ten Cybersecurity Commandments
• 9 Oct '17 - 10 layers of Linux container security
• 5 Oct '17 - How to Maintain Security when Rolling out DevOps
• 26 Jan '17 - DevOps and Separation of Duties
• 26 Jul '16 - Injecting security into Continuous Delivery
• 5 Jun '16 - <— Shifting Security to the Left
• Five Secrets and Two Common “Gotchas” of Vulnerability Scanning