From 122ef1fd4c92e8a12b616877eba58d5171c9d392 Mon Sep 17 00:00:00 2001 From: "A.J. Stein" Date: Tue, 10 Sep 2024 22:17:37 -0400 Subject: [PATCH] Test GHCR integration with "in upstream" branch instead of fork with write perms dropping (#665) * Add initial OCI spec for container for #655 * Now add FR constraint files for #655 * Woops, fix typo in clone path for fd_data_dl scratch container * Constraints in /opt/fedramp sudir, make it WORKDIR * Switch to Alpine Maven scratch image not Debian * Switch to Node for final image, install oscaljs * Add checkout data to final image * Fix missed parameterization of git image * Add non-default OCI image build target for make * Verify GPG signaure of oscal-cli build * Add clean target for OCI image builds * Allow for TLS bypass and proxy in Makefile Disable cert-checking for the local version that is built on laptops for GSA staff who make use of a VPN/proxy solution that intercept all TLS communication for security monitoring. This includes not just Docker, but also the containers as they build an image. Since production images will be made in GitHub Actions without the Makefile, these directives will be ignored. * Do not do slow git clone, use local COPY instead For speed, ease of access, and leave commit metadata from the container ID linked to the commit hash itself, just copy from the outside context of the image build. * Add publish target to Makefile with useful tags Also try docker push to GHCR to start before moving on the "in pipeline" build with GitHub Actions. * Fix repeat docker commands for correct tag-n-push * Correct the org.opencontainers.image.source label * Actions: perms for writing packages (ghcr.io) * Actions: follow GH tutorial, more perms added * Actions: build, sign, push, attest and OCI image This workflow change is the first attempt at building, pushing, and signing the validation-tools image to push to the ghcr.io registry. * Actions: ref_name for image tags problematic For both PRs and non-PR branches, that seems to cause problems for tags that we ought to avoid for now. * Actions: use action correctly, no manual labels * Actions: remove metadata from Dockerfile, use GHA * Actions: woops, forgot explicit checkout path Our GHA CI/CD checks out to `./git-content`, `.` by default so the action directive looking for context did not find the Dockerfile. * Actions: check if least privilege perms block push See more details in this reply and the larger context from others who cannot push a built container to ghcr.io. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: scratch that, `write-all` blocked by org The github.com/GSA organization still blocks the write to an org-level package in very permissive move. Tips from the discussions posts did not help here. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: add metadata action SHA options We need to force SHA1 long (not seven-digit short version to avoid collisions), remove both `sha-` prefix and remove suffix explicitly. * Actions, sigh, really remove `sha256` prefix again It seems that didn't stick the last time, so I will try this config again and follow the official custom hash label strategy from the action example from the official README. * Support MVP platforms, arm64 and amd64 If not we will only support modern Apple computers with modern M1 chips, not Intel environments for PC and older Macs. We need broad support for these top platforms. * Explicit platform option for buildx too for #656 It seems this may be needed because I still get similar but different warnings on multi-platform docker builds when using on macOS on an Apple laptop with a M1 processor and amd64 processor for personal computers with Windows and Linux operating systems respectively. > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v3) and no specific platform was requested * Pin metadata action and update configs for #656 - Had a slightly wrong version of docker/metadata-action that could not use annotations properly, hence no annotations on image. - Use annotations instead of custom override labels with that action. - Update docker/build-push-registry action to retrieve those labels as well. - Change subject name for attestation to end with `-attestation` suffix to make the GHCR registry entries less confusing. * Woops, attestation subject === image name for #656 I re-read the dogs. Attestations will be uploaded to Sigstore but I will not busy up the registry with them every moment as it will make it even more confusing for novice users and advanced developers what data they are looking for by content-addressable git commit hash ID. * Explanatory comments on Dockerfile lint for #656 For future analysis or assessment, I am leaving information in the Dockerfile as comments to address warning output in docker build and push flagging a potential finding re secrets based on variable names. ```sh 4 warnings found (use docker --debug to expand): - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "OSCAL_CLI_GPG_KEY") (line 20) - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TEMURIN_APK_KEY_URL") (line 45) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 43) ``` They are IDs to secrets, not actually secrets, now I have documented it. * Attestations need explicit reg push off for #656 Just removing it may not have done the trick. --- .dockerignore | 3 ++ .github/workflows/content-artifacts.yml | 65 ++++++++++++++++++++++++ Dockerfile | 67 +++++++++++++++++++++++++ Makefile | 31 ++++++++++++ 4 files changed, 166 insertions(+) create mode 100644 .dockerignore create mode 100644 Dockerfile diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 000000000..7c974cc14 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,3 @@ +# Ignore everything +* +!/src/validations/constraints \ No newline at end of file diff --git a/.github/workflows/content-artifacts.yml b/.github/workflows/content-artifacts.yml index e423225ab..6dd0f0b30 100644 --- a/.github/workflows/content-artifacts.yml +++ b/.github/workflows/content-artifacts.yml @@ -15,10 +15,19 @@ on: name: Process Content env: HOME_REPO: GSA/fedramp-automation + IMAGE_NAME: GSA/fedramp-automation/validation-tools + REGISTRY: ghcr.io + # Docs: github.com/docker/metadata-action/?tab=readme-ov-file#typesha + DOCKER_METADATA_PR_HEAD_SHA: true jobs: validate-and-publish-content: name: Content Validation Checking, Conversion and Validation runs-on: ubuntu-20.04 + permissions: + contents: read + packages: write + attestations: write + id-token: write steps: - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 with: @@ -53,3 +62,59 @@ jobs: commit_user_name: OSCAL GitHub Actions Bot commit_user_email: oscal@nist.gov commit_author: OSCAL GitHub Actions Bot + - name: Container image QEMU setup for cross-arch builds + id: image_setup_qemu + uses: docker/setup-qemu-action@49b3bc8e6bdd4a60e6116a5414239cba5943d3cf + - name: Container image buildx setup for cross-arch builds + id: image_setup_buildx + with: + platforms: linux/amd64,linux/arm64 + uses: docker/setup-buildx-action@988b5a0280414f521da01fcc63a27aeeb4b104db + - name: Container image login + id: image_login + uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - name: Container image metadata and tag generation + id: image_metadata + uses: docker/metadata-action@8e5442c4ef9f78752691e2d8f8d19755c6f78e81 + with: + images: + ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} + tags: | + type=sha,prefix=,suffix=,format=long + type=ref,event=branch + type=ref,event=tag + type=ref,event=pr + # For now, do not auto-tag latest, maintainers will decided what is + # release-worthy. + flavor: | + latest=false + annotations: + maintainers="FedRAMP Automation Team " + org.opencontainers.image.authors="FedRAMP Automation Team " + org.opencontainers.image.documentation="https://automate.fedramp.gov" + org.opencontainers.image.source="https://github.com/GSA/fedramp-automation" + org.opencontainers.image.vendor="GSA Technology Transformation Services" + org.opencontainers.image.title="FedRAMP Validation Tools" + org.opencontainers.image.description="FedRAMP's tools for validating OSCAL data" + org.opencontainers.image.licenses="CC0-1.0" + - name: Container image registry push + id: image_registry_push + uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4 + with: + context: git-content + push: true + tags: ${{ steps.image_metadata.outputs.tags }} + labels: ${{ steps.image_metadata.outputs.annotations }} + platforms: linux/amd64,linux/arm64 + cache-from: type=gha + cache-to: type=gha,mode=max + - name: Container image push attestations + uses: actions/attest-build-provenance@1c608d11d69870c2092266b3f9a6f3abbf17002c + with: + subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}} + subject-digest: ${{ steps.image_registry_push.outputs.digest }} + push-to-registry: false diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 000000000..67d8e1b49 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,67 @@ +ARG MAVEN_IMAGE=maven:3.9.9-eclipse-temurin-22-alpine +ARG NODE_IMAGE=node:22-alpine3.20 +ARG APK_EXTRA_ARGS +ARG WGET_EXTRA_ARGS +# Static analysis from docker build and push warns this is a secret, it is not. +# Per official developer instructions, it is necessary to verify the APK packages +# for Alpine or properly signed. This information is inherently public. +# https://adoptium.net/installation/linux/#_alpine_linux_instructions +ARG TEMURIN_APK_KEY_URL=https://packages.adoptium.net/artifactory/api/security/keypair/public/repositories/apk +ARG TEMURIN_APK_REPO_URL=https://packages.adoptium.net/artifactory/apk/alpine/main +ARG TEMURIN_APK_VERSION=temurin-22-jdk +ARG MAVEN_DEP_PLUGIN_VERSION=3.8.0 +ARG OSCAL_CLI_VERSION=2.0.2 +# Current public key ID for maintainers@metaschema.dev releases of oscal-cli +# Static analysis from docker build and push warns this is a secret, it is not +# and is necessary to cross-ref the Maven GPG key for checking build signatures. +# https://keyserver.ubuntu.com/pks/lookup?search=0127D75951997E00&fingerprint=on&op=index +ARG OSCAL_CLI_GPG_KEY=0127D75951997E00 +ARG OSCAL_JS_VERSION=1.4.4 +ARG FEDRAMP_AUTO_GIT_URL=https://github.com/GSA/fedramp-automation.git +ARG FEDRAMP_AUTO_GIT_REF=feature/external-constraints +ARG FEDRAMP_AUTO_GIT_COMMIT + +FROM ${MAVEN_IMAGE} as oscal_cli_downloader +ARG MAVEN_DEP_PLUGIN_VERSION +ARG OSCAL_CLI_VERSION +ARG OSCAL_CLI_GPG_KEY +ARG APK_EXTRA_ARGS +ARG WGET_EXTRA_ARGS +RUN apk add --no-cache gpg gpg-agent unzip && \ + mkdir -p /opt/oscal-cli && \ + mvn \ + org.apache.maven.plugins:maven-dependency-plugin:${MAVEN_DEP_PLUGIN_VERSION}:copy \ + -DoutputDirectory=/opt/oscal-cli \ + -DremoteRepositories=https://repo1.maven.org/maven2 \ + -Dartifact=dev.metaschema.oscal:oscal-cli-enhanced:${OSCAL_CLI_VERSION}:zip:oscal-cli && \ + mvn \ + org.apache.maven.plugins:maven-dependency-plugin:${MAVEN_DEP_PLUGIN_VERSION}:copy \ + -DoutputDirectory=/opt/oscal-cli \ + -DremoteRepositories=https://repo1.maven.org/maven2 \ + -Dartifact=dev.metaschema.oscal:oscal-cli-enhanced:${OSCAL_CLI_VERSION}:zip.asc:oscal-cli && \ + gpg --recv-keys ${OSCAL_CLI_GPG_KEY} && \ + gpg -k ${OSCAL_CLI_GPG_KEY} && \ + cd /opt/oscal-cli && \ + gpg --verify *.zip.asc && \ + unzip *.zip && \ + rm -f *.zip && \ + rm -f *.zip.asc + +FROM ${NODE_IMAGE} as final +ARG OSCAL_JS_VERSION +ARG TEMURIN_APK_KEY_URL +ARG TEMURIN_APK_REPO_URL +ARG TEMURIN_APK_VERSION +ARG APK_EXTRA_ARGS +ARG WGET_EXTRA_ARGS +COPY --from=oscal_cli_downloader /opt/oscal-cli /opt/oscal-cli +RUN wget ${WGET_EXTRA_ARGS} -O /etc/apk/keys/adoptium.rsa.pub "${TEMURIN_APK_KEY_URL}" && \ + echo "${TEMURIN_APK_REPO_URL}" >> /etc/apk/repositories && \ + apk add ${APK_EXTRA_ARGS} --no-cache ${TEMURIN_APK_VERSION} && \ + mkdir -p /opt/fedramp/oscaljs && \ + mkdir -p /opt/fedramp/constraints && \ + (cd /opt/fedramp/oscaljs && npm install oscal@${OSCAL_JS_VERSION}) +COPY ./src/validations/constraints/*.xml /opt/fedramp/constraints/ +ENV PATH="$PATH:/opt/oscal-cli/bin:/opt/fedramp/oscaljs/node_modules/.bin" +WORKDIR /opt/fedramp/constraints +ENTRYPOINT [ "/opt/oscal-cli/bin/oscal-cli" ] diff --git a/Makefile b/Makefile index 77fe3c8a2..7b8f5de32 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,5 @@ export BASE_DIR=$(shell pwd) +OCI_REV_TAG=$(shell git rev-parse HEAD) help: @grep -h -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}' @@ -26,6 +27,12 @@ clean-dist: ## Clean non-RCS-tracked dist files @echo "Cleaning dist..." git clean -xfd dist +clean-oci-image: + docker rmi -f \ + validation-tools:$(OCI_REV_TAG) \ + ghcr.io/gsa/fedramp-automation/validation-tools:$(OCI_REV_TAG) \ + gsatts/validation-tools:$(OCI_REV_TAG) \ + test: build-validations ## Test all build: build-validations build-web dist ## Build all artifacts and copy into dist directory @@ -39,3 +46,27 @@ build: build-validations build-web dist ## Build all artifacts and copy into di @echo '#/bin/bash\necho "Serving FedRAMP ASAP documentation at http://localhost:8000/..."\npython3 -m http.server 8000 --directory web/' > ./dist/serve-documentation chmod +x ./dist/serve-documentation + +build-oci-image: + docker build \ + --build-arg APK_EXTRA_ARGS="--no-check-certificate" \ + --build-arg WGET_EXTRA_ARGS="--no-check-certificate" \ + -t validation-tools:$(OCI_REV_TAG) \ + -t ghcr.io/gsa/fedramp-automation/validation-tools:$(OCI_REV_TAG) \ + -t gsatts/validation-tools:$(OCI_REV_TAG) \ + . + +publish-oci-image: + docker tag \ + validation-tools:$(OCI_REV_TAG) validation-tools:latest + + docker tag \ + ghcr.io/gsa/fedramp-automation/validation-tools:$(OCI_REV_TAG) \ + ghcr.io/gsa/fedramp-automation/validation-tools:latest + + docker tag \ + gsatts/validation-tools:$(OCI_REV_TAG) \ + gsatts/validation-tools:latest + + docker push ghcr.io/gsa/fedramp-automation/validation-tools:$(OCI_REV_TAG) + docker push ghcr.io/gsa/fedramp-automation/validation-tools:latest