From cb75f5935404c4045a9d2217121ff15f8f5f8347 Mon Sep 17 00:00:00 2001 From: Viet Nguyen Duc Date: Tue, 14 May 2024 07:56:47 +0000 Subject: [PATCH] Merging the fork [docker-seleniarm] into docker-selenium (#2257) * add CircleCI status badge * Update README * Adding the code to add all of the browser tags and update the latest tag * 4.1.4 * Bumping grid to 4.1.4 and updating Firefox to v99.0-1 * [deploy] 4.1.4 and Firefox 99.0-1 with browser tags and update latest tag * Updated README and [deploy] grid 4.1.4 and Firefox 99.0 * Attempts to tag multi-arch images by modifying the manifest * Added the functionality to add browser versions to the Docker Hub tags. Also, updated the 'latest' tag. * Allow passing an environment variable GRID_STATUS_MAX_ATTEMPTS to override the default 3 attempts to a differen value. Useful for testing under emulation where things are slower. * Updated Firefox to 100.0.2-1 * [deploy] Firefox 100.0.2-1 and Selenium 4.2.1 * Fixing bug in variable name causing browser tagging with browser version and webdriver and selenium versions to fail. * Fixed merge conflict from earlier * [deploy] 4.2.2 seleniarm images * Automating the process of modifying the selenium grid tag since I keep forgetting to do it when deploying. * run the deploy workflow without deploying * Using instead of backticks * Cleanup config.yml * Debugging why it deployed as 4.1.0... * test [deploy] * test [deploy] and outputting some debug info * test [deploy] and outputting some debug info * test [deploy] is BRANCH a reserved keyword? * test [deploy] is BRANCH a reserved keyword? * More tests [deploy] * debug [deploy] * debug [deploy] * Clean up and [deploy] test * test [deploy] to test account to check tags * Resolved issue with BRANCH environment variable not being set properly so it can be reused from step to step. * [deploy] 4.3.0 * [deploy] 4.3.0 again since it failed * small addition to readme to confirm graviton compatibility * [deploy] 4.3.0 updates * [deploy] 4.4.0 * [deploy] update * update Firefox version * [deploy] 4.4.0 with Firefox 104.0-1 * Update Makefile * Add node and standalone docker build * Fix tabs. Also, replace hard-coded platforms with variable, since we build and test separately but then build and deploy together. * Bump Firefox to 105.0.1-1 * [deploy] Selenium 4.5.0 and Firefox 105.0.1-1 * Fix race condition when launching websockify. Summary of issue #18: noVNC's launch.sh invokes `ps`. Without it, it exits prematurely, thus ruining websockify's stdout and stderr file descriptors. Depending on how fast websockify starts up, this may or may not prevent it from working. * CircleCI: prevent '/' characters in Docker tags. This fixes #21. * [deploy] fixes for #18 and #21 * Removing obsolete and not supported shell scripts. Updating docs with the supported method for building the ARM images locally. * README.md formatting change * Removed retag-latest-images-as-today obsolete script. * Renamed NodeFirefox/Dockerfile.arm64 to NodeFirefox/Dockerfile.multi-arch to reflect the nature of building for multiple architectures, not just arm64. Updated Makefile with new Dockerfile,multi-arch name. Removed obsolete generation and build scripts, outdated experimental Ubuntu Dockerfile, and updated documentation to move how to start containers towards the top. * Make more clear Chrome isn't build for ARM 'on Linux' * Bump geckodriver to 0.32.0. Replace jamesmortensen/geckodriver-arm-binaries with official mozilla/geckodriver repo for arm64. Fixes #22 * [deploy] with upstream changes and official Mozilla geckodriver arm64 v0.32.0 * [deploy] 4.5.2 with Java 11 HTTP client * [deploy] 4.5.3 * [deploy] merge from upstream and deploy 4.6.0 * Modified the telemetry installation commands so that the coursiers launcher pulled the binary based on arch, and only for the supported ones arm64 and x86_64 * [deploy] 4.6.0 * Update Firefox to v107 * [deploy] 4.7.0 * [deploy] 4.7.1 * Publish plain version numbers (e.g.: 4.6.0, 4.6, 4) * [deploy] using the native coursier binaries, since the JAR version takes too long and causes the build to exceed the timeout * [deploy] with jaegar observability jars only for aarch64 since it causes the deployment to timeout for other architectures. * Upgrade to Firefox 108.0-2 for all architectures * Test [deploy]. Add the grid components to the deploy step, and split up building images into different, parallel jobs to reduce build time. * Test [deploy]. Add the grid components to the deploy step, and split up building images into different, parallel jobs to reduce build time. * [deploy] fix job names * test [deploy] debug some broken build processes. * [deploy] Avoid make and run all build commands brom build-and-push.sh * [deploy] 4.7.2 with Firefox 108 * [deploy] testing using an exit code in case a command in the script fails * [deploy] * Change bash to sh * [deploy] test * [deploy] testing to see why building base behaves differently with make vs running via a script * [deploy] re-register binfmt and qemu for each runner * [deploy] testing tagging and release notes job after fixing env var dereferencing * [deploy] testing tagging and release notes job after fixing env var dereferencing * [deploy] try using one of the parameters passed in declaratively instead. * [deploy] try with the job name and also log it for the base job as well * [deploy] using the make targets parameter. * [deploy] using the make targets parameter. * [deploy] test * [deploy] test * test [deploy] Added scripts and logic to add the Major_minor tags to the non-browser images, and added logic to add all of the tags to the release notes. * Changed file permissions so shell script can be executed. * Test [deploy] Remove comment * Test [deploy] Adding NAMESPACE variable to the docker-pull-related-tags script. * Test [deploy] and retag the images with related tags instead of excessive pulling. * Test [deploy]. Rename docker-pull-related-tags to docker-add-related-tags. Clean up the names of the steps in the release notes generation step. Reorder the MAJOR, MAJOR-MINOR, and VERSION tags for the browser images so the order in Docker Hub matches their upstream siblings. Temporarily disable building images to focus on the release notes generation step. * Test [deploy] forgot a tag * Test [deploy] need to set build date to date of prev deployed images in order to test the final job * test [deploy] removed commented test code * Test [deploy] to see the full flow. * test [deploy] after merge from trunk and upstream * Test [deploy] * Test [deploy] * Fix ordering of the grid images * Change everything back to the production 'seleniarm' Docker credentials and namespace. * [deploy] 4.8.0 with upgrade Firefox to 109 and full grid. * If we're getting release notes, pull from docker anonymously * [deploy] with 0.32.1 geckodriver * [deploy] Need to be logged into Docker Hub in order to push tags but not for release notes portion. * [deploy] 4.8.0 with latest noVNC and websockify * Upgrade to Firefox 109 and geckodriver 0.32.2 * [deploy] 4.8.1 and geckodriver 0.32.2 * [deploy] with updates to use buildx imagetools instead of docker manifest to add tags to multi-arch images * [deploy] with 'chomium' typo fix. * [deploy] updates from upstream * Updated multi-arch Firefox to 111.0.1-1. * [deploy] 4.8.2 * [deploy] 4.8.2 with Firefox rollback to v109. * [deploy] 4.8.2 * [deploy] 4.8.3. Updated build-and-push.sh script to use default Selenium version from the Base/Dockerfile so environment variable is only needed if we want the tag version to be something else. * Update Firefox to v112.0-1 * [deploy] 4.9.0 * Adding some delays between building images that are dependent on each other to avoid 'failed to solve' errors * For build errors, we'll retry them 5 times before giving up * Updating the README docs to reflect the state of building the grid components. * [deploy] 4.9.1 * debian:latest is now bookworm, and openjdk-11-jre-headless is missing. So we pin to debian:bullseye * Bump Firefox to 114.0-1 and geckodriver to 0.33.0 * [deploy] Selenium 4.10, Firefox 114,0-1 geckodriver 0.33.0 * Update CircleCI config to deploy grid from trunk * Updated Firefox to 116.0-2 * Downgrade Selenium to 4.10 with updated browsers and drivers * [deploy] updated browsers with Selenium 4.10.0 * Merge from upstream to add Selenium Manager SE_OFFLINE environment variable. Upgrade to Selenium 4.11.0 * update python selenium module to latest. Update Firefox * [deploy] latest browsers for Selenium 4.10.0 * [deploy] 4.10.0 redeploy * Fix build scripts so the RELEASE variable is used. * Fix Makefile - add BASE_RELEASE and BASE_VERSION to the base_multi action * Adding locale to Chromium dockerfile. * Handle usrmerge issues https://wiki.debian.org/UsrMerge * Just install usrmerge * Try chromium without usrmerge. Upgrade Firefox to 120 * Add usrmerge to NodeChromium after setting sources. Split Firefox install into 2 runs for easier debugging * [deploy] test with debian bookworm update - was missing the browser_version and browser_binary_location from upstream. Java package openjdk no longer available in bookworm so replaced with temurin Co-authored-by: Fried Hoeben * [deploy] test * [deploy] test from tmp * [deploy] fixed tests * Fixed tests by waiting for downloadable files to exist in array, fixes Chromium tests. Co-authored-by: Fried Hoeben * Prepare release for 4.16.1 * [deploy] 4.16.1 * test(chart) against build multi Signed-off-by: Viet Nguyen Duc * Sync update from upstream Signed-off-by: Viet Nguyen Duc * Adding script used to build the multi-arch images, but locally using only one architecture. * Fixed issue where xauth not found, preventing xvfb from starting. * CI chart tests against amd64 only Signed-off-by: Viet Nguyen Duc * Skip NodeChromium in Dockerfile scan Signed-off-by: Viet Nguyen Duc * [deploy] v4.17.0 * [build] Update deploy workflow Signed-off-by: Viet Nguyen Duc * [deploy] v4.17.0 * [deploy] v4.17.0 Signed-off-by: Viet Nguyen Duc * [deploy] 4.17.0 - Fix bug in auto-retry logic where if build fails 5 times it needs to also return exit 1 to not let CI think it succeeded. * Sync with upstream * [deploy] 4.18.0 * [deploy] tag image name 4.18.0 * Release v4.18.1 [deploy] Signed-off-by: Viet Nguyen Duc * [deploy] fix broken link to download Firefox v124 Update Dockerfile.multi-arch * update: Debian repo is used in NodeChromium Fixed https://github.com/seleniumhq-community/docker-seleniarm/issues/69 * Temporary disable arch linux/arm/v7 due to NodeFirefox build failed [deploy] Signed-off-by: Viet Nguyen Duc * .circleci fetch selenium version from Makefile Signed-off-by: Viet Nguyen Duc * Add back platform linux/arm/v7 This reverts commit d16ae9ea0a202b528f06a6375f0d1bdfda487fc1. Signed-off-by: Viet Nguyen Duc * Update .circleci [deploy] Signed-off-by: Viet Nguyen Duc --------- Signed-off-by: Viet Nguyen Duc Co-authored-by: James Mortensen Co-authored-by: Peter Triesz Co-authored-by: Xavier G Co-authored-by: DJ Gregor Co-authored-by: Fried Hoeben --- .circleci/config.yml | 405 ++++++++++++++++++++ .github/workflows/build-test-multi-arch.yml | 59 +++ Base/Dockerfile | 7 +- Makefile | 142 +++++++ NodeBase/Dockerfile | 6 - NodeChromium/Dockerfile | 47 +++ NodeChromium/wrap_chromium_binary | 12 + NodeFirefox/Dockerfile.multi-arch | 69 ++++ NodeFirefox/README.md | 28 ++ README.md | 83 ++++ build-and-push.sh | 66 ++++ build-locally.sh | 66 ++++ docker-add-related-tags.sh | 17 + generate_multi-arch-release_notes.sh | 78 ++++ get-access-token.py | 40 ++ get-access-token.sh | 18 + get-all-related-tags.go | 179 +++++++++ get-image-sha256-digest.go | 56 +++ tag-and-push-multi-arch-image.sh | 25 ++ tag_and_push_multi-arch_browser_images.sh | 126 ++++++ tag_and_push_multi-arch_major_minor.sh | 26 ++ tests/test.py | 30 +- 22 files changed, 1562 insertions(+), 23 deletions(-) create mode 100644 .circleci/config.yml create mode 100644 .github/workflows/build-test-multi-arch.yml create mode 100644 NodeChromium/Dockerfile create mode 100755 NodeChromium/wrap_chromium_binary create mode 100644 NodeFirefox/Dockerfile.multi-arch create mode 100644 NodeFirefox/README.md create mode 100644 build-and-push.sh create mode 100755 build-locally.sh create mode 100644 docker-add-related-tags.sh create mode 100755 generate_multi-arch-release_notes.sh create mode 100644 get-access-token.py create mode 100644 get-access-token.sh create mode 100644 get-all-related-tags.go create mode 100644 get-image-sha256-digest.go create mode 100755 tag-and-push-multi-arch-image.sh create mode 100755 tag_and_push_multi-arch_browser_images.sh create mode 100755 tag_and_push_multi-arch_major_minor.sh diff --git a/.circleci/config.yml b/.circleci/config.yml new file mode 100644 index 0000000000..378495a597 --- /dev/null +++ b/.circleci/config.yml @@ -0,0 +1,405 @@ +# .circleci/config.yml +version: 2.1 + +executors: + ubuntu2004arm64: + machine: + image: ubuntu-2004:current + resource_class: arm-medium + ubuntu2004amd64: + machine: + image: ubuntu-2004:current + resource_class: medium + ubuntu2004arm64large: + machine: + image: ubuntu-2004:current + resource_class: arm-large + +jobs: + build-multi-arch: + parameters: + platforms: + type: string + machine-type: + type: executor + executor: << parameters.machine-type >> + environment: + NAMESPACE: seleniarm + BUILD_DATE: today + PLATFORMS: << parameters.platforms >> + steps: + - run: + name: "Prepare workflow environment variables" + command: | + echo 'export BRANCH="${CIRCLE_BRANCH//\//-}"' >> $BASH_ENV + cat $BASH_ENV + source $BASH_ENV + echo "Workflow environment variables:" + echo $BRANCH + - run: uname -a + - run: docker info + - checkout + - run: + name: "Build Docker images" + command: | + echo "Branch is $BRANCH" + NAME=${NAMESPACE} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} PLATFORMS=${PLATFORMS} BUILD_ARGS=${BUILD_ARGS} make build_multi + - run: + name: "Save Docker Images in Cache" + command: | + export VERSION=${CIRCLE_BRANCH//\//-} + echo $NAMESPACE/base:$VERSION-$BUILD_DATE + echo $NAMESPACE/node-base:$VERSION-$BUILD_DATE + echo $NAMESPACE/hub:$VERSION-$BUILD_DATE + echo $NAMESPACE/node-chromium:$VERSION-$BUILD_DATE + echo $NAMESPACE/standalone-chromium:$VERSION-$BUILD_DATE + echo $NAMESPACE/node-firefox:$VERSION-$BUILD_DATE + echo $NAMESPACE/standalone-firefox:$VERSION-$BUILD_DATE + docker save -o multi-arch-images.tar \ + $NAMESPACE/base:$VERSION-$BUILD_DATE \ + $NAMESPACE/node-base:$VERSION-$BUILD_DATE \ + $NAMESPACE/hub:$VERSION-$BUILD_DATE \ + $NAMESPACE/node-chromium:$VERSION-$BUILD_DATE \ + $NAMESPACE/standalone-chromium:$VERSION-$BUILD_DATE \ + $NAMESPACE/node-firefox:$VERSION-$BUILD_DATE \ + $NAMESPACE/standalone-firefox:$VERSION-$BUILD_DATE + - save_cache: + key: multi-arch-images-{{ .Branch }}-{{ .Environment.CIRCLE_WORKFLOW_ID }}-<< parameters.platforms >> + paths: + - multi-arch-images.tar + + test-multi-arch: + parameters: + platforms: + type: string + machine-type: + type: executor + executor: << parameters.machine-type >> + environment: + NAMESPACE: seleniarm + BUILD_DATE: today + steps: + - checkout + - restore_cache: + keys: + - multi-arch-images-{{ .Branch }}-{{ .Environment.CIRCLE_WORKFLOW_ID }}-<< parameters.platforms >> + - run: uname -a + - run: docker info + - run: + name: "Load built images from cache into Docker" + command: | + echo "CIRCLE_WORKFLOW_ID = " $CIRCLE_WORKFLOW_ID + docker load -i multi-arch-images.tar + - run: + name: "Use Python3 and pip3 instead of Python2.7" + command: | + echo "Use Python3 and pip3 instead of Python2.7" + sed -i 's/pip /pip3 /g' tests/bootstrap.sh + sed -i 's/python /python3 /g' tests/bootstrap.sh + sed -i 's/-m pip3 /-m pip /g' tests/bootstrap.sh + - run: + name: "Test Docker images" + no_output_timeout: 2m + command: | + export USE_RANDOM_USER=false + export BRANCH=${CIRCLE_BRANCH//\//-} + USE_RANDOM_USER_ID=${USE_RANDOM_USER} NAMESPACE=${NAMESPACE} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} SKIP_BUILD=true make test_multi_arch + + deploy-multi-arch: + parameters: + platforms: + type: string + build-args: + type: string + machine-type: + type: executor + executor: << parameters.machine-type >> + environment: + NAMESPACE: seleniarm + PLATFORMS: << parameters.platforms >> + BUILD_ARGS: << parameters.build-args >> + DEPLOY_BRANCH: trunk + GITHUB_USER: seleniumhq-community + GITHUB_REPO: docker-seleniarm + steps: + - checkout + - run: + name: "Prepare workflow environment variables" + command: | + export SELENIUM_VERSION=$(grep BASE_VERSION Makefile | sed 's/.*,\([^)]*\))/\1/p' | head -n 1) + export BINDING_VERSION=$(grep BINDING_VERSION Makefile | sed 's/.*,\([^)]*\))/\1/p' | head -n 1) + echo "Prepare workflow environment variables" + echo 'export BRANCH='$SELENIUM_VERSION >> $BASH_ENV + echo 'export BUILD_DATE=$(date '+%Y%m%d')' >> $BASH_ENV + echo 'export RELEASE_TAG="seleniarm-v`echo $BRANCH`-`echo $BUILD_DATE`"' >> $BASH_ENV + source $BASH_ENV + echo "Workflow environment variables:" + echo BRANCH="$BRANCH" + echo BUILD_DATE="$BUILD_DATE" + echo RELEASE_TAG="$RELEASE_TAG" + echo NAMESPACE="$NAMESPACE" + echo PLATFORMS="$PLATFORMS" + echo BUILD_ARGS="$BUILD_ARGS" + echo DEPLOY_BRANCH="$DEPLOY_BRANCH" + echo GITHUB_USER="$GITHUB_USER" + echo GITHUB_REPO="$GITHUB_REPO" + - run: uname -a + - run: docker info + - run: + name: "Check if branch is deployable (contains [deploy] in commit msg on $DEPLOY_BRANCH" + command: | + echo BRANCH="$BRANCH" + echo BUILD_DATE="$BUILD_DATE" + echo RELEASE_TAG="$RELEASE_TAG" + export CI_DEPLOY=`git log --format=oneline -n 1 | grep '\[deploy\]'` && echo "$CI_DEPLOY" + if [ -z "$CI_DEPLOY" ] || [ "$CIRCLE_BRANCH" != "$DEPLOY_BRANCH" ]; then + echo "Cancelling run. Pass [deploy] in commit message on $DEPLOY_BRANCH to deploy" + circleci-agent step halt + else + echo "[deploy] is present in commit message to $DEPLOY_BRANCH. Running workflow to deploy container images" + fi + - run: + name: "Build and Push Docker images" + command: | + echo "Login to Docker, and setup to use a buildx builder and push built multi-arch images" + docker buildx use `docker buildx create` + docker buildx ls + docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD" + echo BRANCH="$BRANCH" + echo BUILD_DATE="$BUILD_DATE" + echo RELEASE_TAG="$RELEASE_TAG" + export CI_DEPLOY=`git log --format=oneline -n 1 | grep '\[deploy\]'` && echo "$CI_DEPLOY" + NAME=${NAMESPACE} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} PLATFORMS=${PLATFORMS} BUILD_ARGS=${BUILD_ARGS} make build_multi + - run: + name: "Tag browser images and update latest tag" + command: | + NAME=${NAMESPACE} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} PUSH_IMAGE=true make tag_and_push_multi_arch_browser_images + NAME=${NAMESPACE} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} make tag_multi_arch_latest + - run: + name: "Generate release notes" + command: | + export LATEST_TAG=$(git describe --tags --abbrev=0) + sh generate_multi-arch-release_notes.sh $LATEST_TAG $CIRCLE_BRANCH $BRANCH $BUILD_DATE + sudo apt-get update -y && sudo apt-get install python3-venv + go install github.com/github-release/github-release@v0.10.0 + cat release_notes.md + export GITHUB_TOKEN=$(sh get-access-token.sh | tail -n 1) + github-release release --tag $RELEASE_TAG --name $RELEASE_TAG --description "`cat release_notes.md`" + + deploy-multi-arch-full-grid: + parameters: + platforms: + type: string + build-args: + type: string + machine-type: + type: executor + circle-job: + type: env_var_name + default: CIRCLE_JOB + make-targets: + type: string + executor: << parameters.machine-type >> + environment: + NAMESPACE: seleniarm + PLATFORMS: << parameters.platforms >> + BUILD_ARGS: << parameters.build-args >> + MAKE_TARGETS: << parameters.make-targets >> + DEPLOY_BRANCH: trunk + GITHUB_USER: seleniumhq-community + GITHUB_REPO: docker-seleniarm + steps: + - checkout + - run: + name: Debug + command: | + echo $CIRCLE_JOB + echo ${<< parameters.circle-job >>} + - when: + condition: + equal: [ base_multi, << parameters.make-targets >> ] + steps: + - run: echo "CIRCLE_JOB name is deploy-multi-arch-base" + - when: + condition: + equal: [ tag_and_push_multi_arch_browser_images, << parameters.make-targets >> ] + steps: + - run: echo "CIRCLE_JOB name is release notes, so we proceed" + - run: + name: "Prepare workflow environment variables" + command: | + echo "export SELENIUM_VERSION=$(grep BASE_VERSION Makefile | sed 's/.*,\([^)]*\))/\1/p' | head -n 1)" >> $BASH_ENV + echo "export RELEASE=$(grep BASE_RELEASE Makefile | sed 's/.*,\([^)]*\))/\1/p' | head -n 1)" >> $BASH_ENV + echo "export BINDING_VERSION=$(grep BINDING_VERSION Makefile | sed 's/.*,\([^)]*\))/\1/p' | head -n 1)" >> $BASH_ENV + source $BASH_ENV + echo "Prepare workflow environment variables" + echo 'export BRANCH='$SELENIUM_VERSION >> $BASH_ENV + echo 'export BUILD_DATE=$(date '+%Y%m%d')' >> $BASH_ENV + echo 'export RELEASE_TAG="seleniarm-v`echo $BRANCH`-`echo $BUILD_DATE`"' >> $BASH_ENV + source $BASH_ENV + echo "Workflow environment variables:" + echo BRANCH="$BRANCH" + echo BUILD_DATE="$BUILD_DATE" + echo RELEASE_TAG="$RELEASE_TAG" + echo NAMESPACE="$NAMESPACE" + echo PLATFORMS="$PLATFORMS" + echo BUILD_ARGS="$BUILD_ARGS" + echo DEPLOY_BRANCH="$DEPLOY_BRANCH" + echo GITHUB_USER="$GITHUB_USER" + echo GITHUB_REPO="$GITHUB_REPO" + echo SELENIUM_VERSION="$SELENIUM_VERSION" + echo RELEASE="$RELEASE" + - run: uname -a + - run: docker info + - run: + name: "Check if branch is deployable (contains [deploy] in commit msg on $DEPLOY_BRANCH" + command: | + echo BRANCH="$BRANCH" + echo BUILD_DATE="$BUILD_DATE" + echo RELEASE_TAG="$RELEASE_TAG" + export CI_DEPLOY=`git log --format=oneline -n 1 | grep '\[deploy\]'` && echo "$CI_DEPLOY" + if [ -z "$CI_DEPLOY" ] || [ "$CIRCLE_BRANCH" != "$DEPLOY_BRANCH" ]; then + echo "Cancelling run. Pass [deploy] in commit message on $DEPLOY_BRANCH to deploy" + circleci-agent step halt + else + echo "[deploy] is present in commit message to $DEPLOY_BRANCH. Running workflow to deploy container images" + fi + - run: + name: "Build and Push Docker images" + command: | + echo "Login to Docker, and setup to use a buildx builder and push built multi-arch images" + docker buildx use `docker buildx create` + docker buildx ls + # For release notes, we'll pull images anonymously without login. + # if [ "$MAKE_TARGETS" != "tag_and_push_multi_arch_browser_images" ]; then + docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD" + # fi + echo BRANCH="$BRANCH" + echo BUILD_DATE="$BUILD_DATE" + echo RELEASE_TAG="$RELEASE_TAG" + export CI_DEPLOY=`git log --format=oneline -n 1 | grep '\[deploy\]'` && echo "$CI_DEPLOY" + #NAME=${NAMESPACE} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} PLATFORMS=${PLATFORMS} BUILD_ARGS=${BUILD_ARGS} sh build-and-push.sh $MAKE_TARGETS + + export NAME=${NAMESPACE} + export VERSION=${BRANCH} + export RELEASE=${RELEASE} + export BUILD_DATE=${BUILD_DATE} + export PLATFORMS=${PLATFORMS} + export BUILD_ARGS=${BUILD_ARGS} + # If there are build errors, let's retry + max=5; until sh build-and-push.sh $MAKE_TARGETS; do if [ $((--max)) = 0 ]; then echo Giving up; exit 1; fi; done + + - when: + condition: + equal: [ tag_and_push_multi_arch_browser_images, << parameters.make-targets >> ] + steps: + - run: + name: "Push major/minor and browser tags to Docker Hub and update latest tag" + command: | + NAME=${NAMESPACE} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} PUSH_IMAGE=true make tag_and_push_multi_arch_browser_images + NAME=${NAMESPACE} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} make tag_multi_arch_latest + - run: + name: "Push major/minor tags to Docker Hub for non-browser images" + command: | + NAME=${NAMESPACE} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} PUSH_IMAGE=true make tag_major_minor_multi_arch + - run: + name: "Generate release notes" + command: | + docker logout + export LATEST_TAG=$(git describe --tags --abbrev=0) + sh generate_multi-arch-release_notes.sh $LATEST_TAG $CIRCLE_BRANCH $BRANCH $BUILD_DATE + sudo apt-get update -y && sudo apt-get install python3-venv + go install github.com/github-release/github-release@v0.10.0 + cat release_notes.md + export GITHUB_TOKEN=$(sh get-access-token.sh | tail -n 1) + github-release release --tag $RELEASE_TAG --name $RELEASE_TAG --description "`cat release_notes.md`" + +workflows: + build-and-test-multi-arch: + jobs: + - build-multi-arch: + name: build-multi-arch-arm64 + platforms: linux/arm64 + machine-type: ubuntu2004arm64 + - build-multi-arch: + name: build-multi-arch-amd64 + platforms: linux/amd64 + machine-type: ubuntu2004amd64 + - test-multi-arch: + name: test-multi-arch-arm64 + requires: [build-multi-arch-arm64] + platforms: linux/arm64 + machine-type: ubuntu2004arm64 + - test-multi-arch: + name: test-multi-arch-amd64 + requires: [build-multi-arch-amd64] + platforms: linux/amd64 + machine-type: ubuntu2004amd64 + + deploy-multi-arch-full-grid: + jobs: + - deploy-multi-arch-full-grid: + name: deploy-multi-arch-base + platforms: linux/arm64,linux/amd64 + build-args: --push + make-targets: base_multi + machine-type: ubuntu2004arm64large + filters: + branches: + only: + - trunk + - deploy-multi-arch-full-grid: + name: deploy-multi-arch-full-grid + requires: [deploy-multi-arch-base] + platforms: linux/arm64,linux/amd64 + build-args: --push + make-targets: grid_multi + machine-type: ubuntu2004arm64 + filters: + branches: + only: + - trunk + - deploy-multi-arch-full-grid: + name: deploy-multi-arch-node-base + requires: [deploy-multi-arch-base] + platforms: linux/arm64,linux/amd64 + build-args: --push + make-targets: node_base_multi + machine-type: ubuntu2004arm64large + filters: + branches: + only: + - trunk + - deploy-multi-arch-full-grid: + name: deploy-multi-arch-firefox + requires: [deploy-multi-arch-node-base] + platforms: linux/arm64,linux/amd64 + build-args: --push + make-targets: firefox_multi + machine-type: ubuntu2004arm64large + filters: + branches: + only: + - trunk + - deploy-multi-arch-full-grid: + name: deploy-multi-arch-chromium + requires: [deploy-multi-arch-node-base] + platforms: linux/arm64,linux/amd64 + build-args: --push + make-targets: chromium_multi + machine-type: ubuntu2004arm64large + filters: + branches: + only: + - trunk + - deploy-multi-arch-full-grid: + name: deploy-multi-arch-release-notes + requires: [deploy-multi-arch-firefox,deploy-multi-arch-chromium] + platforms: linux/arm64,linux/amd64 + build-args: --push + make-targets: tag_and_push_multi_arch_browser_images + machine-type: ubuntu2004arm64 + filters: + branches: + only: + - trunk diff --git a/.github/workflows/build-test-multi-arch.yml b/.github/workflows/build-test-multi-arch.yml new file mode 100644 index 0000000000..b8e9c941d5 --- /dev/null +++ b/.github/workflows/build-test-multi-arch.yml @@ -0,0 +1,59 @@ +name: Build & test multi-arch + +on: + # push: + # branches: + # - multi-arch-tests + # - qemu-user-static + # pull_request: + # branches: + # - trunk + workflow_dispatch: + +jobs: + build-and-test-multi-arch: + # Skip job based on the commit message, only works in push to branches for now + if: contains(toJson(github.event.commits), '[skip ci]') == false + name: Build & test multi-arch + runs-on: ubuntu-20.04 + strategy: + matrix: + use-random-user: [false, true] + arch: [amd64, arm64] + fail-fast: false + + steps: + - uses: actions/checkout@v1 + - name: Output Docker info + run: docker info + - name: Set up Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: 3.8 + - name: Get branch name (only for push to branch) + if: github.event_name == 'push' + run: echo "BRANCH=$(echo ${PUSH_BRANCH##*/})" >> $GITHUB_ENV + env: + PUSH_BRANCH: ${{ github.ref }} + - name: Get target branch name (only for PRs) + if: github.event_name == 'pull_request' + run: echo "BRANCH=$(echo ${TARGET_BRANCH##*/})" >> $GITHUB_ENV + env: + TARGET_BRANCH: ${{ github.head_ref }} + - name: Output branch name + run: echo ${BRANCH} + - name: Sets build date + run: echo "BUILD_DATE=$(date '+%Y%m%d')" >> $GITHUB_ENV + - name: Build Docker images + run: VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} ARCH=${ARCH} make build_multi + if: matrix.arch == 'arm64' && matrix.use-random-user == 'false' + - name: Test Docker images + run: | + docker run -d --platform linux/${ARCH} --rm -it -p 4444:4444 -p 7900:7900 --shm-size 2g selenium/standalone-firefox:${BRANCH}-${BUILD_DATE} + USE_RANDOM_USER_ID=${USE_RANDOM_USER} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} ARCH=${ARCH} SKIP_BUILD=true make test_firefox_standalone_multi + #USE_RANDOM_USER_ID=${USE_RANDOM_USER} VERSION=${BRANCH} BUILD_DATE=${BUILD_DATE} ARCH=${ARCH} SKIP_BUILD=true make test_chromium_standalone_multi + env: + USE_RANDOM_USER: ${{ matrix.use-random-user }} + ARCH: ${{ matrix.arch }} + if: matrix.arch == 'arm64' && matrix.use-random-user == 'false' + diff --git a/Base/Dockerfile b/Base/Dockerfile index e33da3ec69..89f3a41e79 100644 --- a/Base/Dockerfile +++ b/Base/Dockerfile @@ -36,12 +36,7 @@ ENV DEBIAN_FRONTEND=noninteractive \ # Miscellaneous packages # Includes minimal runtime used for executing non GUI Java programs #======================== -RUN if [ "${TARGETARCH}" = "amd64" ]; then \ - echo "deb http://archive.ubuntu.com/ubuntu jammy main universe\n" > /etc/apt/sources.list \ - && echo "deb http://archive.ubuntu.com/ubuntu jammy-updates main universe\n" >> /etc/apt/sources.list \ - && echo "deb http://security.ubuntu.com/ubuntu jammy-security main universe\n" >> /etc/apt/sources.list ; \ - fi \ - && apt-get -qqy update \ +RUN apt-get -qqy update \ && apt-get upgrade -yq \ && apt-get -qqy --no-install-recommends install \ acl \ diff --git a/Makefile b/Makefile index 0296ae48cd..f6272946b8 100644 --- a/Makefile +++ b/Makefile @@ -185,6 +185,77 @@ edge_upgrade_version: docker run --rm $(NAME)/standalone-edge:$(TAG_VERSION) microsoft-edge --version docker run --rm $(NAME)/standalone-edge:$(TAG_VERSION) msedgedriver --version +# Register linux/arm64 and linux/arm/v7 architectures for building with BuildKit +# docker run --rm --privileged aptman/qus -s -- -p # for actions +qemu_user_static: + docker run --rm --privileged aptman/qus -- -r ; \ + docker run --rm --privileged aptman/qus -s -- -p + +# Build multi-arch images +all_multi: base_multi \ + hub_multi \ + chromium_multi \ + firefox_multi \ + docker_multi \ + standalone_chromium_multi \ + standalone_firefox_multi \ + standalone_docker_multi \ + distributor_multi \ + router_multi \ + sessions_multi \ + sessionqueue_multi \ + event_bus_multi \ + video_multi + +build_multi: all_multi + +ci_multi: build_multi test_multi_arch + +base_multi: qemu_user_static + cd ./Base && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) --build-arg VERSION=$(BASE_VERSION) --build-arg RELEASE=$(BASE_RELEASE) -t $(NAME)/base:$(TAG_VERSION) . + +hub_multi: base_multi + cd ./Hub && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/hub:$(TAG_VERSION) . + +distributor_multi: + cd ./Distributor && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/distributor:$(TAG_VERSION) . + +router_multi: + cd ./Router && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/router:$(TAG_VERSION) . + +sessions_multi: + cd ./Sessions && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/sessions:$(TAG_VERSION) . + +sessionqueue_multi: + cd ./SessionQueue && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/session-queue:$(TAG_VERSION) . + +event_bus_multi: base_multi + cd ./EventBus && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/event-bus:$(TAG_VERSION) . + +node_base_multi: base_multi + cd ./NodeBase && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/node-base:$(TAG_VERSION) . + +chromium_multi: node_base_multi + cd ./NodeChromium && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/node-chromium:$(TAG_VERSION) . + +firefox_multi: node_base_multi + cd ./NodeFirefox && docker buildx build -f Dockerfile.multi-arch --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/node-firefox:$(TAG_VERSION) . + +docker_multi: base_multi + cd ./NodeDocker && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/node-docker:$(TAG_VERSION) . + +standalone_firefox_multi: firefox_multi + cd ./Standalone && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) --build-arg BASE=node-firefox -t $(NAME)/standalone-firefox:$(TAG_VERSION) . + +standalone_chromium_multi: chromium_multi + cd ./Standalone && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) --build-arg BASE=node-chromium -t $(NAME)/standalone-chromium:$(TAG_VERSION) . + +standalone_docker_multi: docker_multi + cd ./StandaloneDocker && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) $(FROM_IMAGE_ARGS) -t $(NAME)/standalone-docker:$(TAG_VERSION) . + +video_multi: + cd ./Video && docker buildx build --platform $(PLATFORMS) $(BUILD_ARGS) --build-arg NAMESPACE=$(FFMPEG_BASED_NAME) --build-arg BASED_TAG=$(FFMPEG_BASED_TAG) -t $(NAME)/video:$(FFMPEG_TAG_VERSION)-$(BUILD_DATE) . + # https://github.com/SeleniumHQ/docker-selenium/issues/992 # Additional tags for browser images tag_and_push_browser_images: tag_and_push_chrome_images tag_and_push_firefox_images tag_and_push_edge_images @@ -217,6 +288,58 @@ tag_latest: docker tag $(NAME)/standalone-docker:$(TAG_VERSION) $(NAME)/standalone-docker:latest docker tag $(NAME)/video:$(FFMPEG_TAG_VERSION)-$(BUILD_DATE) $(NAME)/video:latest +# Additional tags for browser images +tag_and_push_multi_arch_browser_images: tag_and_push_multi_arch_chromium_images tag_and_push_multi_arch_firefox_images + +tag_and_push_multi_arch_chromium_images: + ./tag_and_push_multi-arch_browser_images.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) chromium + +tag_and_push_multi_arch_firefox_images: + ./tag_and_push_multi-arch_browser_images.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) firefox + +tag_major_minor_multi_arch: + ./tag_and_push_multi-arch_major_minor.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) base + ./tag_and_push_multi-arch_major_minor.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) hub + ./tag_and_push_multi-arch_major_minor.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) node-base + ./tag_and_push_multi-arch_major_minor.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) node-docker + ./tag_and_push_multi-arch_major_minor.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) standalone-docker + ./tag_and_push_multi-arch_major_minor.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) sessions + ./tag_and_push_multi-arch_major_minor.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) session-queue + ./tag_and_push_multi-arch_major_minor.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) event-bus + ./tag_and_push_multi-arch_major_minor.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) router + ./tag_and_push_multi-arch_major_minor.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) $(PUSH_IMAGE) distributor + +tag_multi_arch_latest: + docker buildx imagetools create -t ${NAMESPACE}/base:latest ${NAMESPACE}/base:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/hub:latest ${NAMESPACE}/hub:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/node-base:latest ${NAMESPACE}/node-base:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/node-chromium:latest ${NAMESPACE}/node-chromium:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/node-firefox:latest ${NAMESPACE}/node-firefox:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/standalone-chromium:latest ${NAMESPACE}/standalone-chromium:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/standalone-firefox:latest ${NAMESPACE}/standalone-firefox:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/node-docker:latest ${NAMESPACE}/node-docker:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/standalone-docker:latest ${NAMESPACE}/standalone-docker:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/sessions:latest ${NAMESPACE}/sessions:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/session-queue:latest ${NAMESPACE}/session-queue:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/event-bus:latest ${NAMESPACE}/event-bus:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/router:latest ${NAMESPACE}/router:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/distributor:latest ${NAMESPACE}/distributor:${TAG_VERSION} + + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) base latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) hub latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) node-base latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) node-chromium latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) node-firefox latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) standalone-chromium latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) standalone-firefox latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) node-docker latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) standalone-docker latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) sessions latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) session-queue latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) event-bus latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) router latest + # ./tag-and-push-multi-arch-image.sh $(VERSION) $(BUILD_DATE) $(NAMESPACE) distributor latest + release_latest: docker push $(NAME)/base:latest docker push $(NAME)/hub:latest @@ -433,6 +556,25 @@ test_firefox: test_firefox_standalone: VERSION=$(TAG_VERSION) NAMESPACE=$(NAMESPACE) BINDING_VERSION=$(BINDING_VERSION) ./tests/bootstrap.sh StandaloneFirefox +# Test multi-arch container images +test_multi_arch: test_chromium_multi \ + test_firefox_multi \ + test_chromium_standalone_multi \ + test_firefox_standalone_multi + + +test_chromium_multi: + VERSION=$(TAG_VERSION) NAMESPACE=$(NAMESPACE) BINDING_VERSION=$(BINDING_VERSION) ./tests/bootstrap.sh NodeChromium + +test_chromium_standalone_multi: + VERSION=$(TAG_VERSION) NAMESPACE=$(NAMESPACE) BINDING_VERSION=$(BINDING_VERSION) ./tests/bootstrap.sh StandaloneChromium + +test_firefox_multi: + VERSION=$(TAG_VERSION) NAMESPACE=$(NAMESPACE) BINDING_VERSION=$(BINDING_VERSION) ./tests/bootstrap.sh NodeFirefox + +test_firefox_standalone_multi: + VERSION=$(TAG_VERSION) NAMESPACE=$(NAMESPACE) BINDING_VERSION=$(BINDING_VERSION) ./tests/bootstrap.sh StandaloneFirefox + test_parallel: hub chrome firefox edge sudo rm -rf ./tests/tests for node in DeploymentAutoscaling JobAutoscaling ; do \ diff --git a/NodeBase/Dockerfile b/NodeBase/Dockerfile index 2967f4fe28..327c0c88db 100644 --- a/NodeBase/Dockerfile +++ b/NodeBase/Dockerfile @@ -97,12 +97,6 @@ RUN apt-get update -qqy \ fonts-noto-color-emoji \ && locale-gen ${LANGUAGE} \ && dpkg-reconfigure --frontend noninteractive locales \ - && if [ "${TARGETARCH}" = "amd64" ]; then \ - apt-get -qqy --no-install-recommends install \ - language-pack-en \ - xfonts-cyrillic \ - fonts-ubuntu ; \ - fi \ && apt-get -qyy autoremove \ && rm -rf /var/lib/apt/lists/* \ && apt-get -qyy clean \ diff --git a/NodeChromium/Dockerfile b/NodeChromium/Dockerfile new file mode 100644 index 0000000000..94c03c1855 --- /dev/null +++ b/NodeChromium/Dockerfile @@ -0,0 +1,47 @@ +ARG NAMESPACE +ARG VERSION +ARG AUTHORS +FROM ${NAMESPACE}/node-base:${VERSION} +LABEL authors=${AUTHORS} + +USER root + +#ENV LANGUAGE=en_US.UTF-8 +#ENV LC_ALL=en_US.UTF-8 +#ENV LANG=en_US.UTF-8 + +# Deal with the usr merge issues in Debian +#RUN apt install usrmerge + +# Install Chromium +# RUN echo "deb http://http.us.debian.org/debian/ stable non-free contrib main" >> /etc/apt/sources.list \ +RUN echo "deb http://deb.debian.org/debian/ sid main" >> /etc/apt/sources.list \ + && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 0E98404D386FA1D9 6ED0E7B82643E131 \ + && apt-get update -qqy \ + # && apt-get -qqy install chromium=89.0.4389.82-1 \ + # && apt-get -qqy install chromium=90.0.4430.212-1 \ + # && apt-get -qqy install chromium=93.0.4577.82-1 \ + # && apt-get -qqy install chromium=97.0.4692.71-0.1 \ + # && apt-get -qqy install chromium=98.0.4758.102-1+b1 \ + && apt-get -qqy install chromium \ + && rm -rf /var/lib/apt/lists/* /var/cache/apt/* + +#================================= +# Chromium Launch Script Wrapper +#================================= +COPY wrap_chromium_binary /opt/bin/wrap_chromium_binary +RUN /opt/bin/wrap_chromium_binary + +#============================================ +# Chromium webdriver +#============================================ +RUN apt-get update -qqy \ + && apt-get -qqy install chromium-driver \ + && rm -rf /var/lib/apt/lists/* /var/cache/apt/* + +RUN echo "chrome" > /opt/selenium/browser_name +RUN chromium --version | awk '{print $2}' > /opt/selenium/browser_version +RUN echo "\"goog:chromeOptions\": {\"binary\": \"/usr/bin/chromium\"}" > /opt/selenium/browser_binary_location + +USER 1200 + diff --git a/NodeChromium/wrap_chromium_binary b/NodeChromium/wrap_chromium_binary new file mode 100755 index 0000000000..0d5947cad2 --- /dev/null +++ b/NodeChromium/wrap_chromium_binary @@ -0,0 +1,12 @@ +#!/bin/bash + +WRAPPER_PATH=$(readlink -f /usr/bin/chromium) +BASE_PATH="$WRAPPER_PATH-base" +mv "$WRAPPER_PATH" "$BASE_PATH" + +cat > "$WRAPPER_PATH" <<_EOF +#!/bin/bash +# Note: exec -a below is a bashism. +exec -a "\$0" "$BASE_PATH" --no-sandbox "\$@" +_EOF +chmod +x "$WRAPPER_PATH" diff --git a/NodeFirefox/Dockerfile.multi-arch b/NodeFirefox/Dockerfile.multi-arch new file mode 100644 index 0000000000..3ba8ab2516 --- /dev/null +++ b/NodeFirefox/Dockerfile.multi-arch @@ -0,0 +1,69 @@ +ARG NAMESPACE +ARG VERSION +ARG AUTHORS +FROM ${NAMESPACE}/node-base:${VERSION} +LABEL authors=${AUTHORS} + +ARG TARGETARCH +ARG TARGETVARIANT + +ARG GECKODRIVER_VERSION=0.34.0 + +USER root + +# Deal with the usr merge issues in Debian +#RUN apt install usrmerge + +#========= +# Firefox +#========= +#RUN echo "deb http://deb.debian.org/debian/ sid main" >> /etc/apt/sources.list \ + # && apt-get update -qqy \ + # && apt-get -qqy install firefox-esr libavcodec-extra \ + # && apt-get -qqy install firefox libavcodec-extra \ + # && rm -rf /var/lib/apt/lists/* /var/cache/apt/* + +# Pulling Firefox from Debian Snapshots so we can control which version we use as latest +RUN echo "deb http://deb.debian.org/debian/ sid main" >> /etc/apt/sources.list \ + && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 0E98404D386FA1D9 6ED0E7B82643E131 \ + && apt-get update -qqy \ + && apt-get install libavcodec-extra firefox -y \ + && rm -rf /var/lib/apt/lists/* /var/cache/apt/* + +#============= +# geckodriver +#============= +RUN if [ $TARGETARCH = "arm" ] && [ $TARGETVARIANT = "v7" ]; then \ + export ARCH=armhf ; \ + else \ + export ARCH=$TARGETARCH ; \ + fi ; \ + if [ -z "$ARCH" ]; then \ + echo "*** BUILD ERROR: \$TARGETARCH must be arm64, amd64, or arm with \$TARGETVARIANT set to v7... exiting..." ; \ + exit 1 ; \ + fi ; \ + if [ "$ARCH" = "arm64" ]; then \ + wget --no-verbose -O /tmp/geckodriver.tar.gz https://github.com/mozilla/geckodriver/releases/download/v$GECKODRIVER_VERSION/geckodriver-v$GECKODRIVER_VERSION-linux-aarch64.tar.gz ; \ + elif [ "$ARCH" = "armhf" ]; then \ + wget --no-verbose -O /tmp/geckodriver.tar.gz https://github.com/jamesmortensen/geckodriver-arm-binaries/releases/download/v$GECKODRIVER_VERSION/geckodriver-v$GECKODRIVER_VERSION-linux-armv7l.tar.gz ; \ + else \ + wget --no-verbose -O /tmp/geckodriver.tar.gz https://github.com/mozilla/geckodriver/releases/download/v$GECKODRIVER_VERSION/geckodriver-v$GECKODRIVER_VERSION-linux64.tar.gz ; \ + fi ; \ + tar -C /tmp -zxf /tmp/geckodriver.tar.gz ; \ + rm /tmp/geckodriver.tar.gz ; \ + mkdir -p /opt/geckodriver-bin ; \ + mv /tmp/geckodriver /opt/geckodriver-bin/geckodriver ; \ + echo "Symlinking geckodriver to /usr/local/bin/geckodriver and /usr/bin/geckodriver" ; \ + ln -s /opt/geckodriver-bin/geckodriver /usr/local/bin/geckodriver ; \ + chmod 755 /usr/local/bin/geckodriver ; \ + ln -s /opt/geckodriver-bin/geckodriver /usr/bin/geckodriver ; \ + chmod 755 /usr/bin/geckodriver + +USER 1200 + +#============================================ +# Dumping Browser name and version for config +#============================================ +RUN echo "firefox" > /opt/selenium/browser_name +RUN firefox --version | awk '{print $3}' > /opt/selenium/browser_version +RUN echo "\"moz:firefoxOptions\": {\"binary\": \"/usr/bin/firefox\"}" > /opt/selenium/browser_binary_location diff --git a/NodeFirefox/README.md b/NodeFirefox/README.md new file mode 100644 index 0000000000..c0b0ef1c64 --- /dev/null +++ b/NodeFirefox/README.md @@ -0,0 +1,28 @@ +## Building Multi-arch NodeFirefox and StandaloneFirefox + +There are two Dockerfiles in NodeFirefox. `Dockerfile` is from the upstream repository for building the standard, official amd64 images. To build `seleniarm/node-firefox` for arm64 or armv7l (or possibly amd64 as well), we use the `Dockerfile.multi-arch` file. + +The easiest way to build the image is to use `make`. See examples below: + + +**To build node/firefox for arm64:** + +``` +$ NAME=local-seleniarm VERSION=4.5.0 BUILD_DATE=$(date '+%Y%m%d') PLATFORMS=linux/arm64 BUILD_ARGS=--load make firefox_multi +``` + +**To build standalone/firefox for arm64:** + +``` +$ NAME=local-seleniarm VERSION=4.5.0 BUILD_DATE=$(date '+%Y%m%d') PLATFORMS=linux/arm64 BUILD_ARGS=--load make standalone_firefox_multi +``` + +NOTE: Replace PLATFORMS environment variable with `linux/arm/v7` for armv7l/armhf, or `linux/amd64` for amd64. + +## Running the standalone image + +``` +$ docker run --rm -it --shm-size 2g -p 4444:4444 -p 5900:5900 -p 7900:7900 local-seleniarm/standalone-firefox:latest +``` + +As with the x86_64 images from upstream, this also includes noVNC on port 7900, which we can access via http://localhost:7900 diff --git a/README.md b/README.md index 07cdc15016..1da21c4ba0 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,89 @@ See issue [#1076](https://github.com/SeleniumHQ/docker-selenium/issues/1076) for If you're working on an Intel or AMD64 architecture, we recommend using the container images in _this_ repository (SeleniumHQ/docker-selenium) instead of the experimental ones. + +### Docker images for Selenium, built for Ubuntu ARM64, ARM/v7, and AMD64 + +This is a fork of [SeleniumHQ/docker-selenium](https://github.com/SeleniumHQ/docker-selenium) for building and maintaining docker-selenium ARM images. This fork is inspired by and based on changes from [sj26/docker-selenium](https://github.com/sj26/docker-selenium) and [rows/docker-selenium](https://github.com/rows/docker-selenium). + +#### Running the ARM Container Images + +The primary motivation for creating this fork and updating the images was so I can use the noVNC client on the latest Selenium versions on the Mac M1, an arm64 architecture. To use noVNC, make sure you open port 7900, and visit localhost:7900 in your browser. + +The images are also successfully tested on AWS graviton nodes, resulting in better price-performance ratio and lower carbon footprint. + +To start the standalone container images, run: + +![Chromium](https://raw.githubusercontent.com/alrra/browser-logos/main/src/chromium/chromium_24x24.png) Chromium +``` +$ docker run --rm -it -p 4444:4444 -p 5900:5900 -p 7900:7900 --shm-size 2g seleniarm/standalone-chromium:latest +``` +> NOTE: Google does not build Chrome for Linux ARM platforms. Instead, docker-seleniarm uses the open source Chromium browser instead, which _is_ built for ARM. + +![Firefox](https://raw.githubusercontent.com/alrra/browser-logos/main/src/firefox/firefox_24x24.png) Firefox +``` +$ docker run --rm -it -p 4444:4444 -p 5900:5900 -p 7900:7900 --shm-size 2g seleniarm/standalone-firefox:latest +``` + +Use your traditional VNC client via port 5900, and noVNC in the browser via port 7900. + +The following multi-arch Seleniarm container images are available on [Docker Hub](https://hub.docker.com/u/seleniarm): + +- [Standalone Chromium](https://hub.docker.com/r/seleniarm/standalone-chromium) +- [Standalone Firefox](https://hub.docker.com/r/seleniarm/standalone-firefox) +- [Node Chromium](https://hub.docker.com/r/seleniarm/node-chromium) +- [Node Firefox](https://hub.docker.com/r/seleniarm/node-firefox) +- [Selenium Hub](https://hub.docker.com/r/seleniarm/hub) +- [Distributor](https://hub.docker.com/r/seleniarm/distributor) +- [Router](https://hub.docker.com/r/seleniarm/router) +- [Node Docker](https://hub.docker.com/r/seleniarm/node-docker) +- [Standalone Docker](https://hub.docker.com/r/seleniarm/standalone-docker) +- [Event Bus](https://hub.docker.com/r/seleniarm/event-bus) +- [Session Queue](https://hub.docker.com/r/seleniarm/session-queue) +- [Sessions](https://hub.docker.com/r/seleniarm/sessions) +- [NodeBase](https://hub.docker.com/r/seleniarm/node-base) +- [Base](https://hub.docker.com/r/seleniarm/base) + +All of the images are here in order to run in standalone mode, full grid mode, and dynamic grid mode. However, browser binaries are only available for Chromium and Firefox. + +> NOTE: Google does not build Chrome for ARM on Linux. Instead, we use Chromium ARM. + +#### Building the ARM Images + +The entire build process is managed via a Makefile. If you want to build the images locally, without pushing to any registry, then use `make`. + +#### Structure of container images: + +- The Standalone folder is the base for all Standalone${browser} images and includes a script that starts the selenium server in standalone mode. +- The NodeBase folder is the base for all Node${browser} images and includes a script that starts the selenium server in node mode. + +To build with a different version of Chromium, change it in NodeChromium/Dockerfile. + +To build the images, run the following make command from the root directory of this repo, and specify your architecture, either arm64, arm/v7, or amd64: + +**To build all arm64 images:** +``` +$ NAME=local-seleniarm VERSION=4.5.0 BUILD_DATE=$(date '+%Y%m%d') PLATFORMS=linux/arm64 BUILD_ARGS=--load make build_multi +``` + +**To build standalone/firefox for arm64:** + +``` +$ NAME=local-seleniarm VERSION=4.5.0 BUILD_DATE=$(date '+%Y%m%d') PLATFORMS=linux/arm64 BUILD_ARGS=--load make standalone_firefox_multi +``` + +**To build standalone/chromium for arm64:** + +``` +$ NAME=local-seleniarm VERSION=4.5.0 BUILD_DATE=$(date '+%Y%m%d') PLATFORMS=linux/arm64 BUILD_ARGS=--load make standalone_chromium_multi +``` + +To build for armv7l/armhf, replace PLATFORMS environment variable with `linux/arm/v7` like so: + +``` +$ NAME=local-seleniarm VERSION=4.5.0 BUILD_DATE=$(date '+%Y%m%d') PLATFORMS=linux/arm/v7 BUILD_ARGS=--load make standalone_chromium_multi +``` + ___ ## Nightly Images diff --git a/build-and-push.sh b/build-and-push.sh new file mode 100644 index 0000000000..250a56d907 --- /dev/null +++ b/build-and-push.sh @@ -0,0 +1,66 @@ +#!/bin/bash + +SELENIUM_VERSION=$(grep BASE_VERSION Makefile | sed 's/.*,\([^)]*\))/\1/p' | head -n 1) +RELEASE=$(grep BASE_RELEASE Makefile | sed 's/.*,\([^)]*\))/\1/p' | head -n 1) +NAME="${NAME:-seleniarm}" +VERSION="${VERSION:-$SELENIUM_VERSION}" +BUILD_DATE="${BUILD_DATE:-$(date '+%Y%m%d')}" +PLATFORMS="${PLATFORMS:-linux/arm64,linux/amd64}" +BUILD_ARGS=--push + +FROM_IMAGE_ARGS="--build-arg NAMESPACE=$NAME --build-arg VERSION=$VERSION-$BUILD_DATE" +TAG_VERSION=$VERSION-$BUILD_DATE + +START=$(date +'%s') +echo $START + +echo "Build and push images for target $1" + +docker run --rm --privileged aptman/qus -- -r +docker run --rm --privileged aptman/qus -s -- -p + +if [ "$1" = "base_multi" ]; then + cd ./Base && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} --build-arg RELEASE=${RELEASE} --build-arg VERSION=${VERSION} -t ${NAME}/base:${TAG_VERSION} . + +elif [ "$1" = "grid_multi" ]; then + cd ./Hub && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/hub:${TAG_VERSION} . + cd ../Distributor && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/distributor:${TAG_VERSION} . + cd ../Router && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/router:${TAG_VERSION} . + cd ../Sessions && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/sessions:${TAG_VERSION} . + cd ../SessionQueue && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/session-queue:${TAG_VERSION} . + cd ../NodeDocker && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/node-docker:${TAG_VERSION} . + cd ../EventBus && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/event-bus:${TAG_VERSION} . + # Prevent "failed to solve" errors by adding delay between NodeDocker and StandaloneDocker + # by building EventBus in between them. + cd ../StandaloneDocker && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/standalone-docker:${TAG_VERSION} . + +elif [ "$1" = "node_base_multi" ]; then + cd ./NodeBase && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/node-base:${TAG_VERSION} . + +elif [ "$1" = "firefox_multi" ]; then + FROM_IMAGE_ARGS="$FROM_IMAGE_ARGS --build-arg BASE=node-firefox" + cd ./NodeFirefox && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -f Dockerfile.multi-arch -t ${NAME}/node-firefox:${TAG_VERSION} . + sleep 5 # Prevent "failed to solve" errors when trying to pull NodeFirefox dependency + cd ../Standalone && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/standalone-firefox:${TAG_VERSION} . + +elif [ "$1" = "chromium_multi" ]; then + FROM_IMAGE_ARGS="$FROM_IMAGE_ARGS --build-arg BASE=node-chromium" + cd ./NodeChromium && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/node-chromium:${TAG_VERSION} . + sleep 5 # Prevent "failed to solve" errors when trying to pull NodeChromium dependency + cd ../Standalone && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/standalone-chromium:${TAG_VERSION} . + +elif [ "$1" = "tag_and_push_multi_arch_browser_images" ]; then + #make tag_and_push_multi_arch_browser_images + echo "Tag images and generate release notes" + +else + echo "$1 not found. Options are 'base_multi', 'grid_multi', 'node_base_multi', 'firefox_multi', and 'chromium_multi'" + SE_BUILD_CODE=1 +fi + +SE_BUILD_CODE=${SE_BUILD_CODE:-$(echo $?)} + +STOP=$(date +'%s') +echo $(( $STOP - $START )) seconds + +exit $SE_BUILD_CODE diff --git a/build-locally.sh b/build-locally.sh new file mode 100755 index 0000000000..b46a3eda5a --- /dev/null +++ b/build-locally.sh @@ -0,0 +1,66 @@ +#!/bin/bash + +SELENIUM_VERSION=$(grep BASE_VERSION Makefile | sed 's/.*,\([^)]*\))/\1/p' | head -n 1) +RELEASE=$(grep BASE_RELEASE Makefile | sed 's/.*,\([^)]*\))/\1/p' | head -n 1) +NAME="${NAME:-seleniarm}" +VERSION="${VERSION:-$SELENIUM_VERSION}" +BUILD_DATE="${BUILD_DATE:-$(date '+%Y%m%d')}" +PLATFORMS="${PLATFORMS:-linux/arm64,linux/amd64}" +#BUILD_ARGS=--push + +FROM_IMAGE_ARGS="--build-arg NAMESPACE=$NAME --build-arg VERSION=$VERSION-$BUILD_DATE" +TAG_VERSION=$VERSION-$BUILD_DATE + +START=$(date +'%s') +echo $START + +echo "Build and push images for target $1" + +#docker run --rm --privileged aptman/qus -- -r +#docker run --rm --privileged aptman/qus -s -- -p + +if [ "$1" = "base_multi" ]; then + cd ./Base && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} --build-arg RELEASE=${RELEASE} --build-arg VERSION=${VERSION} -t ${NAME}/base:${TAG_VERSION} . + +elif [ "$1" = "grid_multi" ]; then + cd ./Hub && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/hub:${TAG_VERSION} . + cd ../Distributor && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/distributor:${TAG_VERSION} . + cd ../Router && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/router:${TAG_VERSION} . + cd ../Sessions && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/sessions:${TAG_VERSION} . + cd ../SessionQueue && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/session-queue:${TAG_VERSION} . + cd ../NodeDocker && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/node-docker:${TAG_VERSION} . + cd ../EventBus && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/event-bus:${TAG_VERSION} . + # Prevent "failed to solve" errors by adding delay between NodeDocker and StandaloneDocker + # by building EventBus in between them. + cd ../StandaloneDocker && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/standalone-docker:${TAG_VERSION} . + +elif [ "$1" = "node_base_multi" ]; then + cd ./NodeBase && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/node-base:${TAG_VERSION} . + +elif [ "$1" = "firefox_multi" ]; then + FROM_IMAGE_ARGS="$FROM_IMAGE_ARGS --build-arg BASE=node-firefox" + cd ./NodeFirefox && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -f Dockerfile.multi-arch -t ${NAME}/node-firefox:${TAG_VERSION} . + sleep 5 # Prevent "failed to solve" errors when trying to pull NodeFirefox dependency + cd ../Standalone && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/standalone-firefox:${TAG_VERSION} . + +elif [ "$1" = "chromium_multi" ]; then + FROM_IMAGE_ARGS="$FROM_IMAGE_ARGS --build-arg BASE=node-chromium" + cd ./NodeChromium && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/node-chromium:${TAG_VERSION} . + sleep 5 # Prevent "failed to solve" errors when trying to pull NodeChromium dependency + cd ../Standalone && docker buildx build --platform ${PLATFORMS} ${BUILD_ARGS} ${FROM_IMAGE_ARGS} -t ${NAME}/standalone-chromium:${TAG_VERSION} . + +elif [ "$1" = "tag_and_push_multi_arch_browser_images" ]; then + #make tag_and_push_multi_arch_browser_images + echo "Tag images and generate release notes" + +else + echo "$1 not found. Options are 'base_multi', 'grid_multi', 'node_base_multi', 'firefox_multi', and 'chromium_multi'" + SE_BUILD_CODE=1 +fi + +SE_BUILD_CODE=${SE_BUILD_CODE:-$(echo $?)} + +STOP=$(date +'%s') +echo $(( $STOP - $START )) seconds + +exit $SE_BUILD_CODE diff --git a/docker-add-related-tags.sh b/docker-add-related-tags.sh new file mode 100644 index 0000000000..1028cc51a0 --- /dev/null +++ b/docker-add-related-tags.sh @@ -0,0 +1,17 @@ +#!/bin/bash + +NAMESPACE=${NAMESPACE:-seleniarm} +IMAGE=$1 +TAG=$2 +NO_PULL=$3 + +echo $NAMESPACE $IMAGE $TAG + +RELATED_TAGS=(`go run get-all-related-tags.go https://hub.docker.com/v2/repositories/$NAMESPACE/$IMAGE/tags/$TAG | tail -n 1`) + +for related_tag in "${RELATED_TAGS[@]}" + do + echo Add tag $NAMESPACE/$IMAGE:${related_tag} + docker tag $NAMESPACE/$IMAGE:$TAG $NAMESPACE/$IMAGE:$related_tag + done + diff --git a/generate_multi-arch-release_notes.sh b/generate_multi-arch-release_notes.sh new file mode 100755 index 0000000000..3167e7db76 --- /dev/null +++ b/generate_multi-arch-release_notes.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash + +LATEST_TAG=$1 +HEAD_BRANCH=$2 +GRID_VERSION=$3 +BUILD_DATE=$4 + +NAMESPACE="${NAMESPACE:-seleniarm}" +TAG_VERSION=${GRID_VERSION}-${BUILD_DATE} + +echo "" >> release_notes.md +echo "### Changelog" > release_notes.md +git --no-pager log "${LATEST_TAG}...${HEAD_BRANCH}" --pretty=format:"* [\`%h\`](http://github.com/seleniumhq-community/docker-seleniarm/commit/%H) - %s :: %an" --reverse >> release_notes.md + +############################################################## +# Pull the images so we can populate the release notes +# We'll pull using the TAG_VERSION and then add the other +# related tags separately to avoid exceeding the rate limits. +############################################################## +docker pull ${NAMESPACE}/base:${TAG_VERSION} +docker pull ${NAMESPACE}/hub:${TAG_VERSION} +docker pull ${NAMESPACE}/node-base:${TAG_VERSION} +docker pull ${NAMESPACE}/standalone-chromium:${TAG_VERSION} +docker pull ${NAMESPACE}/standalone-firefox:${TAG_VERSION} + +docker pull ${NAMESPACE}/node-chromium:${TAG_VERSION} +docker pull ${NAMESPACE}/node-firefox:${TAG_VERSION} +docker pull ${NAMESPACE}/node-docker:${TAG_VERSION} +docker pull ${NAMESPACE}/standalone-docker:${TAG_VERSION} +docker pull ${NAMESPACE}/sessions:${TAG_VERSION} +docker pull ${NAMESPACE}/session-queue:${TAG_VERSION} +docker pull ${NAMESPACE}/event-bus:${TAG_VERSION} +docker pull ${NAMESPACE}/router:${TAG_VERSION} +docker pull ${NAMESPACE}/distributor:${TAG_VERSION} + +###################################################################### +# Tags are already pushed to Docker Hub, but we need them set locally +# to generate release notes. Since we know the tags, we can set +# them locally to avoid exceeding the docker pull rate-limit. +###################################################################### +bash docker-add-related-tags.sh base ${TAG_VERSION} +bash docker-add-related-tags.sh hub ${TAG_VERSION} +bash docker-add-related-tags.sh node-base ${TAG_VERSION} +bash docker-add-related-tags.sh standalone-chromium ${TAG_VERSION} +bash docker-add-related-tags.sh standalone-firefox ${TAG_VERSION} +bash docker-add-related-tags.sh node-chromium ${TAG_VERSION} +bash docker-add-related-tags.sh node-firefox ${TAG_VERSION} +bash docker-add-related-tags.sh node-docker ${TAG_VERSION} +bash docker-add-related-tags.sh standalone-docker ${TAG_VERSION} +bash docker-add-related-tags.sh sessions ${TAG_VERSION} +bash docker-add-related-tags.sh session-queue ${TAG_VERSION} +bash docker-add-related-tags.sh event-bus ${TAG_VERSION} +bash docker-add-related-tags.sh router ${TAG_VERSION} +bash docker-add-related-tags.sh distributor ${TAG_VERSION} + +CHROMIUM_VERSION=$(docker run --rm ${NAMESPACE}/node-chromium:${TAG_VERSION} chromium --version | awk '{print $2}') +CHROMEDRIVER_VERSION=$(docker run --rm ${NAMESPACE}/node-chromium:${TAG_VERSION} chromedriver --version | awk '{print $2}') +FIREFOX_VERSION=$(docker run --rm ${NAMESPACE}/node-firefox:${TAG_VERSION} firefox --version | awk '{print $3}') +GECKODRIVER_VERSION=$(docker run --rm ${NAMESPACE}/node-firefox:${TAG_VERSION} geckodriver --version | awk 'NR==1{print $2}') + +echo "" >> release_notes.md +echo "### Released versions" >> release_notes.md +echo "* Selenium: ${GRID_VERSION}" >> release_notes.md +echo "* Chromium: ${CHROMIUM_VERSION}" >> release_notes.md +echo "* ChromiumDriver: ${CHROMEDRIVER_VERSION}" >> release_notes.md +echo "* Firefox: ${FIREFOX_VERSION}" >> release_notes.md +echo "* GeckoDriver: ${GECKODRIVER_VERSION}" >> release_notes.md + +echo "" >> release_notes.md +echo "### Published Docker images" >> release_notes.md +echo "
" >> release_notes.md +echo "Click to see published Docker images" >> release_notes.md +echo "" >> release_notes.md +echo '```' >> release_notes.md +docker images --filter=reference="seleniarm/*:*" --format "table {{.ID}}\t{{.Repository}}\t{{.Tag}}\t{{.Size}}" >> release_notes.md +echo '```' >> release_notes.md +echo "" >> release_notes.md +echo "
" >> release_notes.md diff --git a/get-access-token.py b/get-access-token.py new file mode 100644 index 0000000000..630783bf68 --- /dev/null +++ b/get-access-token.py @@ -0,0 +1,40 @@ +import time +from cryptography.hazmat.backends import default_backend +import jwt +import os +import requests +import logging + +github_app_id = os.environ.get('GITHUB_APP_ID') +github_installation_id = os.environ.get('GITHUB_INSTALLATION_ID') +private_key = os.environ.get('GITHUB_APP_PEM') +private_key = private_key.replace("\\n", "\n") + +standard_error_msg = 'Seleniarm GitHub App installation environment variables are not set. ' +if github_app_id == '': + raise Exception(standard_error_msg + 'Valid GITHUB_APP_ID is required to obtain an access token.') +if github_installation_id == '': + raise Exception(standard_error_msg + 'Valid GITHUB_INSTALLATION_ID is required to obtain an access token.') +if private_key == '': + raise Exception(standard_error_msg + 'Valid GITHUB_APP_PEM token is required to obtain an access token.') + + +time_since_epoch_in_seconds = int(time.time()) + +payload = { + # issued at time + 'iat': time_since_epoch_in_seconds, + # JWT expiration time (10 minute maximum) + 'exp': time_since_epoch_in_seconds + (10 * 60), + # GitHub App's identifier + 'iss': github_app_id +} + +actual_jwt = jwt.encode(payload, private_key, algorithm='RS256') + +headers = {"Authorization": "Bearer " + actual_jwt, + "Accept": "application/vnd.github.v3+json"} + +resp = requests.post('https://api.github.com/app/installations/' + github_installation_id + '/access_tokens', headers=headers) + +print(resp.json()['token']) diff --git a/get-access-token.sh b/get-access-token.sh new file mode 100644 index 0000000000..3fc7c6daa2 --- /dev/null +++ b/get-access-token.sh @@ -0,0 +1,18 @@ +#!/bin/sh + +python3 -m venv release_sandbox +. release_sandbox/bin/activate +if [ -z "$VIRTUAL_ENV" ]; then + echo "Virtual environment not activated." + exit 1 +fi + +pip3 install cryptography +pip3 install requests +pip3 install PyJWT + +export GITHUB_APP_ID="$SELENIARM_GITHUB_APP_ID" +export GITHUB_INSTALLATION_ID="$SELENIARM_GITHUB_INSTALLATION_ID" +export GITHUB_APP_PEM="$SELENIARM_GITHUB_APP_PEM" + +python3 get-access-token.py diff --git a/get-all-related-tags.go b/get-all-related-tags.go new file mode 100644 index 0000000000..650a9152bf --- /dev/null +++ b/get-all-related-tags.go @@ -0,0 +1,179 @@ +package main + +import ( + "encoding/json" + "fmt" + "io/ioutil" + "log" + "net/http" + "os" + "strconv" + "strings" +) + +type Latest struct { + Images []struct { + Architecture string `json:"architecture"` + Digest string `json:"digest"` + } +} + +type Result struct { + Results []struct { + Images []struct { + Digest string `json:"digest"` + } + Name string `json:"name"` + } +} + +func main() { + argLen := len(os.Args) + if argLen < 2 { + showUsage() + os.Exit(1) + } + url := os.Args[1] // https://hub.docker.com/v2/repositories/selenium/standalone-chrome/tags/latest/ + var inputArch string + if argLen >= 3 { + inputArch = os.Args[2] + } + + if inputArch == "" { + fmt.Println("Get related tags using sha256 for random architecture...") + } else { + fmt.Println("Get related tags using sha256 for " + inputArch + " architecture...") + } + + arch, tagDigest := getDigestForTag(url, inputArch) + fmt.Println("getting related tags for " + arch + " digest = " + tagDigest) + + var allTagsUrl string + var theseRelatedTags []string + var relatedTagsStr string + + allTagsUrl = getAllTagsUrl(url) + theseRelatedTags = getRelatedTagsFromDigest(allTagsUrl, tagDigest) + relatedTagsStr = strings.Join(theseRelatedTags, " ") + for next := true; next; next = (len(theseRelatedTags) != 0) { + allTagsUrl = getAllTagsUrlForNextPage(allTagsUrl) + theseRelatedTags = getRelatedTagsFromDigest(allTagsUrl, tagDigest) + relatedTagsStr = relatedTagsStr + " " + strings.Join(theseRelatedTags, " ") + // fmt.Println(theseRelatedTags) + } + + fmt.Println(relatedTagsStr) +} + +func getDigestForTag(url string, inputArch string) (string, string) { + resp, err := http.Get(url) + if err != nil { + log.Fatalln(err) + } + + body, err := ioutil.ReadAll(resp.Body) + if err != nil { + log.Fatalln(err) + } + + var jsonResp Latest + sb := string(body) + json.Unmarshal([]byte(sb), &jsonResp) + + var digest string + var arch string + if inputArch != "" { + for _, image := range jsonResp.Images { + if inputArch == image.Architecture { + digest = image.Digest + arch = image.Architecture + } + } + } else { + for _, image := range jsonResp.Images { + digest = image.Digest + arch = image.Architecture + } + } + + return arch, digest +} + +func getAllTagsUrlForNextPage(specificTagUrl string) string { + urlPathArr := strings.Split(specificTagUrl, "?") + var pageNum string + if len(urlPathArr) > 1 { + pageKeyVal := strings.Split(urlPathArr[1], "=") + pageNumInt, err := strconv.Atoi(pageKeyVal[1]) + if err != nil { + log.Fatalln(err) + } + pageNum = "?page=" + strconv.Itoa(pageNumInt+1) + } else { + pageNum = "?page=1" + } + return urlPathArr[0] + pageNum +} + +func getAllTagsUrl(specificTagUrl string) string { + urlPathArr := strings.Split(specificTagUrl, "?") + var pageNum string + if len(urlPathArr) > 1 { + pageNum = "?" + urlPathArr[1] + } else { + pageNum = "?page=1" + } + + specificTagUrlWithoutPage := urlPathArr[0] + urlArr := strings.Split(specificTagUrlWithoutPage, "/") + // fmt.Println(urlArr[len(urlArr)-1]) + if urlArr[len(urlArr)-1] == "" { + urlArr[len(urlArr)-1] = "" + urlArr[len(urlArr)-2] = "" + } else { + urlArr[len(urlArr)-1] = "" + } + urlArr = urlArr[:len(urlArr)-1] + allTagsUrl := strings.Join(urlArr, "/") // // https://hub.docker.com/v2/repositories/selenium/standalone-chrome/tags/ + return allTagsUrl + pageNum +} + +func getRelatedTagsFromDigest(allTagsUrl string, digest string) []string { + + resp, err := http.Get(allTagsUrl) + if err != nil { + log.Fatalln(err) + } + + body, err := ioutil.ReadAll(resp.Body) + if err != nil { + log.Fatalln(err) + } + + var jsonResp Result + sb := string(body) + json.Unmarshal([]byte(sb), &jsonResp) + + var relatedTags []string + for _, results := range jsonResp.Results { + for _, image := range results.Images { + imageDigest := image.Digest + if imageDigest == digest { + relatedTags = append(relatedTags, results.Name) + } + } + } + return relatedTags +} + +func showUsage() { + fmt.Println(`Usage: + get-all-related-tags TAG_URL [ARCH] + + TAG_URL -> URL for a container image manifest (Required) + ARCH -> Architecture to use to obtain sha256 (Optional) + + Example Usage: + $ get-all-related-tags https://hub.docker.com/v2/repositories/selenium/standalone-chrome/tags/latest/ + `) +} diff --git a/get-image-sha256-digest.go b/get-image-sha256-digest.go new file mode 100644 index 0000000000..d51b58ca5c --- /dev/null +++ b/get-image-sha256-digest.go @@ -0,0 +1,56 @@ +package main + +import ( + "io/ioutil" + "log" + "net/http" + "fmt" + "encoding/json" + "os" +) + +type Latest struct { + Images []struct { + Architecture string `json:"architecture"` + Digest string `json:"digest"` + } +} + + +func main() { + argLen := len(os.Args) + if argLen < 2 { + showUsage() + os.Exit(1) + } + + url := os.Args[1] // https://hub.docker.com/v2/repositories/selenium/standalone-chrome/tags/latest/ + resp, err := http.Get(url) + if err != nil { + log.Fatalln(err) + } + + body, err := ioutil.ReadAll(resp.Body) + if err != nil { + log.Fatalln(err) + } + + var jsonResp Latest + sb := string(body) + json.Unmarshal([]byte(sb), &jsonResp) + + for _, image := range jsonResp.Images { + fmt.Printf(image.Architecture + " " + image.Digest + "\n") + } +} + +func showUsage() { + fmt.Println(`Usage: + get-image-sha256-digest TAG_URL + + TAG_URL -> URL for a container image manifest (Required) + + Example Usage: + $ get-image-sha256-digest https://hub.docker.com/v2/repositories/selenium/standalone-chrome/tags/latest/ + `) +} diff --git a/tag-and-push-multi-arch-image.sh b/tag-and-push-multi-arch-image.sh new file mode 100755 index 0000000000..7e993dced9 --- /dev/null +++ b/tag-and-push-multi-arch-image.sh @@ -0,0 +1,25 @@ +#!/bin/bash + +VERSION=$1 +BUILD_DATE=$2 +NAMESPACE="${3:-seleniarm}" +IMAGE=$4 +NEW_TAG=$5 + +if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ] || [ -z "$4" ] || [ -z "$5" ]; then + echo "Be sure to pass in all of the values" + exit 1 +fi + + +AMD64_DIGEST=`go run get-image-sha256-digest.go https://hub.docker.com/v2/repositories/$NAMESPACE/$IMAGE/tags/$VERSION-$BUILD_DATE/ | grep -w "amd64" | awk '{print $2}'` +ARM_DIGEST=`go run get-image-sha256-digest.go https://hub.docker.com/v2/repositories/$NAMESPACE/$IMAGE/tags/$VERSION-$BUILD_DATE/ | grep -w "arm" | awk '{print $2}'` +ARM64_DIGEST=`go run get-image-sha256-digest.go https://hub.docker.com/v2/repositories/$NAMESPACE/$IMAGE/tags/$VERSION-$BUILD_DATE/ | grep -w "arm64" | awk '{print $2}'` + +docker manifest create $NAMESPACE/$IMAGE:$NEW_TAG \ + --amend $NAMESPACE/$IMAGE@$AMD64_DIGEST \ + --amend $NAMESPACE/$IMAGE@$ARM_DIGEST \ + --amend $NAMESPACE/$IMAGE@$ARM64_DIGEST + +docker manifest push $NAMESPACE/$IMAGE:$NEW_TAG + diff --git a/tag_and_push_multi-arch_browser_images.sh b/tag_and_push_multi-arch_browser_images.sh new file mode 100755 index 0000000000..f276f6e5a2 --- /dev/null +++ b/tag_and_push_multi-arch_browser_images.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash + +VERSION=$1 +BUILD_DATE=$2 +NAMESPACE=$3 +PUSH_IMAGE="${4:-false}" +BROWSER=$5 + +TAG_VERSION=${VERSION}-${BUILD_DATE} + +function short_version() { + local __long_version=$1 + local __version_split=( ${__long_version//./ } ) + echo "${__version_split[0]}.${__version_split[1]}" +} + +MAJOR=$(cut -d. -f1 <<<"${VERSION}") +MAJOR_MINOR=$(cut -d. -f1-2 <<<"${VERSION}") + +echo "Tagging images for browser ${BROWSER}, version ${VERSION}, build date ${BUILD_DATE}, namespace ${NAMESPACE}" + +case "${BROWSER}" in + +chromium) + CHROMIUM_VERSION=$(docker run --rm ${NAMESPACE}/node-chromium:${TAG_VERSION} chromium --version | awk '{print $2}') + echo "Chromium version -> "${CHROMIUM_VERSION} + CHROME_SHORT_VERSION="$(short_version ${CHROMIUM_VERSION})" + echo "Short Chromium version -> "${CHROME_SHORT_VERSION} + + CHROMEDRIVER_VERSION=$(docker run --rm ${NAMESPACE}/node-chromium:${TAG_VERSION} chromedriver --version | awk '{print $2}') + echo "ChromeDriver version -> "${CHROMEDRIVER_VERSION} + CHROMEDRIVER_SHORT_VERSION="$(short_version ${CHROMEDRIVER_VERSION})" + echo "Short ChromeDriver version -> "${CHROMEDRIVER_SHORT_VERSION} + + CHROME_TAGS=( + # Major Selenium version (X) + ${MAJOR} + # Major-minor Selenium version (X.Y) + ${MAJOR_MINOR} + # Full Selenium version (X.Y.X) + ${VERSION} + ${CHROMIUM_VERSION}-chromedriver-${CHROMEDRIVER_VERSION}-grid-${TAG_VERSION} + # Browser version and browser driver version plus build date + ${CHROMIUM_VERSION}-chromedriver-${CHROMEDRIVER_VERSION}-${BUILD_DATE} + # Browser version and browser driver version + ${CHROMIUM_VERSION}-chromedriver-${CHROMEDRIVER_VERSION} + # Browser version and build date + ${CHROMIUM_VERSION}-${BUILD_DATE} + # Browser version + ${CHROMIUM_VERSION} + ## Short versions + ${CHROME_SHORT_VERSION}-chromedriver-${CHROMEDRIVER_SHORT_VERSION}-grid-${TAG_VERSION} + # Browser version and browser driver version plus build date + ${CHROME_SHORT_VERSION}-chromedriver-${CHROMEDRIVER_SHORT_VERSION}-${BUILD_DATE} + # Browser version and browser driver version + ${CHROME_SHORT_VERSION}-chromedriver-${CHROMEDRIVER_SHORT_VERSION} + # Browser version and build date + ${CHROME_SHORT_VERSION}-${BUILD_DATE} + # Browser version + ${CHROME_SHORT_VERSION} + ) + + for chrome_tag in "${CHROME_TAGS[@]}" + do + if [ "${PUSH_IMAGE}" = true ]; then + docker buildx imagetools create -t ${NAMESPACE}/node-chromium:${chrome_tag} ${NAMESPACE}/node-chromium:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/standalone-chromium:${chrome_tag} ${NAMESPACE}/standalone-chromium:${TAG_VERSION} + #sh tag-and-push-multi-arch-image.sh $VERSION $BUILD_DATE $NAMESPACE node-chromium ${chrome_tag} + #sh tag-and-push-multi-arch-image.sh $VERSION $BUILD_DATE $NAMESPACE standalone-chromium ${chrome_tag} + fi + done + + ;; +firefox) + FIREFOX_VERSION=$(docker run --rm ${NAMESPACE}/node-firefox:${TAG_VERSION} firefox --version | awk '{print $3}') + echo "Firefox version -> "${FIREFOX_VERSION} + FIREFOX_SHORT_VERSION="$(short_version ${FIREFOX_VERSION})" + echo "Short Firefox version -> "${FIREFOX_SHORT_VERSION} + GECKODRIVER_VERSION=$(docker run --rm ${NAMESPACE}/node-firefox:${TAG_VERSION} geckodriver --version | awk 'NR==1{print $2}') + echo "GeckoDriver version -> "${GECKODRIVER_VERSION} + GECKODRIVER_SHORT_VERSION="$(short_version ${GECKODRIVER_VERSION})" + echo "Short GeckoDriver version -> "${GECKODRIVER_SHORT_VERSION} + + FIREFOX_TAGS=( + # Major Selenium version (X) + ${MAJOR} + # Major-minor Selenium version (X.Y) + ${MAJOR_MINOR} + # Full Selenium version (X.Y.X) + ${VERSION} + ${FIREFOX_VERSION}-geckodriver-${GECKODRIVER_VERSION}-grid-${TAG_VERSION} + # Browser version and browser driver version plus build date + ${FIREFOX_VERSION}-geckodriver-${GECKODRIVER_VERSION}-${BUILD_DATE} + # Browser version and browser driver version + ${FIREFOX_VERSION}-geckodriver-${GECKODRIVER_VERSION} + # Browser version and build date + ${FIREFOX_VERSION}-${BUILD_DATE} + # Browser version + ${FIREFOX_VERSION} + ## Short versions + ${FIREFOX_SHORT_VERSION}-geckodriver-${GECKODRIVER_SHORT_VERSION}-grid-${TAG_VERSION} + # Browser version and browser driver version plus build date + ${FIREFOX_SHORT_VERSION}-geckodriver-${GECKODRIVER_SHORT_VERSION}-${BUILD_DATE} + # Browser version and browser driver version + ${FIREFOX_SHORT_VERSION}-geckodriver-${GECKODRIVER_SHORT_VERSION} + # Browser version and build date + ${FIREFOX_SHORT_VERSION}-${BUILD_DATE} + # Browser version + ${FIREFOX_SHORT_VERSION} + ) + + for firefox_tag in "${FIREFOX_TAGS[@]}" + do + if [ "${PUSH_IMAGE}" = true ]; then + docker buildx imagetools create -t ${NAMESPACE}/node-firefox:${firefox_tag} ${NAMESPACE}/node-firefox:${TAG_VERSION} + docker buildx imagetools create -t ${NAMESPACE}/standalone-firefox:${firefox_tag} ${NAMESPACE}/standalone-firefox:${TAG_VERSION} + #sh tag-and-push-multi-arch-image.sh $VERSION $BUILD_DATE $NAMESPACE node-firefox ${firefox_tag} + #sh tag-and-push-multi-arch-image.sh $VERSION $BUILD_DATE $NAMESPACE standalone-firefox ${firefox_tag} + fi + done + + ;; +*) + echo "Unknown browser!" + ;; +esac diff --git a/tag_and_push_multi-arch_major_minor.sh b/tag_and_push_multi-arch_major_minor.sh new file mode 100755 index 0000000000..b7f563754c --- /dev/null +++ b/tag_and_push_multi-arch_major_minor.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash + +VERSION=$1 +BUILD_DATE=$2 +NAMESPACE=$3 +PUSH_IMAGE="${4:-false}" +IMAGE=$5 + +TAG_VERSION=${VERSION}-${BUILD_DATE} + +MAJOR=$(cut -d. -f1 <<<"${VERSION}") +MAJOR_MINOR=$(cut -d. -f1-2 <<<"${VERSION}") + +TAGS=( + $MAJOR + $MAJOR_MINOR + $VERSION +) + +for tag in "${TAGS[@]}" + do + if [ "${PUSH_IMAGE}" = true ]; then + docker buildx imagetools create -t ${NAMESPACE}/${IMAGE}:${tag} ${NAMESPACE}/${IMAGE}:${TAG_VERSION} + #sh tag-and-push-multi-arch-image.sh $VERSION $BUILD_DATE $NAMESPACE $IMAGE ${tag} + fi + done diff --git a/tests/test.py b/tests/test.py index 17add92282..14da41b04b 100644 --- a/tests/test.py +++ b/tests/test.py @@ -41,6 +41,10 @@ # Firefox Images 'NodeFirefox': 'node-firefox', 'StandaloneFirefox': 'standalone-firefox', + + # Chromium Images + 'NodeChromium': 'node-chromium', + 'StandaloneChromium': 'standalone-chromium', } TEST_NAME_MAP = { @@ -58,6 +62,10 @@ 'NodeFirefox': 'FirefoxTests', 'StandaloneFirefox': 'FirefoxTests', + # Chromium Images + 'NodeChromium': 'ChromeTests', + 'StandaloneChromium': 'ChromeTests', + # Chart Parallel Test 'JobAutoscaling': 'JobAutoscalingTests', 'DeploymentAutoscaling': 'DeploymentAutoscalingTests', @@ -168,7 +176,7 @@ def get_build_path(container): def standalone_browser_container_matches(container): - return re.match("(Standalone)(Chrome|Firefox|Edge)", container) + return re.match("(Standalone)(Chromium|Chrome|Firefox|Edge)", container) if __name__ == '__main__': @@ -199,9 +207,9 @@ def standalone_browser_container_matches(container): """ ports = {'4444': 4444} if use_random_user_id: - test_container_id = launch_container(image, ports=ports, user=random_user_id) + test_container_id = launch_container(image, ports=ports, user=random_user_id) else: - test_container_id = launch_container(image, ports=ports) + test_container_id = launch_container(image, ports=ports) else: """ Hub / Node Configuration @@ -211,9 +219,9 @@ def standalone_browser_container_matches(container): hub_id = launch_hub("grid") ports = {'5555': 5555, '7900': 7900} if use_random_user_id: - test_container_id = launch_container(image, network='grid', ports=ports, user=random_user_id) + test_container_id = launch_container(image, network='grid', ports=ports, user=random_user_id) else: - test_container_id = launch_container(image, network='grid', ports=ports) + test_container_id = launch_container(image, network='grid', ports=ports) prune_networks() logger.info('========== / Containers ready to go ==========') @@ -251,13 +259,13 @@ def standalone_browser_container_matches(container): test_container.remove() if standalone: - logger.info("Standalone Cleaned up") + logger.info("Standalone Cleaned up") else: - # Kill the launched hub - hub = client.containers.get(hub_id) - hub.kill() - hub.remove() - logger.info("Hub / Node Cleaned up") + # Kill the launched hub + hub = client.containers.get(hub_id) + hub.kill() + hub.remove() + logger.info("Hub / Node Cleaned up") if failed: exit(1)