Ability to test dynamic config workflows

[jomakori]

I've gotten my config working now, but is there maybe in a future the ability to test dynamic config workflows?
The default config is a trigger based on changes filtered by folder.

@kienstra

ec2-user@ip-172-31-6-253 ~/devops (eks) [1]> doppler run -- /home/ec2-user/.vscode-server/extensions/localci.local-ci-1.7.9/bin/circleci/Linux/x64/bin/circleci local execute --job 'check-updated-files' --config /tmp/local-ci/devops/process.yml -v /tmp/local-ci/devops/volume:/tmp/local-ci
Fetching latest build environment...
Docker image digest: sha256:c3be9fb94fdf377e6a8d508fc0b833bb18a99a36a2d7d89a8e39c7701838f701
====>> Spin up environment
Build-agent version  ()
System information:
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: xfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Kernel Version: 4.14.290-217.505.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64

Starting container cimg/python:3.8
cimg/python:3.8:
  using image cimg/python@sha256:db17fcbb1469f1f70bbbd76867c6d371d4bf9596cae84ece3b62de8fe9c20fb2
  pull stats: Image was already available so the image was not pulled
  time to create container: 67ms
Creating Docker containers in parallel 
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image is cached as cimg/python:3.8, but refreshing...
3.8: Pulling from cimg/python
Digest: sha256:db17fcbb1469f1f70bbbd76867c6d371d4bf9596cae84ece3b62de8fe9c20fb2
Status: Image is up to date for cimg/python:3.8
Time to upload agent and config: 1.065111592s
Time to start containers: 1.170292043s
WARN: Missing API Key.
====>> Preparing environment variables
Using build environment variables:
  BASH_ENV=/tmp/.bash_env-localbuild-1661271079
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=check-updated-files
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=
  CIRCLE_SHA1=
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1661271079
  CIRCLE_WORKING_DIRECTORY=~/project


The redacted variables listed above will be masked in run step output.====>> Ensure volume is writable
  #!/bin/bash -eo pipefail
if [ "$(ls -ld /tmp/local-ci | awk '{print $3}')" != "$(whoami)" ] && [ "$(sudo -V 2>/dev/null)" ]
        then
        sudo chown $(whoami) /tmp/local-ci
      fi
====>> Checkout code
Making checkout directory "/home/circleci/project"
Copying files from "/tmp/_circleci_local_build_repo" to "/home/circleci/project"
Error: tar: s3/.terraform.lock.hcl: Cannot stat: Permission denied
tar: s3/resources-cacentral1/s3.tf: Cannot stat: Permission denied
tar: s3/resources-cacentral1/variables.tf: Cannot stat: Permission denied
tar: s3/resources-main/s3.tf: Cannot stat: Permission denied
tar: s3/resources-main/variables.tf: Cannot stat: Permission denied
tar: s3/resources-useast1/s3.tf: Cannot stat: Permission denied
tar: s3/resources-useast1/variables.tf: Cannot stat: Permission denied
tar: s3/resources-uswest1/s3.tf: Cannot stat: Permission denied
tar: s3/resources-uswest1/variables.tf: Cannot stat: Permission denied
tar: s3/resources-uswest2/s3.tf: Cannot stat: Permission denied
tar: s3/resources-uswest2/variables.tf: Cannot stat: Permission denied
tar: s3/s3-scan.arb: Cannot stat: Permission denied
tar: s3/versions.tf: Cannot stat: Permission denied
tar: Exiting with failure status due to previous errors

====>> Set more environment variables
  #!/bin/bash -eo pipefail
echo 'export CIRCLE_SHA1=$(git rev-parse HEAD)' >> $BASH_ENV
        echo 'export CIRCLE_BRANCH=$(git rev-parse --abbrev-ref HEAD)' >> $BASH_ENV
        echo 'export CIRCLE_PROJECT_REPONAME=$(basename $(git remote get-url origin))' >> $BASH_ENV
====>> Set parameters
  #!/usr/bin/env python3
#!/usr/bin/env python3

import json
import os
import re
import subprocess

def checkout(revision):
  """
  Helper function for checking out a branch

  :param revision: The revision to checkout
  :type revision: str
  """
  subprocess.run(
    ['git', 'checkout', revision],
    check=True
  )

output_path = os.environ.get('OUTPUT_PATH')
head = os.environ.get('CIRCLE_SHA1')
base_revision = os.environ.get('BASE_REVISION')
checkout(base_revision)  # Checkout base revision to make sure it is available for comparison
checkout(head)  # return to head commit

base = subprocess.run(
  ['git', 'merge-base', base_revision, head],
  check=True,
  capture_output=True
).stdout.decode('utf-8').strip()

if head == base:
  try:
    # If building on the same branch as BASE_REVISION, we will get the
    # current commit as merge base. In that case try to go back to the
    # first parent, i.e. the last state of this branch before the
    # merge, and use that as the base.
    base = subprocess.run(
      ['git', 'rev-parse', 'HEAD~1'], # FIXME this breaks on the first commit, fallback to something
      check=True,
      capture_output=True
    ).stdout.decode('utf-8').strip()
  except:
    # This can fail if this is the first commit of the repo, so that
    # HEAD~1 actually doesn't resolve. In this case we can compare
    # against this magic SHA below, which is the empty tree. The diff
    # to that is just the first commit as patch.
    base = '4b825dc642cb6eb9a060e54bf8d69288fbee4904'

print('Comparing {}...{}'.format(base, head))
changes = subprocess.run(
  ['git', 'diff', '--name-only', base, head],
  check=True,
  capture_output=True
).stdout.decode('utf-8').splitlines()

mappings = [
  m.split() for m in
  os.environ.get('MAPPING').splitlines()
]

def check_mapping(m):
  if 3 != len(m):
    raise Exception("Invalid mapping")
  path, param, value = m
  regex = re.compile(r'^' + path + r'$')
  for change in changes:
    if regex.match(change):
      return True
  return False

def convert_mapping(m):
  return [m[1], json.loads(m[2])]

mappings = filter(check_mapping, mappings)
mappings = map(convert_mapping, mappings)
mappings = dict(mappings)

with open(output_path, 'w') as fp:
  fp.write(json.dumps(mappings))


D       s3/.terraform.lock.hcl
D       s3/resources-cacentral1/s3.tf
D       s3/resources-cacentral1/variables.tf
D       s3/resources-main/s3.tf
D       s3/resources-main/variables.tf
D       s3/resources-useast1/s3.tf
D       s3/resources-useast1/variables.tf
D       s3/resources-uswest1/s3.tf
D       s3/resources-uswest1/variables.tf
D       s3/resources-uswest2/s3.tf
D       s3/resources-uswest2/variables.tf
D       s3/s3-scan.arb
D       s3/versions.tf
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
D       s3/.terraform.lock.hcl
D       s3/resources-cacentral1/s3.tf
D       s3/resources-cacentral1/variables.tf
D       s3/resources-main/s3.tf
D       s3/resources-main/variables.tf
D       s3/resources-useast1/s3.tf
D       s3/resources-useast1/variables.tf
D       s3/resources-uswest1/s3.tf
D       s3/resources-uswest1/variables.tf
D       s3/resources-uswest2/s3.tf
D       s3/resources-uswest2/variables.tf
D       s3/s3-scan.arb
D       s3/versions.tf
Note: switching to 'd86e26d78ffc1e311489e24a83ae7e052d4da8b7'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at d86e26d Revert "refactor: removed volume attachments"
Comparing cd6c9f6d54e06c518b41e57a00a0761d3084d83d...d86e26d78ffc1e311489e24a83ae7e052d4da8b7
====>> Continue the pipeline
  #!/bin/bash -eo pipefail
if [ -f /tmp/local-ci/dynamic-config.yml ]
                then
                  rm /tmp/local-ci/dynamic-config.yml
                fi
                cp .circleci/continue_config.yml /tmp/local-ci/dynamic-config.yml
                if [ -f /tmp/pipeline-parameters.json ]
                      then
                        cp /tmp/pipeline-parameters.json /tmp/local-ci/dynamic-config-parameters.json
                      fi
Success!

[kienstra]

Hi @jomakori,
Thanks a lot for bringing that up.

A dynamic config should work, though it sounds like it's not working in this case.

Would you be able to paste the .circleci/config.yml file here, or email it to me, of course redacting anything sensitive?

It looks like it's using the circleci/path-filtering orb. Maybe I need to fix something about Local CI.

[jomakori]

version: 2.1

# this allows you to use CircleCI's dynamic configuration feature
setup: true

# the path-filtering orb is required to continue a pipeline based on
# the path of an updated fileset
orbs:
  path-filtering: circleci/[email protected]

workflows:
  # the always-run workflow is always triggered, regardless of the pipeline parameters.
  always-run:
    jobs:
      # the path-filtering/filter job determines which pipeline
      # parameters to update.
      - path-filtering/filter:
          name: check-updated-files
          # 3-column, whitespace-delimited mapping. One mapping per
          # line:
          # <regex path-to-test> <parameter-to-set> <value-of-pipeline-parameter>
          mapping: |
            api-gateway/.* api-gateway true
            cloudfront/.* cloudfront true
            cloudwatch/.* cloudwatch true
            cluster-1-eksctl/.* cluster-1-eksctl true
            dynamodb/.* dynamodb true
            ec2/.* ec2 true
            ecr/.* ecr true
            efs/.* efs true
            elasticache/.* elasticache true
            iam/.* iam true
            kinesis/.* kinesis true
            lambda/.* lambda true
            rc-production/.* rc-production true
            rds/.* rds true
            route53/.* route53 true
            s3/.* s3 true
            sqs/.* sqs true
            vpc/.* vpc true
          base-revision: master
          # this is the path of the configuration we should trigger once
          # path filtering and pipeline parameter value updates are
          # complete. In this case, we are using the parent dynamic
          # configuration itself.
          config-path: .circleci/continue_config.yml

[kienstra]

Thanks, that config looks good. I'll test that with Local CI, looks like I might have a bug.

[jomakori]

Sounds good, thanks!!

[kienstra]

Another feature would include the ability to pass a prefix or pass env variables.
In our current setup, we use Dopper cli to pass in environmental variables. I manually added the doppler run -- prefix to the command to get my environment vars passed in

That's really interesting. Did you use it like doppler run -- circleci local execute --job your-job-name ?

[jomakori]

see #172
But technically yes, I could run it that way too!

[kienstra]

Oh thanks! Sorry, just looked at #172.

[jomakori]

no problem

[kienstra]

Hi @jomakori,
Would you be able to paste the result of:

cat /tmp/local-ci/devops/volume/dynamic-config-parameters.json

You might have to rerun the check-updated-files job, if that's alright.

I tried to replicate this using your .circleci/config.yml but couldn't. It does sound like a real bug, though.

Thanks a lot, Joel! It'd be great to get this working for you.