From e7cf6e4695e8f5f07d14ad3481279388d2197f6f Mon Sep 17 00:00:00 2001 From: Gene Wood Date: Mon, 8 Apr 2019 13:21:39 -0700 Subject: [PATCH 1/4] Fix incorrect docker tag syntax and add logging --- cloudy_mozdef/ci/docker_tag_or_push | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/cloudy_mozdef/ci/docker_tag_or_push b/cloudy_mozdef/ci/docker_tag_or_push index d4c82542c..407ed6cb1 100755 --- a/cloudy_mozdef/ci/docker_tag_or_push +++ b/cloudy_mozdef/ci/docker_tag_or_push @@ -8,14 +8,13 @@ for name in mozdef_meteor mozdef_base mozdef_tester mozdef_mq_worker mozdef_kiba mozdef_bootstrap mozdef_alerts mozdef_nginx mozdef_alertactions mozdef_rabbitmq \ mozdef_rest mozdef_base ; do if [ "${action}" == "tag" ]; then - if [ "${branch}" == "master" ]; then - docker tag mozdef/${name}:latest mozdef/${name}:${branch} - else - docker tag mozdef/${name}:${branch} - fi + echo "Tagging local image mozdef/${name}:latest as mozdef/${name}:${branch}" + docker tag mozdef/${name}:latest mozdef/${name}:${branch} elif [ "${action}" == "push" ]; then + echo "Pushing local image mozdef/${name}:${branch} to DockerHub" docker push mozdef/${name}:${branch} if [ "${branch}" == "master" ]; then + echo "Pushing local image mozdef/${name}:latest to DockerHub" docker push mozdef/${name}:latest fi fi From a3c0b672ec042045ceda0f020575d4d416b7922f Mon Sep 17 00:00:00 2001 From: Gene Wood Date: Mon, 8 Apr 2019 13:22:16 -0700 Subject: [PATCH 2/4] Add documentation on the CI/CD process --- docs/source/cicd.rst | 193 ++++++++++++++++++++++++++++++++++++++++++ docs/source/index.rst | 1 + 2 files changed, 194 insertions(+) create mode 100644 docs/source/cicd.rst diff --git a/docs/source/cicd.rst b/docs/source/cicd.rst new file mode 100644 index 000000000..eed94fcad --- /dev/null +++ b/docs/source/cicd.rst @@ -0,0 +1,193 @@ +Continuous Integration and Continuous Deployment +================================================ + +Overview +-------- + +Each git commit to the `master` branch in GitHub triggers both the TravisCI +automated tests as well as the AWS CodeBuild building. Each git tag applied to a +git commit triggers a CodeBuild build. + +Travis CI +--------- + +Travis CI runs tests on the MozDef code base with each commit to `master`. The +results can be seen on the +`Travis CI MozDef dashboard `_ + +The Test Sequence +_________________ + +* Travis CI creates webhooks when first setup which allow commits to the MozDef + GitHub repo to trigger Travis +* When a commit is made to MozDef, Travis CI follows the instructions in the + `.travis.yml `_ + file +* `.travis.yml` installs `docker-compose` in the `before_install` phase +* in the `install` phase, Travis runs the + `build-tests `_ + make target which calls `docker-compose build` on the + `docker/compose/docker-compose-tests.yml`_ file which builds a few docker + containers to use for testing +* in the `script` phase, Travis runs the + `tests `_ + make target which + + * calls the `build-tests` make target which again runs `docker-compose build` + on the `docker/compose/docker-compose-tests.yml`_ file + * calls the + `run-tests `_ + make target which + + * calls the + `run-tests-resources `_ + make target which starts the docker + containers listed in `docker/compose/docker-compose-tests.yml`_ + * runs `flake8` with the + `.flake8 `_ + config file to check code style + * runs `py.test tests` which runs all the test cases + +AWS CodeBuild +------------- + +Enabling GitHub AWS CodeBuild Integration +_________________________________________ + +* Request that a github.com/mozilla GitHub Organization owner temporarily + `approve / whitelist + `_ + the `AWS CodeBuild integration `_ + in the github.com/mozilla GitHub Organization +* Manually configure the GitHub integration in AWS CodeBuild which will create + the GitHub webhooks needed using the dedicated, AWS account specific, GitHub + service user. A service user is needed as AWS CodeBuild can only integrate + with GitHub from one AWS account in one region with a single GitHub user. + Technically we could use different users for each region in a single AWS + account, but for simplicity we're limiting to only one GitHub user per AWS + account (instead of one GitHub user per AWS account per region) + + * For the `infosec-prod` AWS account use the `infosec-prod-371522382791-codebuild` + GitHub user + * For the `infosec-dev` AWS account use the `infosec-dev-656532927350-codebuild` + GitHub user + +* Request that a GitHub Organization owner, re-deny the integration for + github.com/mozilla + +The Build Sequence +__________________ + +* A branch is merged into `master` in the GitHub repo +* GitHub emits a webhook event to AWS CodeBuild indicating that a commit was + pushed to `master` +* AWS CodeBuild reads the + `buildspec.yml `_ + file to know what to do +* The `install` phase of the `buildspec.yml` fetches + `packer `_ and unzips it + + * `packer` is a tool that spawns an ec2 instance, provisions it, and renders + an AWS Machine Image (AMI) from it. + +* The `build` phase of the `buildspec.yml` runs the + `cloudy_mozdef/ci/deploy `_ + script in the AWS CodeBuild Ubuntu 14.04 environment +* The `deploy` script calls the + `build-from-cwd `_ + target of the `Makefile` which calls `docker-compose build` on the + `docker-compose.yml `_ + file, building the docker images in the AWS CodeBuild environment. These are + built so they can be uploaded later to DockerHub for use by developers and + the community. +* The `deploy` script next calls the + `packer-build-github `_ + make target in the + `cloudy_mozdef/Makefile `_ + which calls the + `ci/pack_and_copy `_ + script which does the following steps + + * Calls packer which launches an ec2 instance, executing a bunch of steps and + and producing an AMI + * Shares the resulting AMI with the AWS Marketplace account + * Copies the resulting AMI to a list of additional AWS regions + * Copies the tags from the original AMI to these copied AMIs in other regions + * Shares the AMIs in these other regions with the AWS Marketplace account + * Creates a blob of YAML which contains the AMI IDs. This blob will be used + in the CloudFormation templates + +* When `ci/pack_and_copy` calls packer, packer launches an ec2 instance based on + the configuration in + `cloudy_mozdef/packer/packer.json `_ + + * Within this ec2 instance, packer `clones the MozDef GitHub repo and checks + out the branch that triggered this build + `_ + * packer calls the `set-version-and-fetch-docker-container `_ + target of the `Makefile` which + + * Calls the + `build-from-cwd `_ + target of the `Makefile` which calls `docker-compose build` on the + `docker-compose.yml `_ + file, building the docker images in the packer ec2 environment + * Calls the tag-images_ + make target which calls the `cloudy_mozdef/ci/docker_tag_or_push tag `_ + script. + + * This applies a docker image tag to the local image that was just built + by packer. This tag will be referenced by the + `docker-compose-cloudy-mozdef.yml` file when the MozDef ec2 instance is + launched from the AMI. + + * Replaces all instances of the word `latest` in the + `docker-compose-cloudy-mozdef.yml `_ + file with either the branch `master` or the version tag (e.g. `v1.2.3`) + + * packer runs `docker-compose pull` on the + `docker-compose-cloudy-mozdef.yml `_ + file to pull down any remaining non MozDef container images that weren't + just built in preceding packer steps + +* After packer completes executing the steps laid out in `packer.json` inside + the ec2 instance, it generates an AMI from that instance and continues with + the copying, tagging and sharing steps described above +* Now back in the AWS CodeBuild environment, the `deploy` script continues by + calling the + `publish-versioned-templates `_ + make target which runs the + `ci/publish_versioned_templates `_ + script which + + * injects the AMI map yaml blob produced + earlier into the + `mozdef-parent.yml `_ + CloudFormation template so that the template knows the AMI IDs of that + specific branch of code. + * uploads the CloudFormation templates to S3 in a directory either called + `master` or the tag version that was built (e.g. `v1.2.3`) + +* `deploy` then calls the + `docker-push-tagged `_ + make target which calls + + * the tag-images_ + make target which calls the + `cloudy_mozdef/ci/docker_tag_or_push tag `_ + script which applies a docker image tag to the local image that was just + built by AWS CodeBuild. + * the + `hub-tagged `_ + make target which calls the + `cloudy_mozdef/ci/docker_tag_or_push push `_ + script which + + * Uploads the local image that was just built by AWS CodeBuild to DockerHub. + If the branch being built is `master` then the image is uploaded both with + a tag of `master` as well as with a tag of `latest` + * If the branch being built is from a version tag (e.g. `v1.2.3`) then the + image is uploaded with only that version tag applied + +.. _docker/compose/docker-compose-tests.yml: https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-tests.yml +.. _tag-images: https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L109-L110 diff --git a/docs/source/index.rst b/docs/source/index.rst index 2c03670f0..4ab0591bd 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -21,6 +21,7 @@ Table of Contents advanced_settings code benchmarking + cicd contributors From c2f390d01fea5d2d282496d0443238aa73ea2dc9 Mon Sep 17 00:00:00 2001 From: Gene Wood Date: Mon, 8 Apr 2019 13:22:28 -0700 Subject: [PATCH 3/4] Update Changelog to 1.38.4 --- CHANGELOG | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/CHANGELOG b/CHANGELOG index c316effb1..9bf7c64d7 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -5,6 +5,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) ## [Unreleased] +## [v1.38.4] - 2019-04-08 +### Fixed +- Docker image tagging for git version tag builds + +### Added +- Documentation on the CI/CD process + ## [v1.38.3] - 2019-04-01 ### Fixed - AWS CodeBuild tag semver regex @@ -51,7 +58,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) - Added checks on sending SQS messages to only accept intra-account messages - Improved docker performance and disk space requirements -[Unreleased]: https://github.com/mozilla/MozDef/compare/v1.38.3...HEAD +[Unreleased]: https://github.com/mozilla/MozDef/compare/v1.38.4...HEAD +[v1.38.4]: https://github.com/mozilla/MozDef/compare/v1.38.3...v1.38.4 [v1.38.3]: https://github.com/mozilla/MozDef/compare/v1.38.2...v1.38.3 [v1.38.2]: https://github.com/mozilla/MozDef/compare/v1.38.1...v1.38.2 [v1.38.1]: https://github.com/mozilla/MozDef/compare/v1.38...v1.38.1 From 43951fa3105414b5fbf2270fba99d8a64b267807 Mon Sep 17 00:00:00 2001 From: Gene Wood Date: Mon, 8 Apr 2019 13:55:17 -0700 Subject: [PATCH 4/4] Add Changelog entries for existing commits slated for 1.38.4 --- CHANGELOG | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG b/CHANGELOG index 9bf7c64d7..5ec4c3909 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -8,9 +8,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) ## [v1.38.4] - 2019-04-08 ### Fixed - Docker image tagging for git version tag builds +- Correctly propagate the source ip address to the details.sourceipaddress in Duo logpull +- Invalid literal in squidFixup.py destionationport field +- Lowercase TAGS in squidFixup.py +- Adding check for None type object in date fields to address GuardDuty null date ### Added - Documentation on the CI/CD process +- A summary to squidFixup.py +- Tags assertions to tests ## [v1.38.3] - 2019-04-01 ### Fixed