Merge pull request #1210 from gene1wood/fix-docker-tag

Fix docker tagging and add CI/CD docs

CHANGELOG

@@ -5,6 +5,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ## [Unreleased]
 
+## [v1.38.4] - 2019-04-08
+### Fixed
+- Docker image tagging for git version tag builds
+- Correctly propagate the source ip address to the details.sourceipaddress in Duo logpull
+- Invalid literal in squidFixup.py destionationport field
+- Lowercase TAGS in squidFixup.py
+- Adding check for None type object in date fields to address GuardDuty null date
+
+### Added
+- Documentation on the CI/CD process
+- A summary to squidFixup.py
+- Tags assertions to tests
+
 ## [v1.38.3] - 2019-04-01
 ### Fixed
 - AWS CodeBuild tag semver regex
@@ -51,7 +64,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Added checks on sending SQS messages to only accept intra-account messages
 - Improved docker performance and disk space requirements
 
-[Unreleased]: https://github.com/mozilla/MozDef/compare/v1.38.3...HEAD
+[Unreleased]: https://github.com/mozilla/MozDef/compare/v1.38.4...HEAD
+[v1.38.4]: https://github.com/mozilla/MozDef/compare/v1.38.3...v1.38.4
 [v1.38.3]: https://github.com/mozilla/MozDef/compare/v1.38.2...v1.38.3
 [v1.38.2]: https://github.com/mozilla/MozDef/compare/v1.38.1...v1.38.2
 [v1.38.1]: https://github.com/mozilla/MozDef/compare/v1.38...v1.38.1

cloudy_mozdef/ci/docker_tag_or_push

@@ -8,14 +8,13 @@ for name in mozdef_meteor mozdef_base mozdef_tester mozdef_mq_worker mozdef_kiba
         mozdef_bootstrap mozdef_alerts mozdef_nginx mozdef_alertactions mozdef_rabbitmq \
         mozdef_rest mozdef_base ; do
     if [ "${action}" == "tag" ]; then
-        if [ "${branch}" == "master" ]; then
-            docker tag mozdef/${name}:latest mozdef/${name}:${branch}
-        else
-            docker tag mozdef/${name}:${branch}
-        fi
+        echo "Tagging local image mozdef/${name}:latest as mozdef/${name}:${branch}"
+        docker tag mozdef/${name}:latest mozdef/${name}:${branch}
     elif [ "${action}" == "push" ]; then
+        echo "Pushing local image mozdef/${name}:${branch} to DockerHub"
         docker push mozdef/${name}:${branch}
         if [ "${branch}" == "master" ]; then
+            echo "Pushing local image mozdef/${name}:latest to DockerHub"
             docker push mozdef/${name}:latest
         fi
     fi

docs/source/cicd.rst

@@ -0,0 +1,193 @@
+Continuous Integration and Continuous Deployment
+================================================
+
+Overview
+--------
+
+Each git commit to the `master` branch in GitHub triggers both the TravisCI
+automated tests as well as the AWS CodeBuild building. Each git tag applied to a
+git commit triggers a CodeBuild build.
+
+Travis CI
+---------
+
+Travis CI runs tests on the MozDef code base with each commit to `master`. The
+results can be seen on the
+`Travis CI MozDef dashboard <https://travis-ci.org/mozilla/MozDef/>`_
+
+The Test Sequence
+_________________
+
+* Travis CI creates webhooks when first setup which allow commits to the MozDef
+  GitHub repo to trigger Travis
+* When a commit is made to MozDef, Travis CI follows the instructions in the
+  `.travis.yml <https://github.com/mozilla/MozDef/blob/master/.travis.yml>`_
+  file
+* `.travis.yml` installs `docker-compose` in the `before_install` phase
+* in the `install` phase, Travis runs the
+  `build-tests <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L88-L89>`_
+  make target which calls `docker-compose build` on the
+  `docker/compose/docker-compose-tests.yml`_ file which builds a few docker
+  containers to use for testing
+* in the `script` phase, Travis runs the
+  `tests <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L52>`_
+  make target which
+
+  * calls the `build-tests` make target which again runs `docker-compose build`
+    on the `docker/compose/docker-compose-tests.yml`_ file
+  * calls the
+    `run-tests <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L67-L69>`_
+    make target which
+
+    * calls the
+      `run-tests-resources <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L60-L61>`_
+      make target which starts the docker
+      containers listed in `docker/compose/docker-compose-tests.yml`_
+    * runs `flake8` with the
+      `.flake8 <https://github.com/mozilla/MozDef/blob/master/.flake8>`_
+      config file to check code style
+    * runs `py.test tests` which runs all the test cases
+
+AWS CodeBuild
+-------------
+
+Enabling GitHub AWS CodeBuild Integration
+_________________________________________
+
+* Request that a github.com/mozilla GitHub Organization owner temporarily
+  `approve / whitelist
+  <https://help.github.com/en/articles/approving-oauth-apps-for-your-organization>`_
+  the `AWS CodeBuild integration <https://bugzilla.mozilla.org/show_bug.cgi?id=1506740>`_
+  in the github.com/mozilla GitHub Organization
+* Manually configure the GitHub integration in AWS CodeBuild which will create
+  the GitHub webhooks needed using the dedicated, AWS account specific, GitHub
+  service user. A service user is needed as AWS CodeBuild can only integrate
+  with GitHub from one AWS account in one region with a single GitHub user.
+  Technically we could use different users for each region in a single AWS
+  account, but for simplicity we're limiting to only one GitHub user per AWS
+  account (instead of one GitHub user per AWS account per region)
+
+  * For the `infosec-prod` AWS account use the `infosec-prod-371522382791-codebuild`
+    GitHub user
+  * For the `infosec-dev` AWS account use the `infosec-dev-656532927350-codebuild`
+    GitHub user
+
+* Request that a GitHub Organization owner, re-deny the integration for
+  github.com/mozilla
+
+The Build Sequence
+__________________
+
+* A branch is merged into `master` in the GitHub repo
+* GitHub emits a webhook event to AWS CodeBuild indicating that a commit was
+  pushed to `master`
+* AWS CodeBuild reads the
+  `buildspec.yml <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/buildspec.yml>`_
+  file to know what to do
+* The `install` phase of the `buildspec.yml` fetches
+  `packer <https://www.packer.io/>`_ and unzips it
+
+  * `packer` is a tool that spawns an ec2 instance, provisions it, and renders
+    an AWS Machine Image (AMI) from it.
+
+* The `build` phase of the `buildspec.yml` runs the
+  `cloudy_mozdef/ci/deploy <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/deploy>`_
+  script in the AWS CodeBuild Ubuntu 14.04 environment
+* The `deploy` script calls the
+  `build-from-cwd <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L78-L79>`_
+  target of the `Makefile` which calls `docker-compose build` on the
+  `docker-compose.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose.yml>`_
+  file, building the docker images in the AWS CodeBuild environment. These are
+  built so they can be uploaded later to DockerHub for use by developers and
+  the community.
+* The `deploy` script next calls the
+  `packer-build-github <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/Makefile#L34-L36>`_
+  make target in the
+  `cloudy_mozdef/Makefile <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/Makefile>`_
+  which calls the
+  `ci/pack_and_copy <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/pack_and_copy>`_
+  script which does the following steps
+
+  * Calls packer which launches an ec2 instance, executing a bunch of steps and
+    and producing an AMI
+  * Shares the resulting AMI with the AWS Marketplace account
+  * Copies the resulting AMI to a list of additional AWS regions
+  * Copies the tags from the original AMI to these copied AMIs in other regions
+  * Shares the AMIs in these other regions with the AWS Marketplace account
+  * Creates a blob of YAML which contains the AMI IDs. This blob will be used
+    in the CloudFormation templates
+
+* When `ci/pack_and_copy` calls packer, packer launches an ec2 instance based on
+  the configuration in
+  `cloudy_mozdef/packer/packer.json <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/packer/packer.json>`_
+
+  * Within this ec2 instance, packer `clones the MozDef GitHub repo and checks
+    out the branch that triggered this build
+    <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/packer/packer.json#L59-L60>`_
+  * packer calls the `set-version-and-fetch-docker-container <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L148-L149>`_
+    target of the `Makefile` which
+
+    * Calls the
+      `build-from-cwd <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L78-L79>`_
+      target of the `Makefile` which calls `docker-compose build` on the
+      `docker-compose.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose.yml>`_
+      file, building the docker images in the packer ec2 environment
+    * Calls the tag-images_
+      make target which calls the `cloudy_mozdef/ci/docker_tag_or_push tag <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
+      script.
+
+      * This applies a docker image tag to the local image that was just built
+        by packer. This tag will be referenced by the
+        `docker-compose-cloudy-mozdef.yml` file when the MozDef ec2 instance is
+        launched from the AMI.
+
+    * Replaces all instances of the word `latest` in the
+      `docker-compose-cloudy-mozdef.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-cloudy-mozdef.yml>`_
+      file with either the branch `master` or the version tag (e.g. `v1.2.3`)
+
+  * packer runs `docker-compose pull` on the
+    `docker-compose-cloudy-mozdef.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-cloudy-mozdef.yml>`_
+    file to pull down any remaining non MozDef container images that weren't
+    just built in preceding packer steps
+
+* After packer completes executing the steps laid out in `packer.json` inside
+  the ec2 instance, it generates an AMI from that instance and continues with
+  the copying, tagging and sharing steps described above
+* Now back in the AWS CodeBuild environment, the `deploy` script continues by
+  calling the
+  `publish-versioned-templates <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/Makefile#L85-L87>`_
+  make target which runs the
+  `ci/publish_versioned_templates <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/publish_versioned_templates>`_
+  script which
+
+  * injects the AMI map yaml blob produced
+    earlier into the
+    `mozdef-parent.yml <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/cloudformation/mozdef-parent.yml#L86-L87>`_
+    CloudFormation template so that the template knows the AMI IDs of that
+    specific branch of code.
+  * uploads the CloudFormation templates to S3 in a directory either called
+    `master` or the tag version that was built (e.g. `v1.2.3`)
+
+* `deploy` then calls the
+  `docker-push-tagged <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L113>`_
+  make target which calls
+
+  * the tag-images_
+    make target which calls the
+    `cloudy_mozdef/ci/docker_tag_or_push tag <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
+    script which applies a docker image tag to the local image that was just
+    built by AWS CodeBuild.
+  * the
+    `hub-tagged <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L116-L117>`_
+    make target which calls the
+    `cloudy_mozdef/ci/docker_tag_or_push push <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
+    script which
+
+    * Uploads the local image that was just built by AWS CodeBuild to DockerHub.
+      If the branch being built is `master` then the image is uploaded both with
+      a tag of `master` as well as with a tag of `latest`
+    * If the branch being built is from a version tag (e.g. `v1.2.3`) then the
+      image is uploaded with only that version tag applied
+
+.. _docker/compose/docker-compose-tests.yml: https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-tests.yml
+.. _tag-images: https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L109-L110

docs/source/index.rst

@@ -21,6 +21,7 @@ Table of Contents
   advanced_settings
   code
   benchmarking
+  cicd
   contributors