Add dib image build test to hoist #354

eggshell · 2017-04-18T17:00:40Z

Currently, we have no tests for dib image builds using the elements
defined in hoist. This results in untested image builds ending up
in prod. This will add a job for that.

Implements: BonnyCI/projman#219

Signed-off-by: Cullen Taylor [email protected]

bonnyci · 2017-04-18T17:08:39Z

Build failed.

http://logs.bonnyci.org/check_github/BonnyCI/hoist/354/1492534843.82/bonnyci-run-check-multinode : FAILURE in 7m 49s

eggshell · 2017-04-18T18:45:58Z

recheck

bonnyci · 2017-04-18T18:53:46Z

Build failed.

http://logs.bonnyci.org/check_github/BonnyCI/hoist/354/1492541160.34/bonnyci-run-check-multinode : FAILURE in 7m 41s

gandelman-a · 2017-04-18T19:19:52Z

roles/zuul/files/jobs/dib.yaml

+
+ git clone https://github.com/BonnyCI/hoist.git $HOME
+ export ELEMENTS_PATH=$HOME/hoist/roles/nodepool/files/etc/nodepool/elements/nodepool
+ disk-image-create -a amd64 -o test-ubuntu vm ubuntu


So we want to actually test an image build with all the elements we have in-repo. The build command we actually end up running in production is:

disk-image-create -x -t qcow2 --checksum --no-tmpfs --qemu-img-options 'compat=0.10' -o /opt/nodepool/images/ubuntu-xenial-0000000037 ubuntu-minimal vm simple-init growroot openssh-server devuser haveged pip-and-virtualenv nodepool bonnyci-nodepool

Not sure if want to hard-code that into the job definition here, parse the elements-to-test out of a 'nodepool_diskimages' variable somewhere in inventory, or just try to build an image with all elements listed in $HOME/hoist/roles/nodepool/files/etc/nodepool/elements/nodepool ?

gandelman-a

Added some comments. Would like to get input from others on the team about whether we should be adding an extra job to our pipeline, or to continue to use the vanilla bonnyci job to keep up good dog foodin' and avoid taking advantage of features that we dont provide our users

gandelman-a · 2017-04-18T19:27:11Z

roles/zuul/templates/etc/zuul/config/layout.yaml

 gate_github:
 - bonnyci-run-gate-multinode
+ - dib-build-gate

 - name: BonnyCI/sandbox
 check_github:


So we should be able to only run the dib-build-pipeline job when files in the roles/nodepool/files/etc/nodepool/elements are changed by the patch under test. See https://docs.openstack.org/infra/zuul/zuul.html#jobs

I think we just need to add an entry to the layouts "jobs" section that applies the "files:" option to the dib job.

I agree, we should limit this job to only run when the elements are changed.

@gandelman-a @jesusaurus updated.

bonnyci · 2017-04-18T20:29:45Z

Build failed.

http://logs.bonnyci.org/check_github/BonnyCI/hoist/354/1492546917.05/bonnyci-run-check-multinode : FAILURE in 7m 38s

jesusaurus · 2017-04-21T22:32:47Z

roles/zuul/files/jobs/dib.yaml

@@ -0,0 +1,38 @@
+- publisher:


This publisher is already defined in bonnyci.yaml, I don't think we need to redefine it here.

jesusaurus · 2017-04-21T22:41:17Z

roles/zuul/files/jobs/dib.yaml

+ keep-hierarchy: true
+ copy-after-failure: true
+
+- job-template:


We're not doing anything different in the two pipelines, so I think we can just create a single job rather than use a job-template to build out multiple jobs.

@jesusaurus thanks for the feedback! I updated accordingly.

bonnyci · 2017-04-24T18:42:13Z

Build failed.

http://logs.bonnyci.org/check_github/BonnyCI/hoist/354/1493058829.1/bonnyci-run-check-multinode : FAILURE in 8m 15s

bonnyci · 2017-04-24T19:00:30Z

Build failed.

http://logs.bonnyci.org/check_github/BonnyCI/hoist/354/1493059839.61/bonnyci-run-check-multinode : FAILURE in 9m 41s

jesusaurus

+1 LGTM

Done

gandelman-a · 2017-05-01T20:23:10Z

roles/zuul/files/jobs/dib.yaml

+ disk-image-create -x -t qcow2 --checksum --no-tmpfs \
+ --qemu-img-options 'compat=0.10' \
+ -o /opt/nodepool/images/ubuntu-xenial \
+ ubuntu-minimal vm


We need to specify the elements from the hoist repo that we want to include in the build and test.

@gandelman-a isn't that what ELEMENTS_PATH specifies above?

@eggshell no, the elements path is where to find the listed elements, but this command only lists the two elements ubuntu-minimal and vm. I don't think there's a way to re-use the elements list from the nodepool_diskimages variable since it's scoped to a different host and all of these zuul jobs are files not templates, so for now the best thing might be to copy those elements into this command and add comments about keeping the two lists of elements in sync.

jesusaurus · 2017-05-02T18:54:54Z

roles/zuul/files/jobs/dib.yaml

+ disk-image-create -x -t qcow2 --checksum --no-tmpfs \
+ --qemu-img-options 'compat=0.10' \
+ -o /opt/nodepool/images/ubuntu-xenial \
+ ubuntu-minimal vm


@eggshell no, the elements path is where to find the listed elements, but this command only lists the two elements ubuntu-minimal and vm. I don't think there's a way to re-use the elements list from the nodepool_diskimages variable since it's scoped to a different host and all of these zuul jobs are files not templates, so for now the best thing might be to copy those elements into this command and add comments about keeping the two lists of elements in sync.

gandelman-a

Couple more comments.

Also, we need the job to install some distro package requirements for dib builds to work on the slave:
https://github.com/BonnyCI/hoist/blob/master/roles/nodepool/tasks/main.yml#L4

gandelman-a · 2017-05-02T20:13:36Z

roles/zuul/files/jobs/dib.yaml

+ . $HOME/dib-build/bin/activate
+ pip install diskimage-builder
+
+ git clone https://github.com/BonnyCI/hoist.git /opt/hoist


The bonnyci user isn't going to have access to create this directory. Need to sudo this or create it in ~

mlangbehn

This looks good to me.

gandelman-a · 2017-05-04T04:27:42Z

roles/zuul/files/jobs/dib.yaml

+ # with the elements specified in the nodepool role.
+ disk-image-create -x -t qcow2 --checksum --no-tmpfs \
+ --qemu-img-options 'compat=0.10' \
+ -o /opt/nodepool/images/ubuntu-xenial \


It looks like there'll also be an issue with the bonnyci user writing the image to /opt/nodepool/images/ubuntu-xenial

Currently, we have no tests for dib image builds using the elements defined in hoist. This results in untested image builds ending up in prod. This will add a job for that. Implements: BonnyCI/projman#219 Signed-off-by: Cullen Taylor <[email protected]>

gandelman-a

A few more things... FWIW I'm able to test this locally in a fresh ubuntu vm by just copying the shell snippet into a file and running it there. With my comments addressed, the build still quietly fails in elements/nodepool/extra.d/60-copy-nodepool-scripts with a pushd error.

gandelman-a · 2017-05-05T19:17:06Z

roles/zuul/files/jobs/dib.yaml

+ export ELEMENTS_PATH=/opt/hoist/roles/nodepool/files/etc/nodepool/elements/nodepool
+ # NOTE: be sure the list of nodepool elements here stays consistent
+ # with the elements specified in the nodepool role.
+ sudo disk-image-create -x -t qcow2 --checksum --no-tmpfs \


You loose the virtualenv environment when you sudo here, so disk-image-create is not in $PATH. Might make more sense to just write the image to /tmp or somewhere in $HOME instead?

gandelman-a · 2017-05-05T19:18:33Z

roles/zuul/files/jobs/dib.yaml

+ sudo apt-get install -y debootstrap qemu-utils curl \
+ uuid-runtime parted kpartx
+
+ sudo git clone https://github.com/BonnyCI/hoist.git /opt/hoist


So cloning and using the upstream repository means we wouldn't actually be testing the proposed changes here, and instead using whats already been merged. When the test runs, it'll be running from the root of the hoist repo with the proposed PR already checked out, so you should be able to just point ELEMENTS_PATH to ./roles/nodepoole/files/etc/nodepool/elements/

eggshell changed the title ~~Add dib image build test to hoist~~ [WIP] Add dib image build test to hoist Apr 18, 2017

gandelman-a reviewed Apr 18, 2017

View reviewed changes

eggshell force-pushed the dib_tests branch from 1aa4c64 to 941b435 Compare April 18, 2017 20:21

jesusaurus previously requested changes Apr 21, 2017

View reviewed changes

eggshell force-pushed the dib_tests branch 2 times, most recently from c631899 to 048e350 Compare April 24, 2017 00:28

eggshell changed the title ~~[WIP] Add dib image build test to hoist~~ Add dib image build test to hoist Apr 24, 2017

eggshell force-pushed the dib_tests branch 2 times, most recently from e186d21 to 7a89561 Compare April 24, 2017 18:33

eggshell force-pushed the dib_tests branch from 7a89561 to 39bfbb5 Compare April 24, 2017 18:50

eggshell force-pushed the dib_tests branch from 39bfbb5 to 2bef325 Compare April 24, 2017 19:10

eggshell mentioned this pull request Apr 24, 2017

Need to execute DIB image builds as part of CI tests BonnyCI/projman#219

Open

2 tasks

jesusaurus reviewed Apr 27, 2017

View reviewed changes

gandelman-a suggested changes May 1, 2017

View reviewed changes

jesusaurus suggested changes May 2, 2017

View reviewed changes

eggshell force-pushed the dib_tests branch from 2bef325 to 04cf477 Compare May 2, 2017 19:03

gandelman-a suggested changes May 2, 2017

View reviewed changes

eggshell force-pushed the dib_tests branch 2 times, most recently from cfd6bb9 to a31ccfa Compare May 2, 2017 21:08

mlangbehn reviewed May 3, 2017

View reviewed changes

gandelman-a suggested changes May 4, 2017

View reviewed changes

eggshell force-pushed the dib_tests branch from a31ccfa to 016eefd Compare May 4, 2017 14:00

gandelman-a suggested changes May 5, 2017

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dib image build test to hoist #354

Add dib image build test to hoist #354

eggshell commented Apr 18, 2017

bonnyci bot commented Apr 18, 2017

eggshell commented Apr 18, 2017

bonnyci bot commented Apr 18, 2017

gandelman-a Apr 18, 2017

gandelman-a left a comment

gandelman-a Apr 18, 2017

jesusaurus Apr 24, 2017

eggshell Apr 24, 2017

bonnyci bot commented Apr 18, 2017

jesusaurus Apr 21, 2017

jesusaurus Apr 21, 2017

eggshell Apr 24, 2017

bonnyci bot commented Apr 24, 2017

bonnyci bot commented Apr 24, 2017

jesusaurus left a comment

gandelman-a May 1, 2017

eggshell May 2, 2017

jesusaurus May 2, 2017

jesusaurus May 2, 2017

gandelman-a left a comment

gandelman-a May 2, 2017

mlangbehn left a comment

gandelman-a May 4, 2017

gandelman-a left a comment

gandelman-a May 5, 2017

gandelman-a May 5, 2017

Add dib image build test to hoist #354

Are you sure you want to change the base?

Add dib image build test to hoist #354

Conversation

eggshell commented Apr 18, 2017

bonnyci bot commented Apr 18, 2017

eggshell commented Apr 18, 2017

bonnyci bot commented Apr 18, 2017

Choose a reason for hiding this comment

gandelman-a left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bonnyci bot commented Apr 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bonnyci bot commented Apr 24, 2017

bonnyci bot commented Apr 24, 2017

jesusaurus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gandelman-a left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlangbehn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gandelman-a left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment