Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RHCOS 4.10/4.11 x86_64 builds seem to ignore change detection; always building #712

Closed
miabbott opened this issue Feb 10, 2022 · 3 comments · Fixed by coreos/coreos-assembler#2703
Closed

RHCOS 4.10/4.11 x86_64 builds seem to ignore change detection; always building #712

miabbott opened this issue Feb 10, 2022 · 3 comments · Fixed by coreos/coreos-assembler#2703
Labels
jira

Comments

@miabbott
Copy link
Member

miabbott commented Feb 10, 2022 •
edited
Loading

In the downstream pipelines, we've started to see evidence that the RHCOS 4.10/4.11 builds are not doing change detection properly and always making a new build.

For example, comparing two recent 4.10 builds:

410.84.202202101240-0 - https://url.corp.redhat.com/f320249

Previous build: 410.84.202202101041-0
Previous commit: c8ea771601bd21f5198e4504090c9992fac7dcefda108a8877473bc8b3a3a1b9
Image Config checksum: 0f8598063a72d6a5683ad058746edcf9d24345621fa4b88174dd82a11dbce03e

Input state hash: 42c1d15c18c2f003753a2524281df8ca28d422f547c960d2a94348f8a4bd2235

Wrote commit: 979c2da6b42c54e5dd0fd5a9fbe066ba4397eed3fc9888c138031a9218cd4816
ostree diff commit from: c8ea771601bd21f5198e4504090c9992fac7dcefda108a8877473bc8b3a3a1b9
ostree diff commit to:   979c2da6b42c54e5dd0fd5a9fbe066ba4397eed3fc9888c138031a9218cd4816

New image input checksum: 1618875e279df6c1531d4d25f6a13cef2b252761cac83b3acd6683276fda7c26
None
New build ID: 410.84.202202101240-0

410.84.202202101440-0 - https://url.corp.redhat.com/75fd9c3

Previous build: 410.84.202202101240-0
Previous commit: 979c2da6b42c54e5dd0fd5a9fbe066ba4397eed3fc9888c138031a9218cd4816
Image Config checksum: 0f8598063a72d6a5683ad058746edcf9d24345621fa4b88174dd82a11dbce03e

Input state hash: e238bf00d96bd78ea5911fc96e67c5a44680fe2480fbea775a0dc6ff09a1bc78

Wrote commit: ae776855bcb21b91993027cd7d3069b16a2f3ac0f1469b6359ead2b1d7f11f7d
ostree diff commit from: 979c2da6b42c54e5dd0fd5a9fbe066ba4397eed3fc9888c138031a9218cd4816
ostree diff commit to:   ae776855bcb21b91993027cd7d3069b16a2f3ac0f1469b6359ead2b1d7f11f7d

New image input checksum: 132c7e5f06d63424ee4fa7501b983cd6ad0836943bb50c46dd115b46eca248f5
None
New build ID: 410.84.202202101440-0

The build process is generating new checksums for image inputs + input state, but nothing has changed AFAICT.

Even more strange, this doesn't seem to affect the multi-arch pipelines.

I haven't had a chance to dig in further, but the constant builds are contributing to #701 and are generally causing us to make unnecessary builds, use unneeded resources.
@miabbott
Copy link
Member Author

Adding to the mystery, doing consecutive builds locally shows that the change detection is working. 😕

jlebon added a commit to jlebon/coreos-assembler that referenced this issue Feb 10, 2022
@jlebon
cmdlib: consistently commit layers with same options
1377b1e 
Make a helper for committing OSTree layers which contains all the
canonicalization flags. Notably `--timestamp` which ensures that we get
a consistent checksum for the same source git commit.

Fixes: openshift/os#712
Fixes: coreos#2603
@jlebon jlebon mentioned this issue Feb 10, 2022
Merged
@jlebon
Copy link
Member

jlebon commented Feb 10, 2022

I think coreos/coreos-assembler#2703 will fix this.

@jlebon jlebon added the jira label Feb 10, 2022
cgwalters pushed a commit to coreos/coreos-assembler that referenced this issue Feb 10, 2022
@jlebon @cgwalters
cmdlib: consistently commit layers with same options
3fac918 
Make a helper for committing OSTree layers which contains all the
canonicalization flags. Notably `--timestamp` which ensures that we get
a consistent checksum for the same source git commit.

Fixes: openshift/os#712
Fixes: #2603
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/coreos-assembler that referenced this issue Feb 11, 2022
@jlebon
cmdlib: consistently commit layers with same options
bf0c828 
Make a helper for committing OSTree layers which contains all the
canonicalization flags. Notably `--timestamp` which ensures that we get
a consistent checksum for the same source git commit.

Fixes: openshift/os#712
Fixes: coreos#2603
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/coreos-assembler that referenced this issue Feb 11, 2022
@jlebon
cmdlib: consistently commit layers with same options
e2c6b94 
Make a helper for committing OSTree layers which contains all the
canonicalization flags. Notably `--timestamp` which ensures that we get
a consistent checksum for the same source git commit.

Fixes: openshift/os#712
Fixes: coreos#2603
jlebon added a commit to coreos/coreos-assembler that referenced this issue Feb 11, 2022
@jlebon
cmdlib: consistently commit layers with same options
3793708 
Make a helper for committing OSTree layers which contains all the
canonicalization flags. Notably `--timestamp` which ensures that we get
a consistent checksum for the same source git commit.

Fixes: openshift/os#712
Fixes: #2603
@miabbott
Copy link
Member Author

Confirmed that we are correctly seeing no-op builds in the downstream pipeline now.

dustymabe pushed a commit to dustymabe/coreos-assembler that referenced this issue Nov 10, 2022
@jlebon @dustymabe
cmdlib: consistently commit layers with same options
c1331d1 
Make a helper for committing OSTree layers which contains all the
canonicalization flags. Notably `--timestamp` which ensures that we get
a consistent checksum for the same source git commit.

Fixes: openshift/os#712
Fixes: coreos#2603
(cherry picked from commit 3fac918)

dustymabe: This cherry pick needed manual merge conflict resolution.
           No contentsets in 4.9.
dustymabe pushed a commit to coreos/coreos-assembler that referenced this issue Nov 15, 2022
@jlebon @dustymabe
cmdlib: consistently commit layers with same options
f25a0ac 
Make a helper for committing OSTree layers which contains all the
canonicalization flags. Notably `--timestamp` which ensures that we get
a consistent checksum for the same source git commit.

Fixes: openshift/os#712
Fixes: #2603
(cherry picked from commit 3fac918)

dustymabe: This cherry pick needed manual merge conflict resolution.
           No contentsets in 4.9.
jlebon added a commit to coreos/coreos-assembler that referenced this issue Dec 2, 2022
@jlebon
cmdlib: consistently commit layers with same options
e0db126 
Make a helper for committing OSTree layers which contains all the
canonicalization flags. Notably `--timestamp` which ensures that we get
a consistent checksum for the same source git commit.

Fixes: openshift/os#712
Fixes: #2603
(cherry picked from commit 3fac918)

dustymabe: This cherry pick needed manual merge conflict resolution.
           No contentsets in 4.9.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
jira
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cmdlib: consistently commit layers with same options jlebon/coreos-assembler
2 participants
@miabbott @jlebon