next: new release on 2022-05-03 - 36.20220430.1.0 #496

gursewak1997 · 2022-04-27T21:37:27Z

First, verify that you meet all the prerequisites

Edit the issue title to include today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.2.0).

Pre-release

Promote next-devel changes to next

Add the ok-to-promote label to the issue
Review the promotion PR against the next branch on https://github.com/coreos/fedora-coreos-config
Once CI has passed, merge it

Manual alternative

Sometimes you need to run the process manually like if you need to add an extra commit to change something in manifest.yaml. The steps for this are:

git fetch upstream
git checkout next
git reset --hard upstream/next
/path/to/fedora-coreos-releng-automation/scripts/promote-config.sh next-devel
Open PR against the next branch on https://github.com/coreos/fedora-coreos-config

Build

Start a build job (select next, leave all other defaults). This will automatically run multi-arch builds.
Post links to the jobs as a comment to this issue
- x86_64
- aarch64 (multi-arch build job)
Wait for the jobs to finish and succeed
- x86_64
- aarch64

Sanity-check the build

Using the the build browser for the next stream:

Verify that the parent commit and version match the previous next release (in the future, we'll want to integrate this check in the release job)
- x86_64
- aarch64
Check kola AWS runs to make sure they didn't fail
- x86_64
- aarch64
Check kola OpenStack runs to make sure they didn't fail
- x86_64
- aarch64
Check kola Azure run to make sure it didn't fail
Check kola GCP run to make sure it didn't fail

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

Run the release job, filling in for parameters next and the new version ID
Post a link to the job as a comment to this issue
Wait for job to finish

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

Wait for all releases that will be released simultaneously to reach this step in the process
Go to the rollout workflow, click "Run workflow", and fill out the form

Manual alternative

Make sure your fedora-coreos-stream-generator binary is up-to-date.

From a checkout of this repo:

Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/next/releases.json  -output-file=streams/next.json -pretty-print

Add a rollout. For example, for a 48-hour rollout starting at 10 AM ET the same day, run:

./rollout.py add next <version> "10 am ET today" 48

Commit the changes and open a PR against the repo

Verify that the PR contains the expected OS versions
Post a link to the resulting PR as a comment to this issue
Review and approve the PR, then wait for someone else to approve it also
Once approved, merge it and verify that the sync-stream-metadata job syncs the contents to S3
Verify the new version shows up on the download page
Verify the incoming edges are showing up in the update graph.
- x86_64
- aarch64

Update graph manual check

curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=next&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=aarch64&stream=next&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated.

Housekeeping

If one doesn't already exist, open an issue in this repo for the next release in this stream. Use the approximate date of the release in the title.
Issues opened via the previous link will automatically create a linked Jira card. Assign the GitHub issue and Jira card to the next person in the rotation.

The text was updated successfully, but these errors were encountered:

cverna · 2022-05-02T13:17:23Z

Build job :
https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/build/578/
https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/build-arch/241/

cverna · 2022-05-03T05:07:52Z

AWS x86: https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/kola-aws/283/
AWS aarch64: https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/kola-aws/284/
OpenStack x86 : https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/kola-openstack/291/
OpenStack aarch64: https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/kola-openstack/292/
Azure : https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/kola-azure/59/
GCP : https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/kola-gcp/143/

cverna · 2022-05-03T05:09:11Z

Release job : https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/release/294/

dustymabe · 2022-05-03T16:18:10Z

Release job : https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/release/294/

That release job and also release#295 failed with:

[2022-05-03T05:09:59.798Z] + cosa aws-replicate --build=36.20220430.1.0 --arch=aarch64 --log-level=INFO
[2022-05-03T05:10:00.052Z] [INFO]: symlink is for a replication command
[2022-05-03T05:10:00.052Z] [INFO]: ore target aws found via symlink
[2022-05-03T05:10:00.052Z] [INFO]: Targeting architecture: aarch64
[2022-05-03T05:10:00.052Z] [INFO]: Targeting build: 36.20220430.1.0
[2022-05-03T05:10:00.052Z] [INFO]: Processed build for: Fedora CoreOS next (FEDORA-COREOS-aarch64) 36.20220430.1.0
[2022-05-03T05:10:00.052Z] Extracting 749f29e547566fa2d7f134bc9a61a52b579a26b9b829159855a9a963d791712a
[2022-05-03T05:10:00.052Z] �[0m�[31merror: �[0mImporting ostree-unverified-image:oci-archive:/home/jenkins/agent/workspace/release/tmp/tmp2zdn0lhc/fedora-coreos-36.20220430.1.0-ostree.aarch64.ociarchive: remote error: creating temp directory: open /home/jenkins/agent/workspace/release/tmp/tmp2zdn0lhc/fedora-coreos-36.20220430.1.0-ostree.aarch64.ociarchive: no such file or directory
[2022-05-03T05:10:00.052Z] Traceback (most recent call last):
[2022-05-03T05:10:00.052Z]   File "/usr/lib/coreos-assembler/cmd-aws-replicate", line 109, in <module>
[2022-05-03T05:10:00.052Z]     build = get_qemu_variant(target, args)
[2022-05-03T05:10:00.052Z]   File "/usr/lib/coreos-assembler/cosalib/qemuvariants.py", line 138, in get_qemu_variant
[2022-05-03T05:10:00.052Z]     return QemuVariantImage(
[2022-05-03T05:10:00.052Z]   File "/usr/lib/coreos-assembler/cosalib/qemuvariants.py", line 188, in __init__
[2022-05-03T05:10:00.052Z]     import_ostree_commit(repo, self._tmpdir, self.meta)
[2022-05-03T05:10:00.052Z]   File "/usr/lib/coreos-assembler/cosalib/cmdlib.py", line 277, in import_ostree_commit
[2022-05-03T05:10:00.052Z]     subprocess.check_call(['ostree', 'container', 'import', '--repo', tmpd,
[2022-05-03T05:10:00.052Z]   File "/usr/lib64/python3.10/subprocess.py", line 369, in check_call
[2022-05-03T05:10:00.052Z]     raise CalledProcessError(retcode, cmd)
[2022-05-03T05:10:00.052Z] subprocess.CalledProcessError: Command '['ostree', 'container', 'import', '--repo', '/home/jenkins/agent/workspace/release/tmp/tmpuoal7rp8', '--write-ref', '36.20220430.1.0', 'ostree-unverified-image:oci-archive:/home/jenkins/agent/workspace/release/tmp/tmp2zdn0lhc/fedora-coreos-36.20220430.1.0-ostree.aarch64.ociarchive']' returned non-zero exit status 1.

cgwalters · 2022-05-03T16:40:43Z

Hmm may be related to coreos/coreos-assembler#2811

(Before that merged I did coreos/fedora-coreos-tracker#1068 (comment) so we should be able to use that :stable tag to do these releases I hope)

cgwalters · 2022-05-03T19:18:02Z

I'm not sure; I see in the logs

Fetching: s3://fcos-builds/prod/streams/next/builds/36.20220430.1.0/aarch64/fedora-coreos-36.20220430.1.0-ostree.aarch64.ociarchive

so it clearly should be there, and I don't think the signing code is deleting it.

cgwalters · 2022-05-03T19:21:18Z

Although it does seem like cosa aws-replicate should absolutely not be calling get_qemu_variant() which is now (as of cosa PR 2811) calling import_ostree_commit.

This reverts commit e0c3296. This broke the `next` stream release at the `cosa aws-replicate` step: coreos/fedora-coreos-streams#496 (comment)

jlebon · 2022-05-03T19:57:57Z

Although it does seem like cosa aws-replicate should absolutely not be calling get_qemu_variant() which is now (as of cosa PR 2811) calling import_ostree_commit.

Yeah, I think what's going on there is that cmd-artifact-disk uses the same library API and there we do need the image.json for various disk image-related knobs. I'm not entirely sure why the particular operation failed since we do fetch both ociarchives, though the path to the ociarchive looks odd there (it's in tmp/tmp2zdn0lhc, but I'd expect builds/... instead).

Anyway, I think let's revert this for now to unbreak latest cosa: coreos/coreos-assembler#2838 (and maybe we should consider having testing-devel do AWS replication to one other region so it's closer to the production stream flow).

cgwalters · 2022-05-03T21:06:08Z

I think it's that we're trying to allocate a "build" for aws-replicate and so the context is operating in a temporary directory that doesn't actually have an initialized build yet.

Man the code here is...such a mess. I am just not a fan of object orientation anymore, or dynamic languages in general. Then there's this completely dynamic logic that seems related:


        # Setup what is required for this Class.
        #   require_cosa means that the COSA information is need
        #   require_commit is usually only needed at build time
        require_cosa = kwargs.get("require_cosa", False)
        require_commit = kwargs.get("require_commit", False)
        self._exceptions = {
            "commit": CommitMetaRequired if require_commit else None,
            "image":  COSAMetaRequired if require_cosa else None,
            "config": COSAMetaRequired if require_cosa else None,
        }
        # Check the required meta-data by calling the properties.
        (_, _, _, _) = (self.commit, self.meta, self.config, self.image)

But why isn't that like statically expressed as some sort of subclass? Aaaah

This reverts commit e0c3296. This broke the `next` stream release at the `cosa aws-replicate` step: coreos/fedora-coreos-streams#496 (comment)

dustymabe · 2022-05-04T03:32:15Z

Successful release job: release#299 ✔️

cverna · 2022-05-04T07:20:01Z

Rollout PR --> #498

gursewak1997 added kind/release jira labels Apr 27, 2022

gursewak1997 assigned cverna Apr 27, 2022

cverna added the ok-to-promote label May 2, 2022

coreosbot-releng mentioned this issue May 2, 2022

[next] tree: promote changes from next-devel at 3b1a6dd07b72c806cb6466c232f1be087aa26d84 coreos/fedora-coreos-config#1709

Merged

cverna changed the title ~~next: new release on 2022-05-03~~ next: new release on 2022-05-03 - 36.20220430.1.0 May 3, 2022

jlebon mentioned this issue May 3, 2022

Revert "Insert generated image.json into the ostree commit" coreos/coreos-assembler#2838

Merged

cverna closed this as completed May 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

next: new release on 2022-05-03 - 36.20220430.1.0 #496

next: new release on 2022-05-03 - 36.20220430.1.0 #496

gursewak1997 commented Apr 27, 2022 •

edited by cverna

Loading

cverna commented May 2, 2022 •

edited

Loading

cverna commented May 3, 2022

cverna commented May 3, 2022

dustymabe commented May 3, 2022

cgwalters commented May 3, 2022

cgwalters commented May 3, 2022

cgwalters commented May 3, 2022

jlebon commented May 3, 2022

cgwalters commented May 3, 2022

dustymabe commented May 4, 2022

cverna commented May 4, 2022

next: new release on 2022-05-03 - 36.20220430.1.0 #496

next: new release on 2022-05-03 - 36.20220430.1.0 #496

Comments

gursewak1997 commented Apr 27, 2022 • edited by cverna Loading

Pre-release

Promote next-devel changes to next

Build

Sanity-check the build

⚠️ Release ⚠️

Run the release job

Refresh metadata (stream and updates)

Housekeeping

cverna commented May 2, 2022 • edited Loading

cverna commented May 3, 2022

cverna commented May 3, 2022

dustymabe commented May 3, 2022

cgwalters commented May 3, 2022

cgwalters commented May 3, 2022

cgwalters commented May 3, 2022

jlebon commented May 3, 2022

cgwalters commented May 3, 2022

dustymabe commented May 4, 2022

cverna commented May 4, 2022

gursewak1997 commented Apr 27, 2022 •

edited by cverna

Loading

cverna commented May 2, 2022 •

edited

Loading