Prefer dnf5 for fedora CI #2742

dschwoerer · 2023-09-13T08:25:42Z

dnf5 is faster, and is sufficiently feature complete for our needs

* support sudo without password * set actually a password * Use --build-args rather then bash script to generate Dockerfile

* Previously the second cmake invocation failed, e.g. after some source changes or after changing a configure time option * It also generated many files, some of which overwrite git-tracked files from autotools, and others where not git-ignored.

Squash optional

Fix some warnings and deprecated headers

Not comprehensive, still fails to run successfully on basic input file (zero pivot in cyclic reduce) Fixes: - unused variables - wrong variable names (including case) - creating laplacians from wrong input sections - some issues with staggering

MPI_DOUBLE is a macro, so this check can break, even if there is no mismatch.

Fix versions after release

Should be faster and more consistent

ZedThree · 2023-09-20T14:32:46Z

Both OpenMPI and MPICH are failing with "bus error", but OpenMPI gives a bit more of a clue:

It appears as if there is not enough space for /dev/shm/vader_segment.fd7fd1d78666.1000.c41c0001.0 (the shared-memory backing
file). It is likely that your MPI job will now either abort or experience
performance degradation.

Bus errors can apparently happen due to not having enough physical memory for requested mallocs. The VM specs are:

2-core CPU (x86_64)
7 GB of RAM
14 GB of SSD space

which I would've thought would be plenty.

OpenMPI has a stack trace too:

[fd7fd1d78666:03983] Signal: Bus error (7)
[fd7fd1d78666:03983] Signal code: Non-existant physical address (2)
[fd7fd1d78666:03983] Failing at address: 0x7f04ac0ce000
[fd7fd1d78666:03983] [ 0] /lib64/libc.so.6(+0x3e990)[0x7f04b5c5a990]
[fd7fd1d78666:03983] [ 1] /lib64/libc.so.6(+0x16f8ba)[0x7f04b5d8b8ba]
[fd7fd1d78666:03983] [ 2] /lib64/libfabric.so.1(+0x625500)[0x7f04af005500]
[fd7fd1d78666:03983] [ 3] /lib64/libfabric.so.1(+0x60d1be)[0x7f04aefed1be]
[fd7fd1d78666:03983] [ 4] /lib64/libfabric.so.1(+0x611307)[0x7f04aeff1307]
[fd7fd1d78666:03983] [ 5] /lib64/libfabric.so.1(+0x693880)[0x7f04af073880]
[fd7fd1d78666:03983] [ 6] /lib64/libfabric.so.1(+0x5f1080)[0x7f04aefd1080]
[fd7fd1d78666:03983] [ 7] /usr/lib64/openmpi/lib/openmpi/mca_btl_ofi.so(+0x6f83)[0x7f04ae7dbf83]
[fd7fd1d78666:03983] [ 8] /usr/lib64/openmpi/lib/libopen-pal.so.40(mca_btl_base_select+0x126)[0x7f04b40e19f6]
[fd7fd1d78666:03983] [ 9] /usr/lib64/openmpi/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x1e)[0x7f04af1cd2fe]
[fd7fd1d78666:03983] [10] /usr/lib64/openmpi/lib/libmpi.so.40(mca_bml_base_init+0x99)[0x7f04b6262ec9]
[fd7fd1d78666:03983] [11] /usr/lib64/openmpi/lib/libmpi.so.40(ompi_mpi_init+0x60b)[0x7f04b62a030b]
[fd7fd1d78666:03983] [12] /usr/lib64/openmpi/lib/libmpi.so.40(MPI_Init+0x72)[0x7f04b623f462]
[fd7fd1d78666:03983] [13] /home/test/BOUT-dev/build/lib/libbout++.so.5.0.0(_ZN8BoutComm7getCommEv+0x2c)[0x7f04b821422c]
[fd7fd1d78666:03983] [14] /home/test/BOUT-dev/build/lib/libbout++.so.5.0.0(_ZN8BoutComm4rankEv+0x9)[0x7f04b8214309]
[fd7fd1d78666:03983] [15] /home/test/BOUT-dev/build/lib/libbout++.so.5.0.0(_Z14BoutInitialiseRiRPPc+0x2da)[0x7f04b7f9da2a]
[fd7fd1d78666:03983] [16] ./2fluid[0x40f44d]
[fd7fd1d78666:03983] [17] /lib64/libc.so.6(+0x2814a)[0x7f04b5c4414a]
[fd7fd1d78666:03983] [18] /lib64/libc.so.6(__libc_start_main+0x8b)[0x7f04b5c4420b]
[fd7fd1d78666:03983] [19] ./2fluid[0x40f585]

which looks to be basically the same for each test failure. You can see it's crashing in a call to BoutComm::getComm inside BoutInitialise, so this is very early one before we've allocated really anything. So either something is already using most of the VM's RAM before we even start, or there's some other issue. I'll give it a whirl locally

dschwoerer · 2023-09-20T14:39:42Z

I tried it a bit, and I can reproduce this when I run in a container, without a tty.
With a tty it does work (I think mpich gets stuck somewhere) but the bus error is only present without a tty.

That of course adds to the fun of debugging ...

ZedThree · 2023-09-20T15:02:14Z

Ouch. Was that just a matter of running a shell in the container and then running the tests, vs running the tests directly via the container?

ZedThree · 2023-09-20T15:05:28Z

Running ./.ci_fedora.sh locally I get a completely different error:

[684b4db46971:02206] mca_base_component_repository_open: unable to open mca_pmix_ext3x: /usr/lib64/openmpi/lib/openmpi/mca_pmix_ext3x.so: undefined symbol: pmix_value_load (ignored)
[684b4db46971:02206] [[27020,0],0] ORTE_ERROR_LOG: Not found in file ess_hnp_module.c at line 320

ZedThree · 2023-09-21T14:15:02Z

I'm at a bit of a dead end here, MPICH in the container works locally for me. OpenMPI doesn't, but I think that's a bug with Fedora, and it has a different failure to that in CI.

I don't really have time to debug this further, so I think we might have to just disable the Fedora CI for the time being until someone has time to investigate and fix it.

dschwoerer · 2023-09-22T12:36:59Z

This is the error you probably saw:
https://bugzilla.redhat.com/show_bug.cgi?id=2240042 (undetected ABI break in a dependency, a rebuild may fix this one)

Once that is fixed the other error might return, I can go about debugging it.

dschwoerer · 2023-10-06T08:55:16Z

Should be fixed now. Failing run was PETSc #2755

Issue was caused by libfabric, downgrading resolves the issue for now.
Reported as https://bugzilla.redhat.com/show_bug.cgi?id=2242447

ZedThree · 2023-10-06T09:28:37Z

This isn't really a blocker for 5.1, so I'll wait for the CI to finish on master and re-release

ZedThree · 2023-10-06T10:11:00Z

Fedora builds finally working again, merging this into next

ZedThree · 2023-10-06T10:11:49Z

🤦 Need to wait till we've actually merged master into next first

dschwoerer · 2023-10-09T06:56:41Z

Shouldn't the fixes / improvements for CI also go to master?
The fixes for fmt should go into 5.1.1 as well?

dschwoerer · 2023-10-10T15:12:08Z

@ZedThree Any ideas why this are still 200 commits?

dschwoerer · 2023-10-13T07:17:49Z

Replaced by #2768 vi #2764

dschwoerer · 2023-10-13T07:18:52Z

Replaced by #2764 and #2768

dschwoerer and others added 30 commits February 28, 2022 13:34

Build containers in GHA CI

0732af0

Improve docker CI

d903927

* support sudo without password * set actually a password * Use --build-args rather then bash script to generate Dockerfile

Only generate if file is executable

eb7f84a

CI: Nicer name for builds

9a330ab

CI: Also build an image with utils installed

b7dfa17

[skip ci] Apply black changes

b415d37

make executable dir optional

580e784

Fall back to using $PATH to find bout-squashoutput

8b76a8f

Error early with in-source-builds

8cdba94

* Previously the second cmake invocation failed, e.g. after some source changes or after changing a configure time option * It also generated many files, some of which overwrite git-tracked files from autotools, and others where not git-ignored.

Merge pull request #2659 from boutproject/squash-optional

1230989

Squash optional

Declare variable as unused to avoid warning

36cabdf

Avoid no-return warning in pvode

2af404d

Avoid PVEC_REAL_MPI_TYPE redifinition warning

e593a99

Avoid iter not used warning

725bb74

Apply clang-format changes

a057a65

Allow in-source-builds, but recommend against

668de4d

Fix: highlight code

636eb56

Fix: all boutpp members are imported

ba77dd7

Fix more references that should be code

fd35030

Fix: code needs to be separated by an empty line

87fa772

RTD: Add more modules for mock

88956a0

Fix some warnings in output header

e3ce78b

Fix deprecated header includes

d60a5a2

Merge pull request #2666 from boutproject/fix-bad-merge

6e6f09f

Fix some warnings and deprecated headers

Merge branch 'master' into output-warning

b52812d

Some fixes for tokamak-2fluid example

4c05600

Not comprehensive, still fails to run successfully on basic input file (zero pivot in cyclic reduce) Fixes: - unused variables - wrong variable names (including case) - creating laplacians from wrong input sections - some issues with staggering

Apply clang-format changes

e004276

Do not check for PVEC_REAL_MPI_TYPE change

f1b2e04

MPI_DOUBLE is a macro, so this check can break, even if there is no mismatch.

Fix versions after release

bee8eeb

Merge pull request #2669 from boutproject/master-version

754b601

Fix versions after release

Switch to fedora packes

a573c82

Should be faster and more consistent

dschwoerer dismissed ZedThree’s stale review via a573c82 September 13, 2023 12:32

CI: switch to mpich to workaround openmpi issue

7d3fa7c

dschwoerer added 2 commits September 22, 2023 14:57

Switch back to openmpi

2b4cc3e

Also give a backtrace if we are hitting a bus error

1ec4255

dschwoerer mentioned this pull request Oct 6, 2023

Add fallback if the version has been bumped bot not tagged #2754

Merged

dschwoerer added 3 commits October 6, 2023 09:14

Try disabeling PETSc

2aa8a16

CI: Reenable petsc

63f8243

CI: Use more system packages

6651add

dschwoerer added 2 commits October 6, 2023 11:18

Do not try to print KSPConvergedReason

320b88c

Cast reason to int for formatting

12ef939

Apply clang-format changes

3184fe1

ZedThree changed the base branch from v5.1.0-rc to next October 6, 2023 10:11

dschwoerer mentioned this pull request Oct 9, 2023

fmt 10 #2729

Closed

dschwoerer closed this Oct 13, 2023

dschwoerer deleted the dnf5 branch October 13, 2023 07:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prefer dnf5 for fedora CI #2742

Prefer dnf5 for fedora CI #2742

dschwoerer commented Sep 13, 2023

ZedThree commented Sep 20, 2023

dschwoerer commented Sep 20, 2023

ZedThree commented Sep 20, 2023

ZedThree commented Sep 20, 2023

ZedThree commented Sep 21, 2023

dschwoerer commented Sep 22, 2023

dschwoerer commented Oct 6, 2023

ZedThree commented Oct 6, 2023

ZedThree commented Oct 6, 2023

ZedThree commented Oct 6, 2023

dschwoerer commented Oct 9, 2023

dschwoerer commented Oct 10, 2023

dschwoerer commented Oct 13, 2023

dschwoerer commented Oct 13, 2023

Prefer dnf5 for fedora CI #2742

Prefer dnf5 for fedora CI #2742

Conversation

dschwoerer commented Sep 13, 2023

ZedThree commented Sep 20, 2023

dschwoerer commented Sep 20, 2023

ZedThree commented Sep 20, 2023

ZedThree commented Sep 20, 2023

ZedThree commented Sep 21, 2023

dschwoerer commented Sep 22, 2023

dschwoerer commented Oct 6, 2023

ZedThree commented Oct 6, 2023

ZedThree commented Oct 6, 2023

ZedThree commented Oct 6, 2023

dschwoerer commented Oct 9, 2023

dschwoerer commented Oct 10, 2023

dschwoerer commented Oct 13, 2023

dschwoerer commented Oct 13, 2023