bump Kokkos to version 4.2

[BenWibking]

This updates Kokkos to version 4.2.01.

Kokkos 4.2 fixed bugs in reductions on both CUDA and HIP: kokkos/kokkos#6197

[BenWibking]

@pgrete do you know what happened to the CI here? it didn't even initialize the Docker image...

[BenWibking]

@pgrete It looks like there's something wrong with the driver version on the CI machine:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #1: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
  nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown
  Error: failed to start containers: 3f6b9[27](https://github.com/parthenon-hpc-lab/athenapk/actions/runs/8667014387/job/23769079949?pr=109#step:2:30)4a47dd4451cbdae35c0788d44bf2d412fcd8d8e477d2e131700214a8b
  Error: Docker start fail with exit code 1

[pgrete]

The CI machine needs to reload the cuda driver moduels (currently prevented by a running job). Should be fixed later today.

[BenWibking]

@pgrete Do you have any idea what is causing the MPI failures? They look like this:

[93f25432e875:07073] *** Process received signal ***
[93f25432e875:07073] Signal: Segmentation fault (11)
[93f25432e875:07073] Signal code: Address not mapped (1)
[93f25432e875:07073] Failing at address: (nil)
[93f25432e875:07073] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7fb8e7f78090]
[93f25432e875:07073] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x183bf2)[0x7fb8e80b8bf2]
[93f25432e875:07073] [ 2] /opt/openmpi/lib/libopen-rte.so.40(+0x2d821)[0x7fb8e826f821]
[93f25432e875:07073] [ 3] /opt/openmpi/lib/libopen-rte.so.40(orte_show_help_recv+0x177)[0x7fb8e826fcb7]
[93f[254](https://github.com/parthenon-hpc-lab/athenapk/actions/runs/8667014387/job/23847736958#step:6:255)32e875:07073] [ 4] /opt/openmpi/lib/libopen-rte.so.40(orte_rml_base_process_msg+0x3e1)[0x7fb8e82cd7a1]
[93f25432e875:07073] [ 5] /opt/openmpi/lib/libopen-pal.so.40(opal_libevent2022_event_base_loop+0x7b3)[0x7fb8e81b3f13]
[93f25432e875:07073] [ 6] /opt/openmpi/bin/mpiexec(+0x14a1)[0x55c8789de4a1]
[93f25432e875:07073] [ 7] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fb8e7f59083]
[93f25432e875:07073] [ 8] /opt/openmpi/bin/mpiexec(+0x11fe)[0x55c8789de1fe]
[93f25432e875:07073] *** End of error message ***
"

[pgrete]

Do they happen on init?
I've seen them intermittently, and IIRC they were (under some circumstances) also related to Kokkos 4.2.
Also, have you done any performance test with Kokkos 4.2? AthenaK reverted back after discovering performance issues.

[BenWibking]

Do they happen on init? I've seen them intermittently, and IIRC they were (under some circumstances) also related to Kokkos 4.2. Also, have you done any performance test with Kokkos 4.2? AthenaK reverted back after discovering performance issues.

No, they appear to happen at random timesteps in the middle of each test.

Also, no...maybe this is not a good idea. I will close this PR. Maybe things will stabilize in a few months.