Bump Parthenon to latest develop (2024-02-15)

[forrestglines]

Update Parthenon to PR930

[forrestglines]

@par-hermes format

[forrestglines]

Actually this causing issues for the cluster set up, do no merge

[pgrete]

FYI I just bumped to current develop in Parthenon. Let's see how this works.

[BenWibking]

I'm a bit confused as to why this is merging into the donut branch. The main branch doesn't have these changes either, does it?

Is this PR still relevant, or has it been superceded?

[pgrete]

I'm a bit confused as to why this is merging into the donut branch. The main branch doesn't have these changes either, does it?

Is this PR still relevant, or has it been superceded?

Yes, the PR is still relevant and I was putting it on hold while tracing back the issue I saw on Lumi.
I ran further tests earlier today and while I'm reliably able to reproduce errors like

cycle=0 time=0.0000000000000000e+00 dt=3.5156250000000001e-03 zone-cycles/wsec_step=0.00e+00 wsec_total=2.43e-02 wsec_step=8.55e-01
Memory access fault by GPU node-5 (Agent handle: 0x82126c0) on address 0x14e1b1300000. Reason: Unknown.
srun: error: nid005016: task 1: Aborted

with the Parthenon advection example on just two ranks (not showing up on a single ranks), I was neither able to reproduce locally nor on Frontier.
Moreover, the core dump points to some /opt/rocm/lib/libhsa-runtime64.so.1 libs.
This gives me reason to believe that there's an issue with the software stack on Lumi and that's it not an issue on the Parthenon side.

Bottom line: Let me clean up this PR (tomorrow) and then we can merge with the latest Parthenon develop in place.
Does this sound like a plan?

[BenWibking]

I think the core dump will point to that because that's where the error handler is, not necessarily because the underlying bug is there.

(We've been debugging an issue in Quokka that produces that error message. We thought it was a out-of-bounds memory access, which it usually is. However, It turned out to be a compiler bug that appears for specific ROCm versions 🙃)

Yes, that plan sounds good to me.

[BenWibking]

@pgrete just want to check if this is ready to go?

[pgrete]

@pgrete just want to check if this is ready to go?

I double checked on Friday that the (other) crashes I observed are not reproducible any more. They were also not reproducible with the binary I triggered them in first place and given that those were all interconnect related issue, I assume that the machine was unstable by chance when I ran the original tests. (Note this was a second error mode apart from the Memory access fault by GPU above).

I now updated to latest Parthenon develop and fixed all interface changes fingerscrossed.
I also took this as opportunity to introduce a CHANGELOG (where I highlighted the changes in the Parthenon API).

Bottom line: this is ready for review and merge @BenWibking @forrestglines

[BenWibking]

There appears to be some unrelated cluster setup changes (although they are trivial). Everything else looks good to me.

[pgrete]

There appears to be some unrelated cluster setup changes (although they are trivial). Everything else looks good to me.

Yes, I added those (to be used for the histograms).
In general, I'd like to separate the "derived fields" into a more central location as many fields are reusable across problem generators.

[BenWibking]

@forrestglines Did the cluster test runs with this PR work?

[forrestglines]

@forrestglines Did the cluster test runs with this PR work?

Yes -- a Perseus test cluster did not cause any issues when I think it did before. I think this is safe to merge in

[BenWibking]

@pgrete do you want to merge this now?