-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parthenon hangs at the end of simulation #1193
Comments
It hangs inside
Full backtrace from lldb: |
Appears to be a Kokkos regression introduced in Kokkos 4.4.0 (also present in Kokkos 4.4.01). If I swap out the current Kokkos submodule for Kokkos 4.3.01, it finalizes successfully. |
@pgrete Maybe we can revert to Kokkos 4.3.01? |
I suspect that this is not a Kokkos regression but sth on our end. Any idea @lroberts36 (as it seems to point to the buffer cache. So before changing/downgrading the Kokkos version, I'd like to spent a little time to check if this cannot be fixed easily in Parthenon itself. |
I just asked on the Kokkos Slack where we should look first. |
Slide 47 in https://github.com/kokkos/kokkos-tutorials/blob/main/Other/ReleaseBriefings/release-44.pdf: "Otherwise, you program may hang when you upgrade to 4.4" <- Does that sound familiar? So it's very likely on us. I won't be able to look at this today. We might be able to coordinate fixing this as part of the hackathon next week (as we're touching stuff around the buffers anyway). |
Observed by @BenWibking on Stampede3 and on a Mac and by myself on a Linux workstation.
Sims run fine and then hang after printing
The last output does also seem to have been written completely.
The text was updated successfully, but these errors were encountered: