Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove hardcoded iteration number from data shuffler #506

Conversation

franzpoeschel
Copy link
Contributor

Until now, the DataShuffler replaces the * pattern with the snapshot number before passing the file name to openPMD. It's better to set %T here so openPMD knows the structure of the filename. With this fix, the output data can be accessed as one data series with openPMD tooling, e.g.

> openpmd-ls Be_shuffled%T.in.h5
openPMD series: Be_shuffled%T.in
openPMD standard: 1.1.0
openPMD extensions: 0

data author: ...
data created: 2024-03-11 11:31:55 +0100
data backend: HDF5
generating machine: unknown
generating software: MALA (version: 1.2.1)
generating software dependencies: unknown

number of iterations: 2 (fileBased)
  all iterations: 0 1 

number of meshes: 1
  all meshes:
    Bispectrum

number of particle species: 0
franzpoeschel:~/git-repos/mala/examples/advanced
> openpmd-ls Be_shuffled%T.out.h5
openPMD series: Be_shuffled%T.out
openPMD standard: 1.1.0
openPMD extensions: 0

data author: ...
data created: 2024-03-11 11:31:55 +0100
data backend: HDF5
generating machine: unknown
generating software: MALA (version: 1.2.1)
generating software dependencies: unknown

number of iterations: 2 (fileBased)
  all iterations: 0 1 

number of meshes: 1
  all meshes:
    LDOS

number of particle species: 0

@franzpoeschel
Copy link
Contributor Author

franzpoeschel commented Mar 11, 2024

The Be_snapshot files in the test-data repository are affected by the same issue:

> openpmd-ls Be_snapshot%T.out.h5
openPMD series: Be_snapshot%T.out
openPMD standard: 1.1.0
openPMD extensions: 0

data author: ...
data created: 2023-05-23 15:13:58 +0200
data backend: HDF5
generating machine: unknown
generating software: MALA (version: 1.1.0)
generating software dependencies: unknown

number of iterations: 4 (groupBased)
  all iterations: An error occurred while opening the specified openPMD series!
Internal error: Group/Variable-based encoding: Parse preference must be set.
This is a bug. Please report at ' https://github.com/openPMD/openPMD-api/issues'.

> openpmd-ls Be_snapshot%T.in.h5
openPMD series: Be_snapshot%T.in
openPMD standard: 1.1.0
openPMD extensions: 0

data author: ...
data created: 2023-05-23 15:13:45 +0200
data backend: HDF5
generating machine: unknown
generating software: MALA (version: 1.1.0)
generating software dependencies: unknown

number of iterations: 4 (groupBased)
  all iterations: An error occurred while opening the specified openPMD series!
Internal error: Group/Variable-based encoding: Parse preference must be set.
This is a bug. Please report at ' https://github.com/openPMD/openPMD-api/issues'.

(Yes, openPMD seems to have insufficient error handling here, will need to fix separately.)
EDIT: This seems to be fixed already on our dev heh

To fix this, can someone tell me how these files are created in the first place?

@RandomDefaultUser
Copy link
Member

I just saw this PR, do you still need input here?

@franzpoeschel franzpoeschel force-pushed the fix-hardcoded-iteration-filename branch from cc794f6 to ea3fbee Compare May 30, 2024 09:13
@franzpoeschel franzpoeschel marked this pull request as ready for review May 30, 2024 09:31
@franzpoeschel
Copy link
Contributor Author

I just saw this PR, do you still need input here?

This is now ready for review

@RandomDefaultUser
Copy link
Member

Looks good to me, thank you!

@RandomDefaultUser RandomDefaultUser merged commit 9f538c7 into mala-project:develop May 30, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants