ADIOS2 Append Mode

[franzpoeschel]

Our READ_WRITE mode is not quite adequate for use in ADIOS2, since ADIOS2 only allows either reading or writing, but our READ_WRITE mode's workflow needs that both are working (#996). However, Append mode exposes an interface equivalent to that of the Write mode, but steps already on disk are not deleted, new ones are just appended. Hence, add a new Access::APPEND that unlike READ_WRITE does not allow any reading, but does not overwrite anything either.

• Simple proof-of-concept implementation
• Test all possible edge cases:
  • Since there is no reading involved, it's on the user to use the exact same configuration of openPMD in append mode as had been used for the appended-to file.
  • This is now testing new and old ADIOS schema, group- file- and variablebased encoding
  • Overwriting old iterations is still a work-in-progress, non-linear read patterns and truncation of files upon opening are to come in a later PR
• Keep possible a workflow where in file-based iteration encoding, already existing iterations can be read and new ones can be written
  The READ_WRITE mode currently does this. I would suggest to use new Append mode to support this more properly. Will need some tweaks in the frontend though where APPEND mode is currently treated as equivalent to WRITE mode.
  Idea: In filebased iteration encoding, coerce the backendAccess to CREATE. Any written iteration will then be overwritten in doubt.
• Bring this to the other backends. This should become the new default READ_WRITE mode light, unless you really need to read data.
• Discussion point: Since reading does not work at all in this mode, I don't see any way to guarantee that an update to a dataset is written with the same overall settings (schema, usage of steps, ..). Users will need to make sure to create consistent datasets...
• Discussion point: Checkpoint-restarting. Probably one of the main uses of this. Dump every 100 time steps, checkpoint every 1000, crash at 1500, restart from 1000. Consequence: Steps {1000,1100,…,1500} are written twice. See Mapping between ADIOS steps and openPMD iterations #949 for an exploration on how to deal with this, but it's not really ideal.
  Update: See Truncate option for Engine Open mode Append ornladios/ADIOS2#2775
  Will still not be trivial to implement, since a preceding read access might be necessary to figure out how many steps should be dropped. Would also require to specify from which iteration to restart writing.
• Documentation page on access modi
• File-based appending in ADIOS2: Throw error if trying to append to already written iteration?
  Update: Given the proposed workflow with Truncate option for Engine Open mode Append ornladios/ADIOS2#2775, it would be most consistent to just overwrite iterations. Maybe use Access::CREATE again for the backendAccess.
• ADIOS1 UPDATE: OperationUnsupportedInBackend
• Merge Add header for openPMD-specific errors #1080 first
• Document altered meaning of createFile task

Also, this introduces a new Error.hpp, so I don't need to again throw std::runtime_errors around.

[ax3l]

Checkpoint-restarting. Probably one of the main uses of this. Dump every 100 time steps, checkpoint every 1000, crash at 1500, restart from 1000. Consequence: Steps {1000,1100,…,1500} are written twice. See #949 for an exploration on how to deal with this, but it's not really ideal.

I think useful would be a mode that overwrites a step exactly when it is re-opened again, but leaves older (and even newer) steps as they are. That's what people most expect and know from file-based output: files stay around until you overwrite one by one.

Since output from checkpoint-restart should give again the same result as the prior run, this is also ok (no "invalid" older files, so to say).

Alternatively, one could at a certain point (e.g. once during re-open/append) delete all steps after the restart step.

[franzpoeschel]

I think useful would be a mode that overwrites a step exactly when it is re-opened again, but leaves older (and even newer) steps as they are. That's what people most expect and know from file-based output: files stay around until you overwrite one by one.

That is possible to implement with file-based iteration encoding and APPEND mode, but it won't be possible with either group- or variable-based encoding. Adding steps to an existing file in ADIOS2 allows no reading or modifying of old steps, you're just appending to a BP file without caring what had previously been written to it.

The problem is not file-based encoding, but variable-/step-based encodings with steps enabled for output data after loading a checkpoint.

[franzpoeschel]

I have now added some handling in the backend so that this isn't necessary any more. Could remove it.

[franzpoeschel]

Update: Eh, let' keep it. Clean overwriting is better and in file-based encoding it's easy to achieve.

[ax3l]

The problem is not file-based encoding, but variable-/step-based encodings with steps enabled for output data after loading a checkpoint.

Uff, right. I put this on the ADIOS2 agenda. I think we could try to gracefully warn and skip such outputs in user-code if they exist, but definitely have to think about a workflow.

[franzpoeschel]

I quickly deleted the member and looked where compilation failed. It seems that there currently is no other bug of the sort. wrong PR

[ax3l]

Discussion update:

• RFE with ADIOS2: Append mode, with parameter to truncate after the Nth (ADIOS) step: Truncate option for Engine Open mode Append ornladios/ADIOS2#2775
• high-level solution: using the openPMD “step” variable to list the same step again and our logic picks up the “latest” one in reads (will work, but accumulates data if steps are written multiple times)

[franzpoeschel]

I think this is ready for review @ax3l
Note that there is further info found in the added documentation, otherwise I have tried to split this into a sane number of commits.

[franzpoeschel]

I've done further cleanup now. If it runs green, ready for review @ax3l

[ax3l]

@franzpoeschel please needs a rebase. Collided with #1264