In-memory MDF

[jonschumacher]

I am currently working on an implementation of an in-memory MDF.

Missing things:

• Fix error with high relatively deviation
• Add working conversion from and to dicts including tests -> Add direct conversion (see non-optional field /measurement/isSparsityTransformed missing when generating mdf #34)
• Add tests for data integrity functions
• Add documentation

Since I strictly enforce a "missing" on every non-optional key combined with checking data integrity against the MDF specs, this PR should also stop issues like #34 turning up.

@tknopp: I don't really get the failing test. The two system matrices are equal but the test fails anyways. I will further investigate this next week, but if you have a spontaneous idea, I would be glad to hear it.

[tknopp]

I have no spontaneous idea. Might be something with the transfer function.

Need to review the PR in more detail but on a first sight this all looks pretty good. The validation of MDF files is something we wanted for a long time. My feeling is that the entire conversion infrastructure should work on the inMemory MDF and we might then want to give up the dict infrastructure. I suppose that this is the plan anyway since we want the inMemory type to be used directly in MPIMeasurements.

[jonschumacher]

I solved the issue of the failing test. This line

MPIFiles.jl/src/MDF.jl

Line 378 in 4842a98

blockLen = measBGFrameBlockLengths( invpermute!(measIsBGFrame(f), measFramePermutation(f)) )

had a side-effect which I solved using deepcopy(). I hope this is not a big performance issue. By the way: Why is there

MPIFiles.jl/src/MDF.jl

Line 342 in 4842a98

function systemMatrix(f::MDFFileV2, rows, bgCorrection=true)

and

MPIFiles.jl/src/SystemMatrix.jl

Line 13 in 4842a98

function getSystemMatrix(f::MPIFile, frequencies=nothing;

. They seem to do partly the same.

I hope, I considered everything and this is now ready to merge.

[tknopp]

perfect Jonas. Give me please some time to let me go over the PR and then let me merge it.

Regarding getSystemMatrix: This definitely needs some cleanup and one can see that the code evolved over time. We probably want to get rid of getSystemMatrix. But on the other hand getSystemMatrix is useful for all MPIFile implementations while systemMatrix aims to isolate the type specific part. So it seems we need both but I agree that the function names are really confusing right now. Maybe systemMatrix for the public interface and systemMatrixInternal or systemMatrixLowLevel for the internal interface.

[tknopp]

This looks overall very good. I testet MPIReco which also still runs fine and thus merge this now.

[tknopp]

@jonschumacher please delete the branch on your own if you don't need it anymore.

[tknopp]

regarding all the smaller hacks with existing MDF files. In the long rund we should just fix the issues in the underlying test files. It seems that only the MDF1 files are problematic here.

Regarding the interface I found it interesting that the in-memory version was initialized from the lazy one. While this definitely makes sense internally we might want to make the an interface where one creates the in-memory version from a filename. This would also imply that the HDF5 handle is directly closed. Or could be explicitly closed without the need to wait for the garbage collector (we have some issue with HDF5 files being closed too late due to garbage collection being asynchrone and non-deteministic).