Reading Pilgrim traces

[khatharsis42]

From what I understand, currently, Pilgrim does not provide any way to read traces other than using pilgrim2text, and then parsing the generated .txt files.
Is it planned to provide a library one could use to read Pilgrim's trace on their own, and / or a documentation for enabling easier trace reading ?

[wangvsa]

Yes, currently Pilgrim only includes a pilgrim2text tool for converting Pilgrim traces into .txt files. I agree It would be beneficial to have a dedicated library for parsing Pilgrim traces. Is there any specific information you're looking for? I can help developing the library and creating the documentation on the trace parsing. However, please be aware that my availability for this project is limited.

[khatharsis42]

We were thinking of developing a sort of profiler for Pilgrim Traces, to provide statistical analysis of the different events quickly and without having to read the whole trace. We were thinking this could be used to generate communication matrices at a very low cost, for example.
If such a library does not exists, may I ask how you generated the communication matrices in your paper "Near-Lossless MPI Tracing and Proxy Application Autogeneration" ? Did you simply use the pilgrim2text tool and then read the .txt generated, or did you use the structural nature of the Pilgrim traces ?

[wangvsa]

I have some internal reader code to read/decompress the Pilgrim traces. Please take a took at pilgrim_reader.h. The implementation is in src/decoder. I don't currently have a good documentation on the reader code, but I think it is rather easy to follow. Also, I'm glad to help writing more docs on it and help you parsing the traces.