Adds a C++ parser for our EVENT logs #171

Quincunx271 · 2021-08-24T18:34:17Z

Benefits:

Parses a 4 GB log file in ~ 4GB of memory and 10-12 seconds, compared to an unknown amount of memory (>10 GB) and more than a minute.

Drawbacks:

A lot of code that would have to be maintained...
Uses C++20 and a whole bunch of libraries

WIP - need to: - Benchmark name parsing - File name parsing - RawLog parsing (lazy; we don't want to hold onto the memory unless we need it) - Integration into the Python code

Getting close

Previously, accessing a single element required creating the entire array. Adding these reference types solves that issue.

Stray null bytes

absl::flat_hash_{set,map} are called "swiss tables", and they provide a small perf win for us.

Manually triggered script which uploads the artifact.

This way, by placing all of these modules in a directory on the LD_LIBRARY_PATH, g++-11, libtbb, etc. do not need to be installed to use the Python module, as the needed libraries are bundled here.

Quincunx271 added 13 commits June 16, 2021 12:23

Very fast C++ log parsing

8ba0ed8

A python module that parses the blocks

fdfa748

WIP - need to: - Benchmark name parsing - File name parsing - RawLog parsing (lazy; we don't want to hold onto the memory unless we need it) - Integration into the Python code

Restructure type hierarchy, flesh out types

e810c2f

Getting close

Clean up types

0ab7b0e

Clean up parsing

5d412b6

Accept pathlib.Path to parse

9dc3a75

Expose the C++ parser on the Python side

2ae7a2b

Optimize benchmark and block access

7e84b90

Previously, accessing a single element required creating the entire array. Adding these reference types solves that issue.

Fix .raw_log string decoding error

4463b07

Stray null bytes

Smoothly connect the C++ and Python side of things

b05893f

Revert back to using mmap to read the files

116b12b

Slight perf benefit from using swiss tables

1fd40af

absl::flat_hash_{set,map} are called "swiss tables", and they provide a small perf win for us.

Fix parsing skipped elements

9bb6d0f

Quincunx271 added the utils Concerning utilities rather than the main project label Aug 24, 2021

Quincunx271 added 3 commits August 24, 2021 13:15

Appease clang-format-9

ab11924

Add GitHub script to build python module

cab001e

Manually triggered script which uploads the artifact.

Add `info' member to C++ Benchmark type

a61e637

Quincunx271 force-pushed the feature/c++logparsing branch from a8642dc to a61e637 Compare August 24, 2021 23:13

Bundle the shared objects with the python module

debc241

This way, by placing all of these modules in a directory on the LD_LIBRARY_PATH, g++-11, libtbb, etc. do not need to be installed to use the Python module, as the needed libraries are bundled here.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds a C++ parser for our EVENT logs #171

Adds a C++ parser for our EVENT logs #171

Quincunx271 commented Aug 24, 2021

Adds a C++ parser for our EVENT logs #171

Are you sure you want to change the base?

Adds a C++ parser for our EVENT logs #171

Conversation

Quincunx271 commented Aug 24, 2021