Discussion: PyDarshan and libdarshan-util interactions, accumulator interface

[shanedsnyder]

Problem
PyDarshan CFFI bindings convert the record pointers returned by libdarshan-util C library into Pythonic data structures (dictionaries, numpy arrays, pandas dataframes), that cannot be easily passed back into other libdarshan-util routines. That opens up questions on how to generally handle a couple of things in PyDarshan:

1. How (if at all) should we support passing instrumentation records back to darshan-util interfaces? We noticed this issue when trying to use accumulator interface, but would also affect PyDarshan's potential use of other interfaces (like the log writing interface).
2. Depending on above, where should we handle the aggregation and calculation of derived statistics for sets of Darshan records?

Aggregation Use Cases

1. Accumulation of all records in a module in a log to obtain aggregate stats and derived stats
   • E.g., in the Darshan job summary module summary sections
2. Accumulation of records all referencing the same file in a log to obtain a single aggregate file record
   • E.g., the deprecated --file-list option from darshan-parser which provided a single record for every single file accessed by the app
3. General accumulation of file records, either from the same log or from multiple logs
   • Could be used to obtain aggregate stats and derived stats from a set of file records matching a given pattern in a single log, e.g.:
     • Derived POSIX module stats for all file names ending in '.h5'
   • Could be used to obtain aggregate stats and derived stats for file records from multiple Darshan logs, e.g.:
     • Running a --file-list style aggregation across multiple Darshan logs from a workflow
     • Aggregate statistics for all Darshan logs in a given directory

Solutions:

1. Avoid passing records back into libdarshan-util at all, and implement all helper logic (e.g., aggregation) at the Python level
   • This is essentially what we do now in the job summary tool, with some Python (pandas?) code manually aggregating specific counters of interest (e.g., access size histograms, operation counts)
   • Pros:
     • Avoids complexity by minimizing interactions with the C library to only opening and reading/writing log files
   • Cons:
     • I don't think we are in a place to fully move all darshan-util to PyDarshan, even if it sounds nice in theory. Given that, we would almost certainly have to be duplicating critical logic in the C lib and in PyDarshan, which is what we hoped to avoid when adding the accumulator API
     • Existing aggregation code in PyDarshan is not comprehensive, it mostly supports just what is needed to generate plots for Darshan job summary tool -- the aggregation code in the C library aggregates every single counter for a given record format (although it's only implemented for POSIX, MPI-IO, and STDIO as of now)
     • PyDarshan records support multiple formats (pandas, numpy, dict), so presumably this logic needs to be implemented for each, or otherwise we need to commit to a format?
2. Add logic for converting Python representation of records back into format expected by C library (i.e., packed struct)
   • See Tyler's PR for an early attempt at this: WIP, ENH: add df->rec converter #886
   • Pros:
     • Allows use of existing libdarshan-util APIs for any types of PyDarshan record representations, whether they were originally read from libdarshan-util or perhaps even generated synthetically in Python
   • Cons:
     • PyDarshan records support multiple formats (pandas, numpy, dict), so presumably this logic needs to be implemented for each, or otherwise we need to commit to a format?
     • Extra conversion step, even if not super expensive
3. Rework CFFI bindings and DarshanReport API to support accumulation of records before they are converted to Python format
   • This can be viewed as an extension to Tyler's PR for supporting the accumulator interface (ENH: Python derived/accum interface #839), that accumulates as records are read (i.e., all in one pass over the log) and uses data from the accumulator API for providing input to job summary plots (rather than current piecemeal PyDarshan aggregation code).
     • Users can specify what types of accumulations to automatically run (per-module, per-file, per-pattern) when creating a DarshanReport object?
   • Pros:
     • No need for extra record conversion steps, we accumulate using raw records read from libdarshan-util
     • Not tied to a specific Python format (pandas, numpy, dict)
   • Cons:
     • More general usage of accumulator logic (i.e, something more involved than aggregating every record for a given module) may require more complex APIs, or perhaps not be possible.

[shanedsnyder]

For my 2c, I think I'd probably rule out Solution 1 entirely but included it for discussion purposes. I would probably prefer Solution 3 as the immediate path forward (and I could provide more details and/or start hacking on a PR there if its helpful to consider some code), but am open to discussion. I think I prefer Solution 3 as it allows us offer accumulation as a first-class feature of the DarshanReport interface and doesn't rely on standing up additional (unnecessary) logic for converting back from Python data representations, as in Solution 2.

[tylerjereddy]

On my side, I think only solution 1 makes sense long-term, and I'm pretty strongly opposed to the other approaches long-term because it is just an absolute disaster to deal with the error handling/memory management at the C level as a developer, and because we really should start to try to shed our dependence on the in-house binary format idiosyncracies as much as possible.

I think the C library code should really serve just 1 purpose--get the the data out of the custom binary darshan data format as quickly as possible and into the rich Python data analysis ecosystem format as soon as possible. We should just be using DataFrames to store the data, and interchange with C/C++ analysis codes should just happen through the standard Python ecosystem buffer interfaces used in so many other places. CFFI should probably be removed in favor of Cython if the C developers want to continue writing at the C layer, so that we can direclty pass NumPy arrays/memoryviews to/from C as needed, but I'd anticipate that most sane analyses should happen using pandas/SQL-style operations on the rows and columns of data. CFFI may have been a tempting short-term patch, but the amount of time I've spent dealing with memory leaks and segfaults through the opaque CFFI interface has been non-trivial.

To achieve this, I think objective 2 is fine in the short term so the reports can be made to match the Perl reports and an overhaul doesn't interfere with sunsetting the Perl reports. That said, once objective 2 is in place, I think writing thorough Python tests for the aggregation functionality should be done over time, until we reach a point where we can sunset usage of the C bindings for that. I don't really care if that takes a few years for example.

Most importantly, and likely controversially, I don't think we should ever write raw C code that operates on the record structs for analysis again (no new code there)--if the aggregation stuff stays around for a few years to avoid rewriting, "fine for now" think, but I think it is just a poor use of resources to neglect the power of the ecosystem unlocked by NumPy/pandas and so on to be writing raw C routines in pretty much any new analysis cases. For things like machine learning/feature extraction and so on, I think this is even more obvious.

While I know (or hope) you're not proposing to write your own ML code in raw C at this point, I do think that whatever you do in this particular case should be the last case of supporting analysis code written in C to operate on the in-house record formats, and future analysis should be entirely confined to operating on Python-exposed data structures and Cython-level stuff on standard buffer formats if you really need C (but that should be rare).

I also think is not really fair to characterize needing to write the Python algorithms three times for NumPy, pandas, and dictionaries separately. You can pull a NumPy datastructure directly from a pandas DataFrame anytime, so that's a bit unreasonable of a requirement. Dictionaries should only be housing some of the larger organizational stuff leftover from Jakob's efforts, but nothing should be doing data munging on dictionaries directly for array operations...

[carns]

A fourth option for discussion. (I'm not sure this is a good idea either, but it seems like it warrants discussion) For some motivating context, conceptually it doesn't seem like it is supposed to be that hard for us to mix and match C and Python in cases where one or the other is more appropriate for some reason, or even mix and match other toolkits for that matter. From that perspective, maybe the problem is that we are using programmatic APIs as the interface rather than an agnostic data format and schema that would let different things access/manipulate/exchange darshan data without so much direct coupling and translation.

The current in-memory format is always an array of sometimes-large C structs. Analysis code in any language therefore has to interpret C structs and be able to invoke native C functions that don't necessarily match it's memory model.

The most popular language-agnostic format for this kind of thing (to my knowledge) would be Apache Arrow (https://arrow.apache.org/). The description on their page is better than I can paraphrase. It has two types of C bindings (one that just gives you fairly interpretation of data (https://arrow.apache.org/docs/format/CDataInterface.html) but can be embedded in a code base like libdarshan-util, and one that is an external dependency with a far broader scope (https://arrow.apache.org/docs/c_glib/). The python side has all of the trappings that you would expect, including python-native conversion to dataframes (https://arrow.apache.org/docs/python/pandas.html).

We'd have to play with it to see if it does what it says on the tin, but adopting a language-neutral in memory data format might make it easier to use capabilities in both languages a little more easily.

The C library could possibly emit and/or ingest arrow-formatted in-memory data on the fly, or we could consider a bulk conversion to a persistent file in parquet in-file format (https://parquet.apache.org/).

[carns]

Also I agree that we need something short term based on methods we have that already (almost) work, but it seems like a good time while this is on our mind to think about long term strategy too.

[shanedsnyder]

Solution 1 is predicated on some pretty substantial changes, as it basically requires rewriting of nearly all of darshan-util components (sans log reading/writing, by my understanding) in Python -- otherwise we find ourselves maintaining both traditional C tools and new PyDarshan stuff. That's obviously impractical in the short term, maybe even in the longer term when considering funding/manpower.

Even so, what exactly does the aggregation code look like at the Python level? As of now, the aggregation code can be optionally implemented when defining a module's logutil handlers in the darshan-util C library, which makes it an obvious part of a modules interface with Darshan's libraries. I won't argue the code is pretty or anything:

darshan/darshan-util/darshan-stdio-logutils.c

Line 393 in b9043e1

static void darshan_log_agg_stdio_records(void *rec, void *agg_rec, int init_flag)

But I really can't imagine how the pandas code would be any simpler for this. As a comparison, consider some of the example aggregation code in PyDarshan: https://github.com/darshan-hpc/darshan/blob/main/darshan-util/pydarshan/darshan/experimental/aggregators/agg_ioops.py

This code is not comprehensive, it only aggregates portions of the record which are really straightforward sums (operation counts, access size histograms). This pandas code will require detailed knowledge of the C representation of instrumentation records (different counter types use different aggregation operations), which I think also just creates more disconnect as that record representation is maintained in the C library. Just sounds like a mess, but perhaps I could be open minded if I knew exactly what this Python code would look like.

Maybe just moving forward with Solution 2 in the short term is the least controversial way to get to a functional job summary tool, and gives us more time to come up with a longer term solution? It offers us the most flexibility to aggregate basically any records together now, with perhaps something like @carns suggestion for Solution 4 ultimately helping define a common memory format for darshan-util/PyDarshan. If we do eventually PyDarshan all of the things, then we will still need some solution to memory interoperability between Python and C as the log writing functionality will require that. In the meantime, Solution 2 can probably provide the stopgap.

[tylerjereddy]

I'm not sure I understand the long-term issue--are you claiming that darshan-util performs analysis so complex it can't be reasonably represented using standard algorithms on rectangular data structures? That seems like a bit of a stretch, but I can understand it may take a long time to rewrite it and not be a priority to do so. I'm certainly not going to read the entirety of the C aggregation code and then redesign it at the Python layer in the short-term (it sounds like we all agree on that point at least).

But I really can't imagine how the pandas code would be any simpler for this.

There's no manual memory management, nice error tracebacks, segfault should be effectively impossible, there are multi-threaded + GPU backends for acceleration if that's ever desired, it is likely easier to onboard new folks/students to maintain if it is composed in standard scientific Python, etc. Is there really any debate that C code is easier maintain/read here? I agree that Python isn't magic though, and at the of the day the logic should be fundamentally the same, perhaps with some more concision in places, but perhaps not. It would likely have a more "vectorized" flavor with database-like row/column ops replacing some manual loop logic.

A lot of awkwardness in the current Python code results from bending to the will of the heterogeneous structs underneath, so I wouldn't really trust the current state of any pydarshan code as representative of what things would look like when unchained from that. How do you design it? I don't know for sure, but we'll never know if we just stick with "it is so complicated it can't reasonably be done in Python." If the long-term core team isn't in favor, there probably isn't much point--newer Python devs would need a lot of cooperation to achieve an unchaining from all those in-house structs.

One other issue I had when working on the Python side is that some of the total counters in the report proper are produced in non-obvious ways, which just seems like another issue; if the way the numbers are produced is too obscure for a developer to understand, what hope does a user have? If the various aggregation metrics and algorithms are described and documented in clear terms, that would probably benefit users and future developers for code modernization. That takes time too though.

Who would do all the work for migrating aggregation stuff--probably nobody anytime soon. OTOH, for any new analyses/report features I'd personally like to see that scoped to the Python side at this point. So, for example there might be a new feature in the report that could be a bit shorter to deal with if you just add another C function to darshan-util, but I'd like to see that handled at the Python level to start building up the capabilities on the Python side, otherwise it'll probably never happen and you also probably don't need much Python dev bandwidth (well, I suppose that is a + for me..).

I guess, now that I think of it, if you do go the route of basically doing everything in C and only need to expose structs members via CFFI to the HTML reports, that could probably be handled with an occasional summer student, which would certainly save me a lot of bandwidth (though admittedly, this is not particularly interesting for me to work on).

[carns]

I need to think about the broader issue, but I just wanted to quickly comment that I am in complete agreement with "any new analyses/report features I'd personally like to see that scoped to the Python side at this point". We haven't added any new C analysis code for the sake of new analysis capability in several years (more recent analysis efforts were almost all done ad hoc in python outside of the official darshan repo by ingesting and parsing text format counters: painfully slow, but better than trying to do agile analysis in C).

The accumulator API is new for sure, but not because we were creating new capability that didn't exist before. It already existed, but was hard coded in the darshan-parser command line utility where it couldn't be reused. Refactoring it out into a library API reduced the size of darshan-parser.c, fixed a couple of bugs that had crept in due to duplicate implementations of some elements in the C code, made it all unit-testable, and was intended to make the same logic available to PyDarshan. I don't think anyone has an appetite for new or more advanced analysis utilities in C at this point.

[tylerjereddy]

If that's true, then the short-term debate seems about points 2 vs. 3 for the aggregation stuff alone, and for me comes down to the ability to use pandas as a "parent filter" before feeding to the C accumulator. I thought you already went the same direction with Perl pre-filtering records before it feeds to the aggregator when appropriate for the CLI utility.

So, you can apply whatever regexes or mathematics you want with a simple pandas-style query and then send those filtered records to darshan-util for aggregation in point 2 (this is not so different than Perl, except that we should get a bit more mileage from pandas queries/database filters). In point 3, I'm not so sure the C layer would have the same full-blown database-style query support that pandas would have for filtering, and that it would be as convenient for an end-user to use (though most shot-term stuff is just baking into the report I guess). I haven't really seen the demonstration of how that kind of thing is currently being handled in C (I thought it was in Perl only to pre-filter).

If you apply the filtering ops in pure Python for point 3, as you read in the individual records, the design just seems really problematic--without all the data already available in a dataframe some kinds of advanced filtering are probably impossible to start with (how do you filter all > average when average isn't available yet?), even ignoring the speed benefits of using a pandas query on a dataframe vs. over and over on individual records in a slow Python loop.

[carns]

Those points are sensible to me. If you wanted to select a subset of files based on name or path and calculate the aggregate bandwidth for just those files, then it absolutely makes a lot more sense to do the selection/slicing in Python. If that part is true (and I'm pretty sure it is; I'm a "C guy" and even I don't want to do that in C), yet the existing vetted accumulator logic is in a C library, then 2) seems like the path of least resistance in the short term. 4) (or something conceptually similar that avoids the need for either side to manually convert, at any rate, doesn't strictly have to be Arrow) might generalize more beyond the current problem at hand, but it would be considerably bigger undertaking from where we are now.

[shanedsnyder]

I'm not sure I understand the long-term issue--are you claiming that darshan-util performs analysis so complex it can't be reasonably represented using standard algorithms on rectangular data structures? That seems like a bit of a stretch, but I can understand it may take a long time to rewrite it and not be a priority to do so.

I wouldn't say it's super complex, but there are for sure some subtleties that make it difficult (e.g., calculating derived metrics for records representing a single rank vs all ranks). I'm sure it can be done in Python, but it's probably a pain to redo and I don't know that any of us want to work on that. It also doesn't really help us unless everything moves to Python. That's really what I'm trying to say, that this more involved and would take a sustained effort I think.

Regardless of where the logic lives, we need to make sure users are aware that we have a preferred method for aggregating and summarizing record data. Part of the idea of generalizing this stuff was so that users don't think you can just add all the rows in a dataframe together to accurately summarize total file statistics. We should also use these methods wherever possible (e.g., summary reports) for the same reason.

If you apply the filtering ops in pure Python for point 3, as you read in the individual records, the design just seems really problematic--without all the data already available in a dataframe some kinds of advanced filtering are probably impossible to start with (how do you filter all > average when average isn't available yet?), even ignoring the speed benefits of using a pandas query on a dataframe vs. over and over on individual records in a slow Python loop.

I was imagining something closer to this initially. Admittedly half-baked, but I was thinking you could just hold onto all the returned record pointers from the C library rather than immediately converting to Python representation. Then, you could use filtering and accumulating on this set of records, before converting to final Python form. There are probably some flaws there too, but ultimately I've come around to option 2 anyways -- it's more general in the sense that you could just read data from lots of logs into dataframe(s) and then decide what to accumulate from there. I agree that taking advantage of benefits of pandas is appealing.

[jakobluettgau]

Similar to what came through in some of the discussions already I would also consider a hybrid solution as far as the whole of PyDarshan is concerned, but actually, also reduce and eventually drop the dependency for any "in-house" C for aggregations but instead think about how to allow querying/lazy loading into logs without running into double-counting issues on the Python/Pandas/Arrow/or whatever-aggregation-solution side.

1. For the runtime component the binary format has to be implemented and maintained in C anyways. I see no reason why this should be moved or implemented in Python (except for if we wanted to go aim for a pure python package), it would definitely be less efficient and just duplicates code, and the only three real pain points we experienced were having to load records one at a time, having to unpack a nested hierarchy, and running into a cases where holding all data in memory would exceed the host memory (e.g., due to initially always loading all name records). I think we have a roadmap to mitigate those cases as well as cater to most of the existing aggregation frameworks by adding 3-4 additional C APIs.
2. I would not aim for a bi-directional flow between darshan-utils and pydarshan, but assume data flows from log to pydarshan/aggregation lib, and plan for a facility first on batch loading and then filtering by id, rank, maybe counters. I would post-pone adding a filtering mechanism on the C side in favor of making darshan data seekable in batches and at an offset, because that allows to take advantage of aggregation libraries that can do lazy loading, parallel processing or aggregation in memory-constrained environments (I think this is feasible, the logs are stable, and we probably get away with scanning the log once, in the beginning, to obtain counts and build a little bit of an index).
3. As for aggregations, I would also rather rely on existing data science libraries do this. For python/pandas/apache arrow for example, that would most likely mean building a small collection of PyDarshan-specific "aggregation_funcs" that can be passed to reduce/aggregate functionality which take some peculiarities about the counters into account (just as you do in the C-utils side, say applying the num_processes as a multiplier when building the sum for a particular record). See for example https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.aggregate.html For apache arrow, however, this API is still experimental: https://arrow.apache.org/docs/python/compute.html#user-defined-functions
   • There is also the potential to make our life a little easier by avoiding "special records" that can't be aggregated using standard pandas/arrow aggregators (which also benefit from accelerators/are vectorized already!). In particular magic numbers such as Rank "-1" could expand into multiple records instead of being kept in their compact form (but at the same time, as we assume applications to become larger, the opportunity of having the reduced memory footprint at a price at slightly higher agg_func complexity could be worthwhile)
   • The key question from an PyDarshan API perspective would be: Are we wrapping the internal pandas/arrow/etc. representation to add guards to prevent users from accidentally performing "illegal" aggregations. And for a users that asks for the dataframe, maybe expand said magic values into values that can be aggregated with standard facilities.
4. Mid, to Long-Term I also think we should move away from CFFI to a Python Extension as it offers better error handling and requires no workarounds to locate the libraries (there are multiple routes for this, and if Tyler has good experience with Cython I am all for that but I think even a traditional CPython extension (which we sort of build already anyways) will work just fine while avoiding some of the detours we have to take for CFFI at the moment).
   • The only "downside" I see to this is, that it seemed attractive at some point to be able to use the site-installed pydarshan, but given the small footprint of the c-utils library, I see no issues with just switching over completely to a distribution that always relies on the binary wheel that bundles it's own darshan-utils: One less incompatibility to worry about.

[shanedsnyder]

Thanks for the comments, all. I think there's a lot of good discussion here, in terms of shortcomings of what we have now and different short- and long-term plans to improve things.

Acknowledging that it's a shorter term plan and not ideal, it sounds like we may at least have enough agreement to just follow route 2 from my original discussion comment, where we use glue code Tyler has prototyped to convert back from dataframes representation into raw buffers expected by the C library? That will probably give us the quickest path to all the functionality we need for comprehensive Darshan job reports we have been trying to work towards, even if it's an inefficient/quirky process.

Longer term, I completely agree with implementing new analysis capabilities in Python going forward, and potentially just moving all Darshan utilities to Python if it seems tenable. I think we are starting to come around to wanting to experiment with things like Arrow/Parquet to see if they can help alleviate some of the quirks of working with Darshan logs/interfaces -- that's something that we could investigate in parallel with further development/refinement of PyDarshan analysis tools, or maybe even something some summer students could be interested in working on. Definitely worth looking into.

[carns]

I agree with continuing with option 2 in the short term. It would be great to get to a functionally complete point on the summary report without adding a big refactoring dependency. It feels like we are really close, and we've learned a lot along the way :)

We can pursue a more fundamental long-term rethink concurrently to that effort too though, I think.

Parquet looks like a promising avenue. Log files in Parquet format would enable Python tools ingest data without using the darshan-util C library as a middle man at all. It would solve another long-standing pain point by making it easy to find out how many records (i.e. rows) are in a log up front without having to do a full pass (something we cannot currently do without altering the existing binary log format). Parquet and Arrow both appear to handle that naturally as part of their normal metadata schemas.

More generally it's just a bid to make the data format/schema itself the language for interoperability between toolkits so that we don't have to lean so heavily on runtime bindings and data conversions.

Maybe the action item on the long-term front is to do a proof of concept of storing a Darshan log in Parquet (either emitted from the current Python code or translated from a static log)? That would let us a) make sure it can capture everything b) see how well the compression works and c) confirm how efficiently pydarshan could ingest it in comparison to using the darshan-util library? We could make a more informed concrete decisions depending on how that goes and what gotchas we encounter.