Lazy, distributed boost-histogramming of Dask collections with dask-histogram

[douglasdavis]

Hi Henry, Hans, et. al.,
I've spent a bit of time working on histogram support in the Dask universe. First, inside the dask.array module (dask/dask#7387, dask/dask#7634, with an upcoming PR for histogram2d support). And more pertinent for discussion here, using the API provided by boost-histogram to create a new Python library: dask-histogram. As of today the API I had in mind (a lazy-fillable Histogram object, and a mirroring of the classic np.histogram{ , 2d, dd} functions) is fully implemented, but I'm sure there are still some things to iron out. I just wanted to mention the library here and say I'd appreciate any feedback! The library may also end up in the dask-contrib organization at some point.

[douglasdavis]

While discussing a potential move to dask-contrib, the gauging of interest in upstreaming Dask support in boost-histogram was mentioned. @henryiii I'd be interested to hear your thoughts (I'd be happy to help maintain Dask support for histograms at any location).

[henryiii]

I would generally be supportive of this, I think. The important thing would not to add a general dask dependency, obviously, but I'm sure this is how xgboost/lightgdm worked? @HDembinski what do you think? Feel free to look at https://dask-histogram.readthedocs.io/en/latest/intro.html and subsequent pages.

[henryiii]

We could also just advertise dask-histogram at the top. Does dask-histogram support Hist? Providing good support for Hist might help dictate how we implement this, as standalone or as a built-in feature.

[douglasdavis]

The trivial case of passing a dask_histogram.Histogram object to hist.Hist and then doing some plotting works as expected (i.e. it's already filled). Right now filling downstream of the hist API materializes Dask collections killing the laziness. So I would say at the moment support for Hist is partial at best.

[douglasdavis]

Digging into hist.BaseHist.fill a bit I see that it's calling super().fill() where super() is always boost_histogram.Histogram (where the fill method forces the materialization of the Dask collection). This leads to the question: would dask_histogram be a natural place to subclass hist.BaseHist and override fill?

[henryiii]

Hist differs from boost-histogram in .fill because it supports named axes, whereas boost-histogram does not. It's a major pain point at times to make things work downstream, but boost-histogram apparently won't support named axes (#430). So there will need to be some customization there, either from dask_histogram or from Hist.