Meta issue: ongoing work as of early April 2023

[douglasdavis]

This is a meta issue to summarize the status of ongoing work in prep
for me being on leave from April 11 to May 18

I'll likely be editing this issue until Monday April 10

In no particular order of priority here are summaries of ongoing
things:

---

Form re-hydration

When we perform the necessary columns (or "column projection)
optimization; we'd like to retain the complete form.

Relevant issues/PRs

• Add data-less fields to necessary columns optimization #180
• feat: awkward Form rehydration; retain unused fields in name only (no data) #184
• feat: added an unproject_layout function #203

Latest status

@jpivarski is iterating on an unproject_layout routine in #203 that
will do the heavy lifting. We'll need a function for internal use and
library author use that can be used to apply this to arrays that are
instantiated at input nodes in a graph. for example: the __call__
method of the dask_awkward.lib.io.parquet._FromParquetFileWiseFn
class

---

Handling missing implementations

There are still parts of the awkward API that need to be covered by
dask-awkward.

Relevant issues/PRs

• Missing implementations required for analysis functions #214
• Determine if negative axis argument is equivalent to zero #215

Latest status

In recent weeks @lgray has been raising issues for missing
implementations (and including PRs for many!). Issue #214 asks for a
few more, and I created issue #215 as a general need. I know there are
many cases in the code base where we explicity support axis >= 1
without supporting negative axes. "Fixing" #215 (probably with some
kind of utility function to be used at the top of all dask-awkward
implementations) will help in a lot of places.

---

Greedy necessary columns

@lgray discovered a problem in typetracer reports where more columns
than necessary are reported as touched. @jpivarski and @agoose77 have
been digging into this and finding fixes.

Relevant issues/PRs

...

Latest status

There are no relevant issues or PRs in the dask-awkward repository, but
we should follow this closely and perhaps add some tests for
dak.necessary_columns that specifically targets calls that were
originally hurt by the greedy touching bug.

---

DataFrame interop

There was a recent feature request to add awkward's to_dataframe to
the dask-awkward.

Relevant issues/PRs

• feature request: dak.to_dask_dataframe ? #208
• feat: Add dask_awkward.to_dataframe #209
• ak.to_numpy turns an empty array of type string into an empty array of type float64 scikit-hep/awkward#2364

Latest status

@jpivarski found a bug in awkward where ak.to_numpy converts strings
to floats for empty arrays (scikit-hep/awkward#2364). This is needed
to generate a correct metadata-as-empty-dataframe object when we
instantiate a new dask.dataframe collection.

This also resurfaced discussion
(#208 (comment))
around the awkward-pandas work, where we would like to convert an
awkward array (or dask-awkward array) into a Pandas Series object (or
dask.dataframe Series object) backed by the AwkwardDtype extension.

---

JSON input with necessary columns optimization

The necessary columns optimization is currently used with
from_parquet and in uproot. We started to think about how to add
this optimization to reading JSON.

Relevant issues/PRs

• feat: JSON: refactor IO for JSON; make input implementations column optimizable #94

Latest status

So far we've only explored how to roundtrip between awkward Forms and
JSON schema. Once we have the projected Form we should be able to
generate a schema that can be passed to the from_json nodes in a
task graph.

---

Slow optimization

It was discovered that optimizing many repeated blockwise layers can
be slow. This is more of a general Dask issue, not necessarily a
dask-awkward issue

Relevant issues/PRs

• optimise linear layer chains #210
• douglasdavis@ef9a980

Latest status

@martindurant starting working on a general solution in #210. I
started working on a solution specific to getitem field access in a
branch on my fork (douglasdavis/dask-awkward@ef9a980)