support Apache Arrow as a normalized data representation

[mbostock]

This allows an Apache Arrow Table to be used directly as the normalized representation of mark data (computed in mark.initialize), rather than requiring conversion to an array. This requires patching a few places to support both types, and then finally in valueof we materialize the columns as arrays using table.getChild and vector.toArray, which should be as efficient as possible.

It’s a bit tricky without type checking to make sure we haven’t missed things, but it doesn’t seem too bad, and we should be able to find the places with tests.

This also uncovered a number of preexisting problems with BigInt coercion (since Arrow represents dates as BigInt).

Fixes #191.
Fixes observablehq/framework#1376.
Supersedes #2030.
Supersedes #2096.

[mbostock]

(This fixes coerceNumbers for BigInt arrays which is important because Apache Arrow uses BigInt64Array for dates… somewhat unnecessarily in my opinion when Float64Array would work better in practice.)

[Fil]

I think this works! I will close #2030 because the description and discussion here is much better.

[Fil]

If we convert on the fly all the eligible data to Arrow tables with tableFromJSON, and run the tests, we see a few things:

• most tests pass
• a few tests crash when they try to call Date methods directly on the values (which are not dates anymore but numbers)
• a few tests result in different charts — mostly because invalid values are converted to 0 by tableFromJSON
• textOverflowClip &c crash because they use a custom transform which calls data.flatMap.

Nothing to see there.

More interestingly, we see crashes with:

• differenceY &c, and findArrow: because Plot.find in group.js reads data[i] directly
• musicRevenueCustomOrder crashes because its custom order function gets passed undefined values (error coming from orderComparator in stack.js reading data[i] directly)

… and that is all!

So I guess these are two places that need an additional arrayify. [FIXED]

code for the global conversion

in options.js, replace dataify by this version:

import {tableFromJSON} from "apache-arrow";

// Like arrayify, but also allows data to be an Apache Arrow Table.
export function dataify(data) {
  return isArrowTable(data)
    ? data
    : Array.isArray((data = arrayify(data))) && data.length > 0 && isObject(data[0]) && !data[0].type
    ? tableFromJSON(data)
    : data;
}

[Fil]

We'll have to review all the remaining occurrences of data[:

• pointer.js (visible at http://localhost:8008/?test=arrowTestPointer)
• sortData (visible at the new http://localhost:8008/?test=arrowTestSort)
• exclusiveFacets (not applicable, reindex is a Symbol)
• tree.js (visible at the new http://localhost:8008/?test=arrowTestTree)

[domoritz]

a few tests crash when they try to call Date methods directly on the values (which are not dates anymore but numbers)

It would probably be a good to have some kind of compatibility mode for arrow that returns dates instead of numbers. Jeff suggested this before.

[mbostock]

Great job testing and finding these remaining incompatibilities, @Fil! Thank you.

[mbostock]

I reviewed the other places you identified and optimized the handling of Arrow tables. PTAL!

[Fil]

I've added a (convoluted) test for the code in tree.js, and a fix. I've also added tests for some of the other places we identified. Note that the comparator sort was not tested at all (with arrays nor arrow) until now!

[mbostock]

Looks good to me! Please approve if you agree.