Coredumps: how to support reference values and tables?

[fitzgen]

The coredump spec currently doesn't mention tables, presumably because it isn't clear how to encode reference values in core dumps. But this is still an issue today even without tables because globals can store references, as well as locals and the operand stack, all of which are included in core dumps.

So we should figure out what to do about references and then also allow encoding tables in core dumps.

Today, Wasmtime is encoding funcrefs and externrefs globals as null, unconditionally.

The coredump value encoding doesn't even have options for reference values at the moment.

Things we should figure out:

• What to do with externrefs? Ideally they would at least have their null vs nonnull state encoded in the core dump. Maybe it also makes sense to have some other host-specific serialized form of the extern data encoded somewhere?

• What to do with funcrefs? We could use ref.func N except that this gets hard with funcrefs that were passed into a function that took a funcref parameter, and so there is no requirement that the funcref is actually in the function index space, so we don't have an N to choose.

  Also, for the wasm coredumps to remain valid wasm, the ref.func Ns need to be valid, and that means we will need to have functions inside the coredump.

  Also there is the question of which index space we are talking about, since core dumps have multiple instances. We could do what we do for globals/memories where the core dump has an index space for all globals/memories in all instances, and then the instance section says which of those are in its space. But it is all kinda wonky due to the fact that there can be funcrefs that aren't in any index space.

  I think this is all surmountable but I just haven't thought deeply about it yet.

• Looking ahead: what to do about GC references? The various allocation instructions are all generally constant, so we could create tree-like data directly in globals as const expressions. But DAGs will require temporary named storage so we can reference shared tails multiple times, and cycles present more hurdles.

My biggest take away from thinking about this so far is that I think we need to switch the encoding of locals and the operand stack to use optional const expressions (so we can get ref.func, struct.new and friends; optional so we can represent optimized away values) or something even more general (like functions/code sections that can do arbitrary allocation and have access to locals to facilitate cycle creation) or something coredump-specific rather than trying to reuse core Wasm constructs.

We might also want to consider switching away from using the core Wasm global section to encode globals so that coredumps can have optimized-away global values.

---

cc @xtuc