Web Playground with WASM

[waynexia]

Hi! I've made a WASM binding for DataFusion (https://github.com/waynexia/datafusion-wasm) and a Web Playground based on it (repo, demo)

It now supports basic queries (like create, insert, select and explain). Generally speaking, DataFusion works pretty well in WASM!

My definition of this Web Playground is something like DuckDB WASM, or the datafusion-cli in browser. But not sure about what the shape of datafusion-wasm binding should be. I.e., expose as many APIs as possible like an embedded computing library, or just focus on the core SQL/DataFrame interface. I would like to hear feedback from you.

[alamb]

But not sure about what the shape of datafusion-wasm binding should be. I.e., expose as many APIs as possible like an embedded computing library, or just focus on the core SQL/DataFrame interface. I would like to hear feedback from you.

My opinion is that the WASM bindings have two primary usecases:

1. "embedded analytics" in a webapp (aka running SQL or dataframe queries to drive a UI without requiring a backend server connection)
2. Computation on edge (e.g. CloudFlare's recently announced Workers APIL https://developers.cloudflare.com/workers/runtime-apis/webassembly/

Thus I would personally suggest focusing first on the SQL API (as it has the lowest surface area) and then maybe the dataframe API

[waynexia]

Thanks for your insight 👍 My personal short-term plan for this binding would look like:

• Expose configuration APIs (SessionContext)
• Define and expose RecordBatch
• Support GCS and Azure Bolb
• Support more kinds of queries (fix them in the "upstream" datafusion)

I'll open a tracking issue in the binding repo for easier collaboration.

[waynexia]

FYI waynexia/datafusion-wasm#2

[alamb]

BTW: What would you think about moving your playground to https://github.com/datafusion-contrib (I can set you up a repo there with full admin rights) ? I think that might allow more people to find and potentially contribute to your project if that is something you are interested with

[Omega359]

Singlestore as well - https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/create-wasm-udfs/

[waynexia]

Just came back from holiday, sorry for the delay.

What would you think about moving your playground to https://github.com/datafusion-contrib

I'd like to! There are two repos, one for the Web playground and one for the WASM binding. Maybe we need two positions in datafusion-contrib?

[alamb]

awesome @waynexia what should they be called? datafusion-wasm-playground and datafusion-wasm-bindings (to mirror the name of python bindings)?

[waynexia]

Looks good to me 👍 I'll transfer the code tomorrow

[alamb]

New repos created:

https://github.com/datafusion-contrib/datafusion-wasm-playground
https://github.com/datafusion-contrib/datafusion-wasm-bindings

🚀