Declarative Graphing + JSON/DataFrames #482

srush · 2021-01-23T15:59:42Z

This example code implements:

Dex-y Bindings to the Vega-Lite graph library https://vega.github.io/vega-lite/
a JSON show style type class to write out Dex types to JSON strings. JSON implementation #474
A simple Dex flat heterogenous dataframe

(This is not meant to overlap / intersect with the Dex plotting. I just got interested in how clean this Vega-Lite Library was compared to Plotly and wanted to try it out. )

Here's what the rendered output looks like https://srush.github.io/dex-lang/examples/json.html

Neat parts:

Dex is really good as a backbone for the this style of dataframe based declarative method. I mostly like the chart functional interface. (Besides maybe the use of numbers for variable id.)

Questionable Parts:

The DataFrame is basically a JSON table. Probably would have been right to do it with records. However I don't know if it is possible to enumerate over records or associate accessors with names?
I don't think Dex allows arbitrary javascript injection (nor should it? it seems stateful). So I am outputting iframes.

Nasty parts (most of this I knew already):

The json implementation is a pseudo tree. I collapse recursion to strings.
String stuff is slow (but works).

danieldjohnson · 2021-01-23T18:07:36Z

I don't know if it is possible to enumerate over records or associate accessors with names?

Not at the moment (see #258) but this is on the roadmap!

apaszke

Wohooo, this is amazing! This mostly looks good, but can you please add some more docs? It's a bit difficult to review as it is

apaszke · 2021-01-25T11:14:33Z

examples/json.dx

+
+
+data DataFrame key n value =
+     AsDataFrame (n => key => value) (key => String)


Since value doesn't depend on key this is really a homogeneous data frame instead, right?

Good point... it's heterogeneous because of the variant type for value. I guess ideally this would be a table where columns are of different types? I will have to think more if that is possible.

Ohhh it's heterogeneous because you store all values as JSON! When we initially thought of data frames, we usually considered them as tables-of-records (n=>{field1: Float & field2 : Int}). A nice benefit of that is that those do have an efficient lowering, unlike this huge variant. But it would make it much harder to store the column=>meta association, because the column index set really indexes the fields of the record 🤔

Yeah records are definitely the right solution to this. But I couldn't really figure out how to cleanly enumerate over fields.

Although maybe I just need like a variant over Iso's instead of values. I guess the Meta could could do that work.

Oh actually @danieldjohnson , maybe the cute solution is to have col metadata Iso Variant indexed and the columns in a Iso record. The metadata needs to provide a function to extract it's column (say into json)

Downside: The user would have to manually link them. Same names?

Upside: homogenous data columns, variant named columns (not ordering based), column enumeration.

Tried a slightly different strategy. I think maybe flat dataframes are a hack and kind of unneeded in dex. To make graphing work, you can just construct the flat data representation along with the graph description. This can come from a record or from any other source.

n = Fin 100 df2 : {x1: n => Float & x2: n => Float & weight: n => Float & label : n => Class} ... chart2 = (AsVLDescriptor (pure Point) [("title", "Scatter")] [{title="X1", encodings=pureLs X, encType=Quantitative, rows=wrapCol #x1 df2}, {title="X2", encodings=pureLs Y, encType=Quantitative, rows=wrapCol #x2 df2}, {title="Weight", encodings=pureLs Size, encType=Quantitative, rows=wrapCol #weight df2}, {title="Label", encodings=toList [pure Color, pure Tooltip], encType=Nominal, rows=wrapCol #label df2}]) :html showVega $ toJSON chart2

Ooh, I like the most recent version!

I think that manipulating records will be much, much easier once we can enumerate over fields in userspace. So hopefully the "manually linking" downsides will no longer be a problem at that point.

examples/json.dx

srush · 2021-01-25T19:17:59Z

static/index.html

@@ -10,6 +10,12 @@
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.css" integrity="sha384-AfEj0r4/OFrOo5t7NnNe46zW/tFgW6x/bCJG8FqQCEo3+Aro6EYUG4+cU+KJWu/X" crossorigin="anonymous">
  <script defer src="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.js" integrity="sha384-g7c+Jr9ZivxKLnZTDUhnkOnsh30B4H0rpLUpJ4jAIKs4fnJI+sEnkvrMWph2EDg4" crossorigin="anonymous"></script>
  <script defer src="https://cdn.jsdelivr.net/npm/[email protected]/dist/contrib/auto-render.min.js" integrity="sha384-mll67QQFJfxn0IYznZYonOWZ644AWYC+Pt2cHqMaRhXVrursRwvLnLaebdGIlYNa" crossorigin="anonymous"></script>
+
+<script>
+  function resizeIframe(obj) {


This is maybe lame, I need to look more into how to sandbox javascript per cell.

Moved to dynamic.js

apaszke

Thanks for adding the docs, this is super helpful. I would only like to sort out the resizeIframe situation and I think it might be good to merge. Also, how about we rename the example to vega-plotting.dx or something like that? That part is even cooler than JSON for me 😛

apaszke · 2021-01-26T10:31:19Z

examples/json.dx

+def pureLs (x:a) : List (Opts a) =
+   AsList 1 [WithOpts x AsNone]
+
+def mergeOpts [ToJSON a, ToJSON b] (x : a) (y : b) : Value =


I guess one caveat here is that you probably shouldn't be merging too many overlapping options 😃 But this is fine for now!

apaszke · 2021-01-26T10:35:52Z

examples/json.dx

+
+
+def showVega (x: Value) : String =
+    "<iframe width=\"100%\" frameborder=\"0\" scrolling=\"no\" onload=\"resizeIframe(this)\" srcdoc='<html> 


Can't this resizeIframe business be done with something like style="height:100%" on a bunch of the tags below?

Apparently you can't do this :( Need to resize with js as far as I can tell. But the JS seems to work.

apaszke

This LGTM. I'm particularly happy that the data frames themselves don't hold any JSON, because that would make it very hard to transform them from inside Dex.

apaszke · 2021-01-27T10:21:08Z

examples/vega-plotting.dx

+' Start with a well type and useful Dex record
+
+df1 = {a = ["A", "B", "C", "D", "E", "F", "G", "H", "I"],
+       b = [28, 55, 43, 91, 81, 53, 19, 87, 52]}


Any reason to make data frames records-of-tables? I thought that table-of-records is a bit more intuitive, because that's closer to relational algebra

Makes sense. Switched to rows.

apaszke · 2021-02-01T16:24:56Z

I've pushed a small update that avoids modifying our builtin JS sources just for this example. Will merge once the CI confirms that it's all good!

srush added 3 commits January 23, 2021 00:36

json first

ee87c06

.

228bad6

ex

2336b20

google-cla bot added the cla: yes label Jan 23, 2021

apaszke reviewed Jan 25, 2021

View reviewed changes

update json demo

e84ce08

srush commented Jan 25, 2021

View reviewed changes

apaszke reviewed Jan 26, 2021

View reviewed changes

srush added 2 commits January 26, 2021 12:26

remove the dataframe requirement

95cf9a5

remove js from static page

4bc91e3

srush changed the title ~~[Preliminary] Declarative Graphing + JSON/DataFrames~~ Declarative Graphing + JSON/DataFrames Jan 26, 2021

change name of file

fb72596

apaszke approved these changes Jan 27, 2021

View reviewed changes

srush and others added 5 commits January 27, 2021 08:39

move to rows

b4a2a06

Merge branch 'main' into json

c51cb06

add last line because tests break

de59579

Merge branch 'json' of https://github.com/srush/dex-lang into json

7b2c2ba

Don't modify compiler JS sources

50693ae

apaszke merged commit bd06d8f into google-research:main Feb 1, 2021

apaszke pushed a commit that referenced this pull request Feb 1, 2021

Declarative Graphing + JSON/DataFrames (#482)

93ddd40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Declarative Graphing + JSON/DataFrames #482

Declarative Graphing + JSON/DataFrames #482

srush commented Jan 23, 2021 •

edited

Loading

danieldjohnson commented Jan 23, 2021

apaszke left a comment

apaszke Jan 25, 2021

srush Jan 25, 2021

apaszke Jan 26, 2021

srush Jan 26, 2021

srush Jan 26, 2021 •

edited

Loading

srush Jan 26, 2021

danieldjohnson Jan 26, 2021

srush Jan 25, 2021

srush Jan 26, 2021

apaszke left a comment

apaszke Jan 26, 2021

apaszke Jan 26, 2021

srush Jan 26, 2021

apaszke left a comment

apaszke Jan 27, 2021

srush Jan 27, 2021

apaszke commented Feb 1, 2021



		data DataFrame key n value =
		AsDataFrame (n => key => value) (key => String)



		def showVega (x: Value) : String =
		"<iframe width=\"100%\" frameborder=\"0\" scrolling=\"no\" onload=\"resizeIframe(this)\" srcdoc='<html>

Declarative Graphing + JSON/DataFrames #482

Declarative Graphing + JSON/DataFrames #482

Conversation

srush commented Jan 23, 2021 • edited Loading

danieldjohnson commented Jan 23, 2021

apaszke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srush Jan 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apaszke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apaszke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apaszke commented Feb 1, 2021

srush commented Jan 23, 2021 •

edited

Loading

srush Jan 26, 2021 •

edited

Loading