Data Exploration and Software Tools and Resources at WRC

[shaunwbell]

In response to Oct 11th TUG Meeting - Please feel free to use this discussion formum to share software, approaches, and challenges.

The initial discussion was inpired by the following tweet - https://twitter.com/simonw/status/1572285367382061057 which asked the question:

If someone gives you a CSV file with 100,000 rows in it, what tools do you use to start exploring and understanding that data? and Follow up: same question for 1 million rows, 10 million rows, 1 billion rows

---

Links to useful resources from the discussion:

• datasette a SQlite based data explore and hoster, can read CSV files too. Good for tabular data
• visidata a tabular data explore
• IDV UCAR deveoped Geophysical data explore

[noaaroland]

@Roman-Battisti mentioned dask.dataframe as a potentially useful extension to pandas. FWIW, I have been using it to read and manipulate some biggish data sets (these are cruise tracks with about 90 columns and anywhere from 1.9 million to 54 million rows, read from an ERDDAP and saved to parquet file for re-reading via dask.dataframe). dask.dataframe is a direct analog to pandas, except with lazy loading and parallel processing so if you are used to using pandas you'll find this pretty straight forward.

So far, I've experimented with using datashader to make an image of all of the cruise tracks to overlay onto a map via Plotly, but eventually I'll be trying to take advantage of the supposedly fast sub-setting to pull out either geographic sub-sets or groups of individual cruises for plotting.

I have also tried a similar tool called vaex. It's less of a direct analog to pandas (though similar). I was mystified for a long time how to even dereference the objects once the sub-set was completed, but finally figured it out using the examples from the Dash Enterprise sample app for vaex.

Either of these tools could be used in a notebook, but if you happen to want to build a Web app around these tools head over to our Dash Enterprise [lab network or VPN required] instance for examples.

*edit to note Dash portal requires VPN
*second edit to tag Roman after Shaun fixed the collaborators.