📊 Data

[shanejearley]

Notes on data, analytics, and ML

[shanejearley]

@hawyar Superset interesting option for no-code internal viz with Athena and Presto support. QuickSight of course has the built-into-AWS appeal and can be deployed with our project CDK stacks.

Curious to hear what features you liked in Metabase and any other potential platforms you dug up!

[hawyar]

@shanejearley recently there has been a wave of no code/low tools on my GitHub feed. I think they could come in handy for quick prototyping, visual aid for team discussions, and inspiration for our frontend. Nothing special with Metabase I just wanted to experiment with it. Here is demo if you want to check it out

https://metabase.hawyar.repl.co/

email: [email protected]
pass: hawyar123

[shanejearley]

Freedom from AWS Glue Crawlers found in MSCK REPAIR TABLE (documentation):

• Update our Glue table partitions on the fly without additional service overhead
• Supports our serverless, simple, block-by-block ETL model
• Depend on an Athena query that can be swapped out when/if we move from Athena/Glue to Presto/Hive counterparts
• ^Note: Not sure if the Presto Hive connector supports the same simple query, but at the very least Hive supports it (so same workflow, different API, potentially)

The Glue Crawler seems to be worth the service overhead when we want to quickly dump and query semi-structured big data in storage (HCF CMS was a perfect case), but it also seems unnecessary when we want to structure and query big data in storage (Casimir), especially with a simple approach to updating table partitions.

[shanejearley]

Moving from #94.

Proposed schema change workflow:

1. Open a GitHub issue regarding a change that needs to be made to the schemas to achieve a data goal
2. Discuss and provide background to make an initial guess about the required change
3. Change the schema and crawler + processor logic accordingly, and push to a sandbox stage environment
4. Test queries in Athena (Athena console, Datagrip, Athena Notebook) and refine the proposed changes as needed in the issue or corresponding pull request
5. Document (in the Athena Notebook) the query + calculation process (if that data goal is sufficiently important or large)
6. Merge and deploy changes via Casimir gitops workflows

[shanejearley]

BlockSci is an old high-end (no longer maintained) blockchain analysis engine that probably inspired the current commercial software design (thinking Dune, Messari, etc). From a very high level, we've worked on a similar approach, even though they were building in an academic setting and were primarily focused on Bitcoin at the time. They built their own query layer in C++ while we are embracing the Javaland Presto/Trino SQL engine for out-of-the-box query optimization.

There are likely some helpful design/meta concepts to pull from their work, which is well documented in the paper linked in their README.md.