Task - Create Clear Workflow to Populate Dev Environment with Example Data

[adambechtold]

Background

We're opening this project up to more collaborators.

Frictions

• It's not clear how to add example data
• Adding data requires collaborators to have their own to last.fm and Spotify developer apps

Goals

• 🎯 Goal - It is very easy spin up Taste Explorer with example data
• 🎯 Goal - Maintain Security of Production Data
• 🎯 Goal - Keep costs low

[adambechtold]

Approach - Write Instructions for How to Add Users from Scratch

1. Create database
2. Create api key to last.fm and spotify
3. Add users using admin api
4. Run cron jobs to backfill listening history

▼ Con - Requires collaborators to have their own last.fm account and spotify developer account
▼ Con - Complicated and time-consuming

[adambechtold]

Approach - Host a Small, Read-only Database

Create a very small database on AWS with example data and make read-only permissions to it public.

▲ Pro - Very easy for collaborators to get started
▼ Con - Performance could degrade as more people use it
▼ Con - Cost - This isn't free

Variant - Create separate database on existing prod infrastructure

▲ Pro - Free
▼ Con - Use by collaborators could degrade prod's performance

[adambechtold]

Approach - Provide Database Dump(s)

Provide .dump files with example data.

▲ Pro - Collaborators can choose their favorite database platform/approach
▲ Pro - Easy to provide various dataset sizes

Variant - Host on GitHub

▲ Pro - Version control
▼ Con - Some of the dumps could be very large, making the repo unnecessarily large

Variant - Host on S3

▲ Pro - Dumps can be very large
▼ Con - Version control is harder

[adambechtold]

Approach - Host docker Container Instance

Collaborators could simply pull down this repo, spin it up, and start going.

▲ Pro - Very easy to get started
▼ Con - More work to create than the .dump option

Variant - Have the container download a database .dump during start up

Host a .dump file on S3. Host a container on AWS container registry. Give the container an entrypoint script that checks if the database is already populated and, if not, downloads the dump and populates the database.

(See chat)

[adambechtold]

Update - .dump files are available on S3

I was able to create dump files and put them on S3 but am running into some issues creating a docker container that can download them and pre-populate the database.

I'll keep working on it, but here are the .dump files in the meantime:

• Example Data: https://music.adambechtold.xyz/api/setup/seed.zip

cc:
@tusharwebd