JFY/extract data from discord & load to bigquery

[joemarkyangag]

ITEM#1: filename "extract_discord.py"

I created a bot and add it to my discord channel. I use this code to command that bot to extract messages in my discord channel.

ITEM#1: filename "load_bigquery.py"

I used the Activity Watch application to monitor my Windows OS screentime. After extracting the data on the application, I load the data to my BigQuery. I used this code to do the extraction and loading.

[philgerardsoto]

Hi @joemarkyangag thanks for this!

Few suggestions:

1. For the Python packages, instead of installing them on the new .py files, kindly put them on setup.py.
2. Create a separate .py for extracting Activity Watch data
3. Wrap the loading of data to BigQuery into a Python function. Something like

def load_data_to_bigquery(df, dataset_id, table_id):

This will help us re-use the code for loading data from other sources.

4. Document setup guides for Discord and Activity Watch in README.md. We might also need to modify the setup guide for BigQuery.

Thank you!

[joemarkyangag]

okay bro. Got this. Thank you

[philgerardsoto]

Hi @joemarkyangag thanks for applying the comments! Will review soon. In case you get a chance, kindly create separate PRs instead of one PR:

1. Given df, load to BQ
2. Extract Discord data
3. Extract Activity Watch data

[philgerardsoto]

On second thought let me review the BigQuery load process first then we can create separate PRs for the other functions later. Thanks bro @joemarkyangag

[philgerardsoto]

We'll be using DuckDB instead of BigQuery. We'll wrap Discord extraction in dlt in the future. Let's keep this PR open for now. Thank you. cc: @joemarkyangag @sairilseb-me