CLIA-1 - Specification for Uploading Dataset to MongoDB #29
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![gitstart-app[bot]](https://avatars.githubusercontent.com/in/46283?s=60&v=4){kind=small}
|
This PR is estimated to cost between 100 and 120 credits. |
|
@leej3 FYI the changes in this PR that are unrelated are due to fixing the lint failing in pre-commit tests. tox test seems to be failing in main as well due to python versioning issue |
scripts/invocation_upload.py
Outdated
Comment on lines
21
to
26
| # NOTICE: items without mapping to the dataframe | ||
| unmapped = { | ||
| "article": "", | ||
| "is_relevant": None, | ||
| "is_explicit": None, | ||
| "user_comment": "", | ||
| "osm_version": "", | ||
| } |
There was a problem hiding this comment.
@leej3 I retained the unmapped dictionary because some of the values are needed outside the metrics. the unmapped dic holds all missing values and are referenced in the appropriate location
|
Looks good thanks. We can look at speeding things up separately. |
leej3
approved these changes
Aug 7, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR was created by GitStart to address the requirements from this ticket: CLIA-1.
Specification for Uploading Dataset to MongoDB
Overview
The task involves downloading a 3GB CSV file, converting it to a more efficient format, and
then uploading the data to a MongoDB database using a defined schema. This is a one-off
task but the schema for the mongoDB may change in the future so it may need to be
adapted for final usage to match any changes in the schema.
Tasks and Requirements\
We want to upload the data in this 3GB CSV file from here: https://osf.io/2f857
(metadata also available at the link).
A smaller file that you can work from is available at this link. You can read this with
pd.read_feather("all_indicators.feather")
\
Create a pipeline to convert the output of the previous step to a dataframe ideal for
use with the provided schema.
Abstract operations into functions as required to keep the command chain concise
and readable.
Utilize Polars or Pandas for the dataframe operations.
\
Use the output of the previous step with the preliminary schema provided for
MongoDB. Specifically use the Invocation class for the document model.
\
Write tests for the database upload step using a stub for the database. These tests
do not need to test for unforeseen issues, this dataset is the only one the script will
be used for.
Use pytest with fixtures as required.
\
All code, including tests, should be contained within a single Python module.
Ensure the conversion/upload functionality is accessible from Python as an
importable function from the module.
Write the script in a way that it can easily be altered to deal with schema changes.