add : mongodb integration #110
Conversation
for more information, see https://pre-commit.ci
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 Looks good to me! Reviewed everything up to 6788f90 in 13 seconds
More details
- Looked at
245lines of code in4files - Skipped
1files when reviewing. - Skipped posting
4drafted comments based on config settings.
1. src/vdf_io/export_vdf/mongodb_export.py:3
- Draft comment:
The import statementimport bsonis redundant since specific imports frombsonare already made. Consider removing it to clean up the code. - Reason this comment was not posted:
Confidence changes required:50%
The import statement forbsonis redundant since specific imports frombsonare already made. This redundancy can be removed to clean up the code.
2. src/vdf_io/export_vdf/mongodb_export.py:10
- Draft comment:
Theset_arg_from_passwordfunction is imported but not used. Consider removing this import to clean up the code. - Reason this comment was not posted:
Confidence changes required:50%
Theset_arg_from_passwordfunction is imported but not used in themongodb_export.pyfile. This is unnecessary and should be removed to clean up the code.
3. src/vdf_io/export_vdf/mongodb_export.py:14
- Draft comment:
TheConnectionFailureandOperationFailureimports frompymongo.errorsare not used. Consider removing these imports to clean up the code. - Reason this comment was not posted:
Confidence changes required:50%
TheConnectionFailureandOperationFailureimports frompymongo.errorsare not used in the code. These should be removed to clean up the code.
4. src/vdf_io/export_vdf/mongodb_export.py:117
- Draft comment:
Theobject_columns_listis populated but never used. Consider removing it to clean up the code. - Reason this comment was not posted:
Confidence changes required:50%
Theobject_columns_listis being populated but never used. This is unnecessary and should be removed to clean up the code.
Workflow ID: wflow_cuCIRBnrVVwADNP6
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
…maheshwari/vector-io into vipul/mongodb-integration
for more information, see https://pre-commit.ci
![kaizen-bot[bot]](https://avatars.githubusercontent.com/in/480502?s=60&v=4){kind=small}
| total=total, | ||
| num_vectors_exported=total, | ||
| dim=expected_dim, | ||
| vector_columns=vector_columns, |
There was a problem hiding this comment.
Comment: Inefficient data retrieval in get_data method
Solution: Consider using MongoDB's aggregation framework for more efficient data processing.
Potential Fix:
cursor = self.collection.aggregate([...]) # Use aggregation pipeline here
🔍 Code Review Summary✅ All Clear: This commit looks good! 👍 Overview
Useful Commands
|
| except pymongo.errors.ServerSelectionTimeoutError as err: | ||
| logger.error(f"Failed to connect to MongoDB: {err}") | ||
| raise | ||
| self.db = self.client[args["database"]] |
There was a problem hiding this comment.
let's add an error check here
| logger.error(f"Failed to connect to MongoDB: {err}") | ||
| raise | ||
| self.db = self.client[args["database"]] | ||
| self.collection = self.db[args["collection"]] |
| self.collection = self.db[args["collection"]] | ||
|
|
||
| def get_index_names(self): | ||
| if self.args.get("collection", None) is not None: |
There was a problem hiding this comment.
need to check if it exists
| flattened_data = [] | ||
| for document in batch_data: | ||
| flat_doc = self.flatten_dict(document) | ||
|
|
||
| for key in flat_doc: | ||
| if isinstance(flat_doc[key], dict): | ||
| flat_doc[key] = json.dumps(flat_doc[key]) | ||
| elif flat_doc[key] == "": | ||
| flat_doc[key] = None | ||
|
|
||
| flattened_data.append(flat_doc) | ||
|
|
||
| df = pd.DataFrame(flattened_data) | ||
| df = df.dropna(axis=1, how="all") |
There was a problem hiding this comment.
need to push data to disk as it is streamed, so that the RAM doesn't fill up
|
Thanks for contributing to Vector-io! please also give a short readme or how-to for exporting data from mongo, as it is a bit harder than a normal VectorDB (connection string v/s looking up fields like admin password from the portal). Thanks. |
|
Got you comments! Will do the needful! |
Add MongoDB Export Support
Add support for exporting data from MongoDB databases to the VDF format.
ExportMongoDBclass that inherits from theExportVDBbase class.This change will allow users to export data from MongoDB databases to the VDF format, enabling them to leverage the VDF ecosystem for vector search, embeddings, and other machine learning tasks.
Original Description
# Add MongoDB Export Functionality**
Add support for exporting data from MongoDB databases to the VDF format.
ExportMongoDBclass that inherits from the baseExportVDBclass.**
Users can now export data from MongoDB databases to the VDF format, enabling seamless integration with the VDF ecosystem and downstream applications.
Original Description
# Add MongoDB Export Functionality**
**
Introduces a new feature to export data from MongoDB into a specified format.
.cfgand environment-related entries to.gitignore.requirements.txtto includepymongo.mongodb_export.pyfor handling MongoDB data exports.**
**
This addition allows users to seamlessly export data from MongoDB, enhancing the tool's versatility.
Original Description
# Add MongoDB Export Functionality**
**
**
Introduce functionality to export data from MongoDB to a specified format.
.cfgand environment-related entries to.gitignore.requirements.txtto includepymongofor MongoDB support.ExportMongoDBclass for handling MongoDB data exports.**
**
**
This enhancement allows users to seamlessly export data from MongoDB, improving data integration capabilities.
Original Description
# Add MongoDB Export Functionality**
**
**
**
Adds the ability to export data from a MongoDB database to the VDF format.
ExportMongoDBclass that inherits from theExportVDBbase class.mongodbsubparser to the command-line interface to allow users to specify MongoDB connection details and export options.**
**
**
**
This change will enable users to export data from MongoDB databases to the VDF format, allowing for easier integration with the VDF ecosystem and downstream applications.
Original Description
- [ ] export script - [ ] import script