Feat/automatic enable disable datasource #52

amindadgar · 2024-04-23T08:51:11Z

Summary by CodeRabbit

New Features
- Introduced a new retrieval system to handle both legacy and new versions of data indexing.
- Enhanced data source selection based on community ID for more tailored data fetching.
- Added GitHub data retrieval capabilities to the querying system.
Bug Fixes
- Improved error handling in the data retrieval process to ensure consistency across different data versions.
Tests
- Added comprehensive tests for new data source selection functionality and GitHub query capabilities.
Refactor
- Updated inheritance and removed redundant methods in the query engine classes to streamline operations.

coderabbitai · 2024-04-23T08:51:23Z

Walkthrough

The recent updates bring a comprehensive framework for data retrieval, enhancing adaptability and efficiency. Key additions include a GitHub query engine, a custom vector store retriever for error handling, and tests ensuring system reliability and robustness in handling diverse data queries.

Changes

Files	Change Summaries
`.../custom_retriever.py`	Introduced `CustomVectorStoreRetriever` to handle errors and retrieval processes for different index versions.
`celery_app/tasks.py`	Added `DataSourceSelector` for dynamic data source selection based on `community_id`.
`subquery.py`, `.../__init__.py`	Integrated `GitHubQueryEngine` for GitHub data querying capabilities.
`.../test_data_source_selector.py`	Created tests for `DataSourceSelector` validating data source selection logic.
`.../test_base_engine.py`, `.../test_github_query_engine.py`	Added tests for `BaseEngine` setup and `GitHubQueryEngine` functionality.
`utils/data_souce_selector.py`	Implemented `DataSourceSelector` class for community-specific data source selection.
`.../base_engine.py`, `.../github.py`	Introduced `BaseEngine` for vector store setup and `GitHubQueryEngine` for specific GitHub querying.
`.../level_based_platform_query_engine.py`	Updated inheritance to enhance modularity and remove redundant code.

🐰✨🎉
A hop, a skip, a code deploy,
Changes abound, oh what a joy!
From vectors fetched to sources picked,
Each line of code, meticulously fixed.
Celebrate the tweaks, both big and small,
For a better query, one and all! 🎉✨🐰

Recent Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 84b67c7 and b33526f.

Files selected for processing (2)

celery_app/tasks.py (2 hunks)
tests/integration/test_data_source_selector.py (1 hunks)

Files skipped from review as they are similar to previous changes (2)

celery_app/tasks.py
tests/integration/test_data_source_selector.py

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 1

Out of diff range and nitpick comments (1)

tests/unit/test_github_query_engine.py (1)

14-14: Consider removing the print statement from the test method. It's generally not useful in automated test environments and can clutter test output.

coderabbitai · 2024-04-23T10:16:56Z

celery_app/tasks.py

@@ -17,6 +17,7 @@
 from tc_messageBroker.rabbit_mq.payload.payload import Payload
 from tc_messageBroker.rabbit_mq.queue import Queue
 from traceloop.sdk import Traceloop
+from utils.data_souce_selector import DataSourceSelector


Fix typo in the import statement.

- from utils.data_souce_selector import DataSourceSelector + from utils.data_source_selector import DataSourceSelector

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change

from utils.data_souce_selector import DataSourceSelector

from utils.data_source_selector import DataSourceSelector

coderabbitai

Actionable comments posted: 2

Out of diff range and nitpick comments (1)

utils/data_source_selector.py (1)

26-51: Consider adding error handling for database operations to ensure robustness.

coderabbitai · 2024-04-23T11:42:18Z

utils/data_source_selector.py

+        Returns
+        ----------
+        data_sources : dict[str, bool]
+            a dictionary representing what data sources is selcted


Correct the typo in the word "selected".

coderabbitai · 2024-04-23T11:42:19Z

utils/data_source_selector.py

+        platforms = list(map(lambda data: data["platform"]["name"], db_results))
+        data_sources = dict.fromkeys(platforms, True)


Consider using a dictionary comprehension for clarity and Pythonic style.

- platforms = list(map(lambda data: data["platform"]["name"], db_results)) - data_sources = dict.fromkeys(platforms, True) + data_sources = {data["platform"]["name"]: True for data in db_results}

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change

platforms = list(map(lambda data: data["platform"]["name"], db_results))

data_sources = dict.fromkeys(platforms, True)

data_sources = {data["platform"]["name"]: True for data in db_results}

TjitsevdM · 2024-04-24T06:54:43Z

subquery.py

+        github_query_engine = GitHubQueryEngine(community_id=community_id).prepare()
+        tool_metadata = ToolMetadata(
+            name="GitHub",
+            description="Hosts code repositories and project materials from the GitHub platform.",


Still the old description

It might be nice to save all descriptions and other prompts in a single document. Then import them from there whenever we need them like an env var file. That could make updating prompts more easy in the future

Thanks, that was because I started it from the previous branch yesterday. will consider updating it.

It will be resolved when merging with main.

amindadgar added 4 commits April 23, 2024 10:28

feat: Added github query engine!

21ed6d4

fix: linter issues!

261c8af

feat: Enabled querying github data source!

6077677

feat: Added automatic data source selector!

ac73706

amindadgar added 2 commits April 23, 2024 12:27

fix: isort linter issue!

c91ae34

fix: isort linter issue!

1273df9

amindadgar linked an issue Apr 23, 2024 that may be closed by this pull request

Feat: Enable/Disable data source based on community id #51

Closed

3 tasks

coderabbitai bot reviewed Apr 23, 2024

View reviewed changes

amindadgar added 2 commits April 23, 2024 15:08

fix: typo in file name!

84b67c7

fix: imports based on typo fix!

b33526f

coderabbitai bot reviewed Apr 23, 2024

View reviewed changes

amindadgar requested a review from TjitsevdM April 23, 2024 11:47

TjitsevdM reviewed Apr 24, 2024

View reviewed changes

TjitsevdM approved these changes Apr 24, 2024

View reviewed changes

amindadgar merged commit fe7fd5f into main Apr 24, 2024
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/automatic enable disable datasource #52

Feat/automatic enable disable datasource #52

amindadgar commented Apr 23, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Apr 23, 2024 •

edited

Loading

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

coderabbitai bot left a comment

coderabbitai bot Apr 23, 2024

coderabbitai bot left a comment

coderabbitai bot Apr 23, 2024

coderabbitai bot Apr 23, 2024

TjitsevdM Apr 24, 2024

TjitsevdM Apr 24, 2024

amindadgar Apr 24, 2024

amindadgar Apr 24, 2024

	from utils.data_souce_selector import DataSourceSelector
	from utils.data_source_selector import DataSourceSelector

		platforms = list(map(lambda data: data["platform"]["name"], db_results))
		data_sources = dict.fromkeys(platforms, True)

	platforms = list(map(lambda data: data["platform"]["name"], db_results))
	data_sources = dict.fromkeys(platforms, True)
	data_sources = {data["platform"]["name"]: True for data in db_results}

Feat/automatic enable disable datasource #52

Feat/automatic enable disable datasource #52

Conversation

amindadgar commented Apr 23, 2024 • edited by coderabbitai bot Loading

Summary by CodeRabbit

coderabbitai bot commented Apr 23, 2024 • edited Loading

Walkthrough

Changes

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot Apr 23, 2024

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot Apr 23, 2024

Choose a reason for hiding this comment

coderabbitai bot Apr 23, 2024

Choose a reason for hiding this comment

TjitsevdM Apr 24, 2024

Choose a reason for hiding this comment

TjitsevdM Apr 24, 2024

Choose a reason for hiding this comment

amindadgar Apr 24, 2024

Choose a reason for hiding this comment

amindadgar Apr 24, 2024

Choose a reason for hiding this comment

amindadgar commented Apr 23, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Apr 23, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)