Feat add hivemind etl scripts #15
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- The discord vectorstore, and discord summary indexes are finished. - The discourse vectorstore is at its final stages, just needed the data on staging to be updated. - some gdrive codes are written but they need to be completed and its test cases should be updated.
the parts were related to discourse.
TjitsevdM
reviewed
Dec 13, 2023
TjitsevdM
reviewed
Dec 13, 2023
TjitsevdM
reviewed
Dec 13, 2023
TjitsevdM
reviewed
Dec 13, 2023
TjitsevdM
approved these changes
Dec 13, 2023
There was a problem hiding this comment.
Great work! I left some small comments here and there and also the metadata comment from the conversation in discord conversation: https://discord.com/channels/915914985140531240/1184488640743735297/1184530840261251182.
For the rest, everything looks good so I'll already submit my approval
It seems airlfow in docker was having trouble when having that library. It was the hdbscan not installing which was a sub-dependency of phoenix.
we're now having a prefix and per thread, channel and day it would be updated.
It seems the `DiscoursePosts` with raw equal to NULL, were deleted before.
In case of 256 the documents with long metadata was producing error.
- If a channel only has a single thread, the thread summary is stored as the channel summary as well. - If a server only has a single channel, the channel summary is stored as the server summary as well.
moved some discord summarizer codes to utils as it could be used across multiple summarizers.
In previous commits we were assuming a post can have multiple category as we're doing a COLLECT on DiscourseCategory.name
TjitsevdM
reviewed
Dec 14, 2023
TjitsevdM
approved these changes
Dec 14, 2023
- Also, cleaning codes with isort and black
they are <@&[role_id]>
the linter is giving error for the pdf part of the lib. trying to adjust a version and see if the textlint ignore that.
it was needed for google drive etl which isn't fully implemented for now.
…rse-summary feat: hivemind etl discourse summary and CI
TjitsevdM
reviewed
Dec 20, 2023
There was a problem hiding this comment.
Is google drive vector store and summary already fully implemented or is this just a first start?
There was a problem hiding this comment.
No, it isn't implemented. The written scripts were just a first start.
TjitsevdM
approved these changes
Dec 20, 2023
cyri113
approved these changes
Dec 21, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this PR we're aiming to add
Notes: