Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create periodic replay embedding task #20178

Conversation

daibhin
Copy link
Contributor

@daibhin daibhin commented Feb 7, 2024

Problem

We want to generate embeddings on a periodic schedule

Changes

This sets up one task which runs every five minutes (configurably)
That task returns a list of recordings, each recording is pushed onto a separate celery queue for embedding

The code that does this work is in #20046
and that will internally handle not trying to process too many at a time and measuring what is run so we can tune it

@pauldambra pauldambra marked this pull request as ready for review February 12, 2024 20:28
@pauldambra pauldambra requested a review from a team February 12, 2024 20:28
Copy link
Contributor

github-actions bot commented Feb 12, 2024

Size Change: 0 B

Total Size: 2.22 MB

ℹ️ View Unchanged
Filename Size
frontend/dist/toolbar.js 2.22 MB

compressed-size-action

@pauldambra pauldambra changed the base branch from master to dn-feat/recording-text-embeddings February 12, 2024 20:52
@pauldambra
Copy link
Member

I'm going to merge this into the other PR so there's just one PR - the two aren't quite lining up and it'll be easier in one

@pauldambra pauldambra merged commit 1fbb4da into dn-feat/recording-text-embeddings Feb 12, 2024
69 of 70 checks passed
@pauldambra pauldambra deleted the dn-feat/setup-periodic-replay-embedding branch February 12, 2024 21:43
@pauldambra
Copy link
Member

merged this into #20046

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants