-
Notifications
You must be signed in to change notification settings - Fork 963
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BigQuery bigquery-public-data.pypi.distribution_metadata
missing data
#16008
Comments
The task that ensures consistency was disabled due to poor performance in... 2021 🙃 But was never subsequently re-enabled that I can tell, as the contributor never returned to address the issue. For triage, I have manually run this task, can you confirm if you're seeing consistency? |
@ewdurbin was all the data synced? That is, should all the historical gaps be filled now? When I query Is there any other endpoint to get the recent releases data? |
@ewdurbin I'm still not seeing the newer releases of |
Hmmm, unclear what the issue is. @di are you familiar with why the sync wouldn't capture past releases? |
That's not the job that inserts new metadata, that job just syncs missing metadata if insertion fails for some reason. Insertion of new metadata happens on upload: https://github.com/pypi/warehouse/blob/main/warehouse/forklift/legacy.py#L1222-L1223 The timeline here is suspiciously close to when we did some migrations on these schemas, my guess is that the |
So |
It is, but it shouldn't be necessary anymore, metadata should be reliably getting inserted on upload (but it appears it isn't anymore). |
hm, okay I ran |
Probably failing for the same reason the individual job is failing I would venture a guess! |
Running this query:
misses several new versions available here: https://pypi.org/project/virtualenv/#history released in April and May. It's similar for some other packages.
Describe the bug
All versions info should be available in BigQuery.
Expected behavior
I would expect them (except eventual consistency ofc) to be available in BQ.
To Reproduce
Run in BigQuery:
and see versions are missing.
The text was updated successfully, but these errors were encountered: