Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update original filename image url for Questagame #770

Open
patkyn opened this issue Nov 5, 2021 · 2 comments
Open

Update original filename image url for Questagame #770

patkyn opened this issue Nov 5, 2021 · 2 comments

Comments

@patkyn
Copy link
Contributor

patkyn commented Nov 5, 2021

There are about 252,689 records in DB with http://api.questagame.com in original_filename for dr1902. Because, the raw csv file from Questagame are sent with 'https://api.questagame.com', pipelines treat these as new images due to url change and exported to image-service as new images to upload. Large number of images in the image-service queue had caused some performance issues in image-service.

@sadeghim
Copy link
Member

Maybe this AtlasOfLivingAustralia/image-service#164 can help to solve the problem in future.

@patkyn
Copy link
Contributor Author

patkyn commented Nov 11, 2021

As communicated with Questagame, the move to switch to https is a once-off effort and it won't be switched back and forth. However, Questagame also indicated that there will be a move to switch to Amazon S3 some time in future and this will be communicated to us before they switch as they're aware of this issue that we are facing.

Updated backend postgres with a script that does the batch updates of 500 records it takes about 10 minutes to execute to update 250k records for dr1902

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants