resources batch load of multiple GIF images can lock up a manifold server

[hardyoyo]

In submitting a resource batch load process, which included many animated GIFs, we observed our Manifold instance peg two CPUs and remain at a load average of 15 for several minutes. It eventually cooled off, but it was nerve-wracking to watch. There should be a way to throttle the image processing queue to keep the CPU load workable.

For reproduction purposes, the google sheet of the metadata we're using is here:
https://docs.google.com/spreadsheets/d/1ATwb0Jf4dbNqFZd7faRnLEqq898I662G9AnKNRvM6Ps/edit?usp=sharing

and the drive folder is here:
https://drive.google.com/drive/folders/1Uj81gzMflKiyeTObg5npZ7aoIby3d6hz?usp=sharing

[zdavis]

Hey Hardy, thanks for the issue. It looks like the drive folder has been deleted, so I wasn't able to test and try to reproduce.

That said, image processing jobs are dropped into Sidekiq, which is what Manifold uses for background processing.

@scryptmouse, you have a better understanding of tuning Sidekiq than I do and perhaps you can weigh in here. Right now, we start Sidekiq with this runnit service definition:

https://github.com/ManifoldScholar/manifold-omnibus/blob/master/cookbooks/manifold/templates/default/sv-sidekiq-run.erb

Instances can manage the concurrency, which defaults to 25 threads, here: https://github.com/ManifoldScholar/manifold-omnibus/blob/master/files/manifold-config-template/manifold.rb#L100

I don't set us setting the number of workers anywhere—I suppose I'd expect to see it here: https://github.com/ManifoldScholar/manifold-omnibus/blob/ff9f4b786d97853618ab4530cd9da45370e237f2/cookbooks/manifold/templates/default/api-env.erb

Alexa, do you have any recommendations for how we might better tune our default sidekiq settings in the omnibus package? Would be nice to make improvements before the impending v6 release.

It looks to me like we only have a single Sidekiq worker, as I don't see