Request for reaper documentation

[RobsonKing]

Describe the bug
This is a request for documentation more than a bug. I cannot find anything that tells me what the reaper process does or why I might need it.

Maybe you feel like this is something internal to the implantation, and users of the gem should not be required to understand this to use it? My problem is that sometimes we have jobs that are being blocked from starting (they end up on the retry queue because of our :raise conflict strategy) but there is no job currently running. When we look at the list of locks, it appears like there is a lock. But why? Is it orphaned? Should the reaper be cleaning it up? If I delete the lock... the job starts.

I am trying to understand enough of the process so I know if I have a problem with my implementation, or if what I am seeing is expected.

1. What does the reaper process do? My understanding is that it cleans ups locks that are orphaned.

2. What are orphaned locks in the context of sidekiq unique jobs? Are they locks for jobs that are no longer running, or is it related to lock timeouts?

3. What is the reason that locks became orphaned? Are they expected? Or should they only occur in rare situations (restarts, crashes, etc)

4. What affect do orphan locks have? Do they block jobs from running or being enqueued?

5. Is the reaper on by default? Looking through the source I believe the answer is yes.

6. Can you tell the reaper is running, or has run? If I think a job is not running because of an orphan lock, can I tell if the reaper has run recently? I watch the logs and sometimes I see uniquejobs=reaper INFO: Starting Reaper

7. While the reaper is running, what is the affect on sidekiq? Can jobs still be enqueued and run? I have seen some issues logged talking about the performance with lots of jobs. My understanding of the documentation is that if I use :lua then sidekiq will be locked. But If I use :ruby it might be slow. What is the affect of a slow running reaper?

8. If I turn off the reaper, what is the affect. If we do not clean up orphans, does this stop jobs from running? Will jobs be blocked forever?

9. What is the reaper resurrector? Reading through issues, it seems like this brings the reaper back if it dies. How long should this take? Can I tell if the reaper is dead and might need resurrection? We can experience periods of large load (sometimes for several hours), if the reaper is unable to run, could I have jobs that are blocked from starting, or even blocked from being scheduled?

10. In the logs for one of my queues I see
    | 2021-12-17T21:57:19.255Z pid=1 tid=gpkz0pj65 uniquejobs=upgrade_locks INFO: Skipping upgrade because uniquejobs:dead has been set
    but I am unable to find any information on what that might mean.

Thanks so much for all the work you do on this gem. Maybe someone who is using this functionality could help fill in some of the documentation?

[mhenrixon]

1. What does the reaper process do? My understanding is that it cleans ups locks that are orphaned.

It indeed cleans up orphaned locks

2. What are orphaned locks in the context of sidekiq unique jobs? Are they locks for jobs that are no longer running, or is it related to lock timeouts?

Jobs that were completed but for whatever reason the lock wasn't cleaned up.

3. What is the reason that locks became orphaned? Are they expected? Or should they only occur in rare situations (restarts, crashes, etc)

The most common use case for this is the restart of sidekiq, or killing the sidekiq server process. There are a lot of things that can go wrong in a distributed redis setup and the reaper just makes sure your sidekiq won't choke forever on one of those edge cases.

4. What affect do orphan locks have? Do they block jobs from running or being enqueued?

Correct, they would block jobs from either being run or enqueued (depends on the lock type).

5. Is the reaper on by default? Looking through the source I believe the answer is yes.

It is on by default and since I added it, I have far fewer complaints about the lock mechanism :)

6. Can you tell the reaper is running, or has run? If I think a job is not running because of an orphan lock, can I tell if the reaper has run recently? I watch the logs and sometimes I see uniquejobs=reaper INFO: Starting Reaper

There would be also a log entry like: Execution successfully returned, Execution timed out or Cleanup failed with error message every time the reaper runs.

7. While the reaper is running, what is the affect on sidekiq? Can jobs still be enqueued and run? I have seen some issues logged talking about the performance with lots of jobs. My understanding of the documentation is that if I use :lua then sidekiq will be locked. But If I use :ruby it might be slow. What is the affect of a slow running reaper?

You will reap less jobs but to be honest, the ruby reaper is the only one that makes sense. The lua reaper should most likely be removed.

It is a separate process completely (from sidekiq) and should not affect Sidekiq at all. At least not for the worse.

8. If I turn off the reaper, what is the affect. If we do not clean up orphans, does this stop jobs from running? Will jobs be blocked forever?

Exactly, the risk is that you run into situations where a lock is kept forever (or until manually cleaned up; which isn't a super straight forward process).

9. What is the reaper resurrector? Reading through issues, it seems like this brings the reaper back if it dies. How long should this take? Can I tell if the reaper is dead and might need resurrection? We can experience periods of large load (sometimes for several hours), if the reaper is unable to run, could I have jobs that are blocked from starting, or even blocked from being scheduled?

The resurrector was something someone added because their reaper process crashed and therefor they could not start new jobs (they had a bunch of orphaned locks). Some other people reported that the resurrector itself was causing problems so I disabled it by default.

10. In the logs for one of my queues I see
    2021-12-17T21:57:19.255Z pid=1 tid=gpkz0pj65 uniquejobs=upgrade_locks INFO: Skipping upgrade because uniquejobs:dead has been set but I am unable to find any information on what that might mean.

Just means that there is nothing to upgrade.

[jpriollaud]

I was looking in the readme for this kind of explanation :| This library kicks ass once you understand how it works. The documentation doesn't do it justice.

[mhenrixon]

Thank you @jpriollaud perhaps it is time to rewrite the readme.