Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch Support #142

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
Draft

Batch Support #142

wants to merge 15 commits into from

Commits on Sep 26, 2024

  1. Batch job POC

    * Introduces a "batch" concept, similar to batches present in Sidekiq Pro and GoodJob
    
    * Batches monitor a set of jobs, and when those jobs are completed can fire off a final job
    
    * This introduces a SolidQueue::JobBatch model, as well as the ability to enqueue jobs and associate them with the batch
    
    * There are still more ideas to figure out, but this provides a basic batch scaffolding to spark discussion
    jpcamara committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    db7d651 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    308775e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    52086ac View commit details
    Browse the repository at this point in the history
  4. Allow batch jobs to be instances

    * This means we can store the arguments and settings by letting the user do `BatchJob.new(arguments).set(options)`
    
    * Yield the batch in `enqueue` in case someone needs info from it
    
    * When you serialize then deserialize an activejob instance, the arguments are in the serialized_arguments field and can only be transferred over by the private method `deserialize_arguments_if_needed`. This is pretty janky, so there is probably something i'm missing
    
    * `perform_all_later` let's us do a perform_later even with instance, which does not seem to be possible on the instances themselves
    jpcamara committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    1057152 View commit details
    Browse the repository at this point in the history
  5. Make sure batch is still first arg of the batch callback

    * Add spec for adding arguments and options to the batch callback
    jpcamara committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    c8d784d View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    e49ee43 View commit details
    Browse the repository at this point in the history
  7. Rubocop changes

    * Spacing, double quotes
    
    * Support Ruby < 3.2 by removing the implicit key/variable syntax
    jpcamara committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    b8badaa View commit details
    Browse the repository at this point in the history
  8. Handle on_failure and on_success

    * on_failure fires the first time any of the jobs fail, even once
    
    * on_success only fires if all jobs work (after retries)
    
    * remove unused job_id
    jpcamara committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    0f16d1c View commit details
    Browse the repository at this point in the history
  9. Allow enqueueing into a batch instance

    * Allows enqueueing a job within a job, as part of the batch
    jpcamara committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    9cd7d5b View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    b494353 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    7c1c31d View commit details
    Browse the repository at this point in the history
  12. Support nested batches

    * Parent batches will not complete until all child batches have been completed
    jpcamara committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    07821d6 View commit details
    Browse the repository at this point in the history
  13. Expanded batch readme

    jpcamara committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    1a60a9e View commit details
    Browse the repository at this point in the history
  14. Force an initial batch check

    jpcamara committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    b4d3f8d View commit details
    Browse the repository at this point in the history
  15. Initial batch lifecycle tests

    * Attach success jobs to the parent batch, not to the current batch (which has already finished at this point)
    jpcamara committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    1a0846a View commit details
    Browse the repository at this point in the history