Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow multiple workers for the same queue and configuration #35

Merged
merged 1 commit into from
Nov 3, 2023

Conversation

rosa
Copy link
Member

@rosa rosa commented Nov 3, 2023

From @djmb:

A single worker won't use more than 1 CPU core, so if you need to
dedicate more than on CPU core on a VM to a queue you'll need multiple
workers. I think I'll need this for the new campfire

This changes the configuration format to be like this:

workers:
    - queues: haystack_production_incineration
      processes: 3
      threads: 4
      polling_interval: 2
    - queues: haystack_production_recycling
      threads: 3
      polling_interval: 5

processes is 1 by default, and queues should be now an array of queues (which can include wildcards).

From Donal:

> A single worker won't use more than 1 CPU core, so if you need to
> dedicate more than on CPU core on a VM to a queue you'll need multiple
> workers. I think I'll need this for the new campfire

This changes the configuration format to be like this:

```
workers:
    - queues: haystack_production_incineration
      processes: 3
      threads: 4
      polling_interval: 2
    - queues: haystack_production_recycling
      threads: 3
      polling_interval: 5
```

`processes` is 1 by default, and `queues` should be now an array of
queues (which can include wildcards).
@rosa rosa force-pushed the multiple-processes-per-queue branch from e72cd5a to 425f7bb Compare November 3, 2023 10:31
@rosa rosa merged commit 2843b16 into main Nov 3, 2023
4 checks passed
@rosa rosa deleted the multiple-processes-per-queue branch November 3, 2023 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant