silverback worker does not use independent processes per worker

[fubuloubu]

What went wrong?

When testing with silverback worker -w 3 we noticed that it does not use separate process IDs per worker:

$ silverback worker -w 3 example:app ...
...
SUCCESS: Loaded Silverback App:
  ...
worker_startup process ID 128903
worker_startup process ID 128903
worker_startup process ID 128903

(using os.getpid to print the process ID)

How can it be fixed?

Debug cli implementation of silverback worker

silverback/silverback/_cli.py

Lines 130 to 132 in 29c24f1

	def worker(cli_ctx, account, workers, max_exceptions, shutdown_timeout, path):
	app = import_from_string(path)
	asyncio.run(run_worker(app.broker, worker_count=workers, shutdown_timeout=shutdown_timeout))

[linear]

SBK-430 `silverback worker` does not use independent processes per worker

[fubuloubu]

More testing showing thread ID:

SUCCESS: Loaded Silverback App:
  ...
worker_startup process ID 132726 thread ID 132726
worker_startup process ID 132726 thread ID 132726
worker_startup process ID 132726 thread ID 132726
INFO: Found cached value ...
app_startup process ID 132726 thread ID 132908
SUCCESS: app_startup - 0.000s (0.0%)
exec_block process ID 132726 thread ID 132908
exec_event1 process ID 132726 thread ID 132726
SUCCESS: exec_block[...]
SUCCESS: exec_event1[...]
exec_event1 process ID 132726 thread ID 132726
SUCCESS: exec_event1[...]
exec_event1 process ID 132726 thread ID 132726
exec_block process ID 132726 thread ID 132908
SUCCESS: exec_block[...]
SUCCESS: exec_event1[...]
exec_event1 process ID 132726 thread ID 132726
Exception found while executing function:  ...
ERROR: exec_event1[...]
exec_event1 process ID 132726 thread ID 132726
SUCCESS: exec_event1[...]
exec_event1 process ID 132726 thread ID 132726
exec_block process ID 132726 thread ID 132908
SUCCESS: exec_block[...]
SUCCESS: exec_event1[...]
app_shutdown process ID 132726 thread ID 132908
SUCCESS: app_shutdown - 0.000s (0.0%)
exec_event1 process ID 132726 thread ID 132726
SUCCESS: exec_event1[...]
^CShutting down the broker.
worker_shutdown process ID 132726 thread ID 132726

[mikeshultz]

Ah, I completely forgot I sort of reimplemented this. Though I mostly recall copying it from TaskIQ's code. Still the same issue with threads though, no?

[fubuloubu]

Ah, I completely forgot I sort of reimplemented this. Though I mostly recall copying it from TaskIQ's code. Still the same issue with threads though, no?

It does seem to use separate threads, although I think it's a bit bound by how we do handle the event/block streams in the runner