Dagster Execution Error due to simultaneous config read

[webb-ben]

I'm encountering an issue where multiple processes are attempting to read the same Docker configuration simultaneously, which seems to be causing failures in my Dagster pipeline. Specifically, the error is triggered when the docker_client_environment op tries to remove a configuration that is still in use by another service.

Here’s the error stack trace:

dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "docker_client_environment":

  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_plan.py", line 245, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 499, in core_dagster_event_sequence_for_step
    for user_event in _step_output_error_checked_user_event_sequence(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 183, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 87, in _process_asset_results_to_events
    for user_event in user_event_sequence:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/compute.py", line 193, in execute_core_compute
    for step_output in _yield_compute_results(step_context, inputs, compute_fn, compute_context):
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/compute.py", line 162, in _yield_compute_results
    for event in iterate_with_context(
  File "/usr/local/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 480, in iterate_with_context
    with context_fn():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 84, in op_execution_error_boundary
    raise error_cls(

The above exception was caused by the following exception:
docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.47/configs/dbl7kpwgzovni5pa6yn8wbrmy: Bad Request ("rpc error: code = InvalidArgument desc = config 'gleaner' is in use by the following service: ref_dams_dams__1_gleaner")

  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/usr/local/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 482, in iterate_with_context
    next_output = next(iterator)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 140, in _coerce_op_compute_fn_to_iterator
    result = invoke_compute_fn(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 128, in invoke_compute_fn
    return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
  File "/opt/dagster/app/code/main.py", line 178, in docker_client_environment
    api_client.remove_config(gleanerconfig[0].id)
  File "/usr/local/lib/python3.10/site-packages/docker/utils/decorators.py", line 32, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/docker/api/config.py", line 74, in remove_config
    self._raise_for_status(res)
  File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 277, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
  File "/usr/local/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e

The above exception was caused by the following exception:
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.47/configs/dbl7kpwgzovni5pa6yn8wbrmy

  File "/usr/local/lib/python3.10/site-packages/docker/api/client.py", line 275, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

[C-Loftus]

Should be fixed by #53