Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Don't store job in JobIO instance attributes
and invalidate `_output_hdas_and_paths` when current session is not the same session that was used to populate `_output_hdas_and_paths`. The Job instance may originate from a session that is associated to another thread, and when that thread closes the session the instance (along with other instances retrieved through loading relationships) becomes detached. I am not sure at all if this will fix ``` DetachedInstanceError: Instance <HistoryDatasetAssociation at 0x7fe68bbf14f0> is not bound to a Session; attribute refresh operation cannot proceed (Background on this error at: https://sqlalche.me/e/14/bhk3) File "galaxy/jobs/runners/__init__.py", line 291, in prepare_job job_wrapper.prepare() File "galaxy/jobs/__init__.py", line 1248, in prepare tool_evaluator.set_compute_environment(compute_environment, get_special=get_special) File "galaxy/tools/evaluation.py", line 162, in set_compute_environment self.param_dict = self.build_param_dict( File "galaxy/tools/evaluation.py", line 204, in build_param_dict self.__populate_output_dataset_wrappers(param_dict, output_datasets, job_working_directory) File "galaxy/tools/evaluation.py", line 447, in __populate_output_dataset_wrappers param_dict[name] = DatasetFilenameWrapper( File "galaxy/tools/wrappers.py", line 403, in __init__ path_rewrite = compute_environment and compute_environment.output_path_rewrite(dataset_instance) File "galaxy/job_execution/compute_environment.py", line 132, in output_path_rewrite return str(self.job_io.get_output_path(dataset)) File "galaxy/job_execution/setup.py", line 226, in get_output_path if hda.id == dataset.id: File "sqlalchemy/orm/attributes.py", line 487, in __get__ return self.impl.get(state, dict_) File "sqlalchemy/orm/attributes.py", line 959, in get value = self._fire_loader_callables(state, key, passive) File "sqlalchemy/orm/attributes.py", line 990, in _fire_loader_callables return state._load_expired(state, passive) File "sqlalchemy/orm/state.py", line 712, in _load_expired self.manager.expired_attribute_loader(self, toload, passive) File "sqlalchemy/orm/loading.py", line 1369, in load_scalar_attributes raise orm_exc.DetachedInstanceError( ``` but it seems to make some sense. JobIO crosses thread boundaries as part of the job wrapper getting put into threading queues. Ideally we'd make sure that no ORM instance crosses the thread boundary (or we systematically re-associated with a session). I also tried flagging these patterns automatically using something like: ``` @event.listens_for(session, "persistent_to_detached") def on_detach(sess, instance): if not getattr(instance, "allow_detatch", False): raise Exception(f"{instance} detached. This ain't good for how we do things ?") ``` but it seems tricky to figure out when this is fine and when it is not.
- Loading branch information