Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot import RO-Crate of an Invocation using Collection as input #18927

Closed
patrick-austin opened this issue Oct 2, 2024 · 3 comments
Closed

Cannot import RO-Crate of an Invocation using Collection as input #18927

patrick-austin opened this issue Oct 2, 2024 · 3 comments
Assignees
@davelopez
Labels
area/backend kind/bug

Comments

@patrick-austin
Copy link
Contributor

Describe the bug
When attempting to import an exported RO-Crate for a Workflow Invocation that uses a Dataset List Collection as input, an error is returned to the UI: History import failed.

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 24.1.1
Commit: 940a87c8bf248c0d1cecd61b2e19fd37ce097777

Also tried on usegalaxy.eu and got the same error.
Also tried on 23.2, and the import did not succeed but also did not error (job stays running indefinitely).

Browser and Operating System
Operating System: Windows
Browser: Firefox

To Reproduce
Steps to reproduce the behavior:

  1. Import an invocation with Dataset List Collection as input
    Invocation-of-Input-Dataset-Collection-Test-at-2024-10-02T170508077452.rocrate.zip

Since this will fail, I also provide the simplest example workflow I could think of (used in the above):
Galaxy-Workflow-Input_Dataset_Collection_Test.txt
This should be a .ga but had to rename to .txt to allow upload to GitHub.

Expected behavior
Import to succeed.

Screenshot
image

Additional context
Ultimately, the error seems to be:

CircularDependencyError
Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70dc3fd280>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70dc3fd280>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70dc3fd280>, delete=False))

We found this in our log files, wrapped in a SQLAlchemy rollback error. I don't have the knowledge of Galaxy server to dig any further into the cause, but we did a lot of trial and error on our instance and the presence of the Collection as input seemed to be the common link, and is referenced in the error message above.
@bgruening
Copy link
Member

bgruening commented Oct 6, 2024 •
edited
Loading

@patrick-austin do you have the entire stacktrace? Thanks for the detailed description!

@patrick-austin
Copy link
Contributor Author

After some more experimentation it looks like this might only be a problem when using the "Send results to a new history" option. Maybe there's a problem in how the metadata is formatted at export/import time for collections that are sent to a new history?

For a "real" example we had, not sending to new history resulted in an RO-Crate that could be re-imported. But for the example I gave above it seems to fail with a different error, this time (see bottom). An even more simple example that doesn't even run a tool does work though:

Invocation without sending to new history (should work):
Invocation-of-Collection-input-only-at-2024-10-07T101050454964.rocrate.zip

Invocaion with sending to new history (doesn't work for me):
Invocation-of-Collection-input-only-at-2024-10-07T102428277441.rocrate.zip

Apologies for not including the full stack for the original error, it seemed to be a lot of SQLAlchemy rolling back for the same underlying error but perhaps there is something useful in there:

galaxy.workflow.run_request INFO 2024-10-04 15:19:44,279 [pN:main.1,p:3238871,tN:AnyIO worker thread] Creating a step_state for step.id 9295
galaxy.workflow.run_request INFO 2024-10-04 15:19:44,280 [pN:main.1,p:3238871,tN:AnyIO worker thread] Creating a step_state for step.id 9296
galaxy.web_stack.handlers INFO 2024-10-04 15:19:44,281 [pN:main.1,p:3238871,tN:AnyIO worker thread] (WorkflowInvocation[unflushed]) Handler '_default_' assigned using 'HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED' assignment method
multipart.multipart DEBUG 2024-10-04 15:20:15,007 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_begin with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_field with data[60:79]
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_value with data[81:193]
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_end with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_field with data[195:207]
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_value with data[209:224]
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_end with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,008 [pN:main.1,p:3238871,tN:MainThread] Calling on_headers_finished with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,010 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_data with data[228:208896]
multipart.multipart DEBUG 2024-10-04 15:20:15,011 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_data with data[0:56404]
multipart.multipart DEBUG 2024-10-04 15:20:15,011 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_end with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_begin with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_field with data[56466:56485]
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_value with data[56487:56519]
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_header_end with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_headers_finished with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_part_end with no data
multipart.multipart DEBUG 2024-10-04 15:20:15,012 [pN:main.1,p:3238871,tN:MainThread] Calling on_end with no data
galaxy.web_stack.handlers INFO 2024-10-04 15:20:15,052 [pN:main.1,p:3238871,tN:AnyIO worker thread] Flushed transaction for Job[id=17137,tool_id=__IMPORT_HISTORY__] (10.270 ms)
galaxy.web_stack.handlers INFO 2024-10-04 15:20:15,052 [pN:main.1,p:3238871,tN:AnyIO worker thread] (Job[id=17137,tool_id=__IMPORT_HISTORY__]) Handler '_default_' assigned using 'HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED' assignment method
galaxy.jobs.handler DEBUG 2024-10-04 15:20:15,446 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] Grabbed Job(s): 17137
galaxy.jobs.mapper DEBUG 2024-10-04 15:20:15,469 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] (17137) Mapped job to destination id: local_dest
galaxy.jobs.handler DEBUG 2024-10-04 15:20:15,479 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] (17137) Dispatching to local_runner runner
galaxy.jobs DEBUG 2024-10-04 15:20:15,492 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] (17137) Persisting job destination (destination id: local_dest)
galaxy.jobs DEBUG 2024-10-04 15:20:15,508 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] (17137) Working directory for job is: /mnt/nfs/data/job_work_dir/017/17137
galaxy.jobs.runners DEBUG 2024-10-04 15:20:15,517 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] Job [17137] queued (38.066 ms)
galaxy.jobs.handler INFO 2024-10-04 15:20:15,522 [pN:main.1,p:3238871,tN:JobHandlerQueue.monitor_thread] (17137) Job dispatched
galaxy.jobs DEBUG 2024-10-04 15:20:15,591 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] Job wrapper for Job [17137] prepared (60.970 ms)
galaxy.jobs.command_factory INFO 2024-10-04 15:20:15,638 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] Built script [/mnt/nfs/data/job_work_dir/017/17137/tool_script.sh] for tool command [python '/srv/galaxy/server/lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py' 'L3Nydi9nYWxheHkvdmFyL3RtcC90bXBlOXhzN2JkZS56aXA=' 'L3Nydi9nYWxheHkvdmFyL3RtcC90bXAxcl8xZ3RscQ==' --file --file-sources '/mnt/nfs/data/job_work_dir/017/17137/configs/tmpx548bpin' --encoded]
galaxy.jobs.runners DEBUG 2024-10-04 15:20:15,708 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] (17137) command is: mkdir -p working outputs configs
if [ -d _working ]; then
    rm -rf working/ outputs/ configs/; cp -R _working working; cp -R _outputs outputs; cp -R _configs configs
else
    cp -R working _working; cp -R outputs _outputs; cp -R configs _configs
fi
cd working; /bin/bash /mnt/nfs/data/job_work_dir/017/17137/tool_script.sh > '../outputs/tool_stdout' 2> '../outputs/tool_stderr'; return_code=$?; echo $return_code > /mnt/nfs/data/job_work_dir/017/17137/galaxy_17137.ec; cd '/mnt/nfs/data/job_work_dir/017/17137'; 
[ "$GALAXY_VIRTUAL_ENV" = "None" ] && GALAXY_VIRTUAL_ENV="$_GALAXY_VIRTUAL_ENV"; _galaxy_setup_environment True; python metadata/set.py; sh -c "exit $return_code"
galaxy.jobs.runners.local DEBUG 2024-10-04 15:20:15,763 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] (17137) executing job script: /mnt/nfs/data/job_work_dir/017/17137/galaxy_17137.sh
galaxy.jobs.runners.util.process_groups DEBUG 2024-10-04 15:20:24,081 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] check_pg(): No process found in process group 3827210
galaxy.jobs.runners.local DEBUG 2024-10-04 15:20:24,085 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] execution finished: /mnt/nfs/data/job_work_dir/017/17137/galaxy_17137.sh
galaxy.jobs ERROR 2024-10-04 15:20:24,219 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] exec_after_process hook failed for job 17137
Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 81, in cleanup_after_job
    model_store.perform_import(new_history, job=job, new_history=True)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 418, in perform_import
    self._import_jobs(object_import_tracker, history)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1224, in _import_jobs
    self._flush()
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1330, in _flush
    self.sa_session.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 597, in commit
    return self._proxied.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2017, in commit
    trans.commit(_to_root=True)
  File "<string>", line 2, in commit
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1302, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl
    self.session.flush()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4341, in flush
    self._flush(objects)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4477, in _flush
    transaction.rollback(_capture_exception=True)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush
    flush_context.execute()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 457, in execute
    for subset in topological.sort_as_subsets(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/topological.py", line 47, in sort_as_subsets
    raise CircularDependencyError(
sqlalchemy.exc.CircularDependencyError: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2039, in finish
    task_wrapper = self.tool.exec_after_process(
  File "/srv/galaxy/server/lib/galaxy/tools/__init__.py", line 3100, in exec_after_process
    JobImportHistoryArchiveWrapper(self.app, job.id).cleanup_after_job()
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 87, in cleanup_after_job
    jiha.job.tool_stderr += f"Error cleaning up history import job: {e}"
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 566, in __get__
    return self.impl.get(state, dict_)  # type: ignore[no-any-return]
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1121, in _fire_loader_callables
    return self.callable_(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 927, in _load_for_state
    primary_key_identity = self._get_ident_for_use_get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 989, in _get_ident_for_use_get
    return [
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 990, in <listcomp>
    get_attr(state, dict_, self._equated_columns[pk], passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3581, in _get_state_attr_by_column
    return state.manager[prop.key].impl.get(state, dict_, passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1116, in _fire_loader_callables
    return state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)
galaxy.jobs.runners ERROR 2024-10-04 15:20:24,264 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] (17137/3827210) Job wrapper finish method failed
Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 81, in cleanup_after_job
    model_store.perform_import(new_history, job=job, new_history=True)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 418, in perform_import
    self._import_jobs(object_import_tracker, history)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1224, in _import_jobs
    self._flush()
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1330, in _flush
    self.sa_session.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 597, in commit
    return self._proxied.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2017, in commit
    trans.commit(_to_root=True)
  File "<string>", line 2, in commit
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1302, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl
    self.session.flush()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4341, in flush
    self._flush(objects)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4477, in _flush
    transaction.rollback(_capture_exception=True)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush
    flush_context.execute()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 457, in execute
    for subset in topological.sort_as_subsets(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/topological.py", line 47, in sort_as_subsets
    raise CircularDependencyError(
sqlalchemy.exc.CircularDependencyError: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2039, in finish
    task_wrapper = self.tool.exec_after_process(
  File "/srv/galaxy/server/lib/galaxy/tools/__init__.py", line 3100, in exec_after_process
    JobImportHistoryArchiveWrapper(self.app, job.id).cleanup_after_job()
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 87, in cleanup_after_job
    jiha.job.tool_stderr += f"Error cleaning up history import job: {e}"
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 566, in __get__
    return self.impl.get(state, dict_)  # type: ignore[no-any-return]
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1121, in _fire_loader_callables
    return self.callable_(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 927, in _load_for_state
    primary_key_identity = self._get_ident_for_use_get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 989, in _get_ident_for_use_get
    return [
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 990, in <listcomp>
    get_attr(state, dict_, self._equated_columns[pk], passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3581, in _get_state_attr_by_column
    return state.manager[prop.key].impl.get(state, dict_, passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1116, in _fire_loader_callables
    return state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 676, in _finish_or_resubmit_job
    job_wrapper.finish(
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2044, in finish
    return fail("exec_after_process hook failed", exception=e)
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1849, in fail
    return self.fail(
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1396, in fail
    job = self.get_job()
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1188, in get_job
    job = self.sa_session.get(Job, self.job_id)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 1060, in get
    return self._proxied.get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3682, in get
    return self._get_impl(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3826, in _get_impl
    instance = self._identity_lookup(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2999, in _identity_lookup
    return_value = loading.get_from_identity(self, mapper, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 478, in get_from_identity
    state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)
galaxy.jobs.runners ERROR 2024-10-04 15:20:24,327 [pN:main.1,p:3238871,tN:LocalRunner.work_thread-2] (17137) Unhandled exception calling queue_job
Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 81, in cleanup_after_job
    model_store.perform_import(new_history, job=job, new_history=True)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 418, in perform_import
    self._import_jobs(object_import_tracker, history)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1224, in _import_jobs
    self._flush()
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1330, in _flush
    self.sa_session.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 597, in commit
    return self._proxied.commit()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2017, in commit
    trans.commit(_to_root=True)
  File "<string>", line 2, in commit
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1302, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl
    self.session.flush()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4341, in flush
    self._flush(objects)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4477, in _flush
    transaction.rollback(_capture_exception=True)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush
    flush_context.execute()
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 457, in execute
    for subset in topological.sort_as_subsets(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/util/topological.py", line 47, in sort_as_subsets
    raise CircularDependencyError(
sqlalchemy.exc.CircularDependencyError: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2039, in finish
    task_wrapper = self.tool.exec_after_process(
  File "/srv/galaxy/server/lib/galaxy/tools/__init__.py", line 3100, in exec_after_process
    JobImportHistoryArchiveWrapper(self.app, job.id).cleanup_after_job()
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 87, in cleanup_after_job
    jiha.job.tool_stderr += f"Error cleaning up history import job: {e}"
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 566, in __get__
    return self.impl.get(state, dict_)  # type: ignore[no-any-return]
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1121, in _fire_loader_callables
    return self.callable_(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 927, in _load_for_state
    primary_key_identity = self._get_ident_for_use_get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 989, in _get_ident_for_use_get
    return [
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 990, in <listcomp>
    get_attr(state, dict_, self._equated_columns[pk], passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3581, in _get_state_attr_by_column
    return state.manager[prop.key].impl.get(state, dict_, passive=passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 1116, in _fire_loader_callables
    return state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 676, in _finish_or_resubmit_job
    job_wrapper.finish(
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2044, in finish
    return fail("exec_after_process hook failed", exception=e)
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1849, in fail
    return self.fail(
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1396, in fail
    job = self.get_job()
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1188, in get_job
    job = self.sa_session.get(Job, self.job_id)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 1060, in get
    return self._proxied.get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3682, in get
    return self._get_impl(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3826, in _get_impl
    instance = self._identity_lookup(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2999, in _identity_lookup
    return_value = loading.get_from_identity(self, mapper, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 478, in get_from_identity
    state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 174, in run_next
    method(arg)
  File "/srv/galaxy/server/lib/galaxy/jobs/runners/local.py", line 155, in queue_job
    self._finish_or_resubmit_job(job_state, stdout, stderr, job_id=job_id)
  File "/srv/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 686, in _finish_or_resubmit_job
    job_wrapper.fail("Unable to finish job", exception=True)
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1396, in fail
    job = self.get_job()
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 1188, in get_job
    job = self.sa_session.get(Job, self.job_id)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/scoping.py", line 1060, in get
    return self._proxied.get(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3682, in get
    return self._get_impl(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 3826, in _get_impl
    instance = self._identity_lookup(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2999, in _identity_lookup
    return_value = loading.get_from_identity(self, mapper, key, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 478, in get_from_identity
    state._load_expired(state, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state.py", line 796, in _load_expired
    self.manager.expired_attribute_loader(self, toload, passive)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
    result = load_on_ident(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
    return load_on_pk_identity(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
    session.execute(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2226, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 2095, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 103, in _go
    self._raise_for_prerequisite_state(fn.__name__, current_state)
  File "/srv/galaxy/venv/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 973, in _raise_for_prerequisite_state
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: Circular dependency detected. (SaveUpdateState(<HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>), ProcessState(OneToManyDP(HistoryDatasetCollectionAssociation.copied_to_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False), ProcessState(ManyToOneDP(HistoryDatasetCollectionAssociation.copied_from_history_dataset_collection_association), <HistoryDatasetCollectionAssociation at 0x7f70ba0dad90>, delete=False)) (Background on this error at: https://sqlalche.me/e/20/7s2a)

As for trying to do a dataset extraction, re-importing the invocation, e.g.:
Invocation-of-Input-Dataset-Collection-Test-at-2024-10-07T084853539674.rocrate.zip

fails with a different error:

galaxy.web_stack.handlers INFO 2024-10-07 11:10:12,231 [pN:main.1,p:3238871,tN:AnyIO worker thread] Flushed transaction for Job[id=17505,tool_id=__IMPORT_HISTORY__] (10.455 ms)
galaxy.web_stack.handlers INFO 2024-10-07 11:10:12,231 [pN:main.1,p:3238871,tN:AnyIO worker thread] (Job[id=17505,tool_id=__IMPORT_HISTORY__]) Handler '_default_' assigned using 'HANDLER_ASSIGNMENT_METHODS.DB_SKIP_LOCKED' assignment method
galaxy.jobs.handler DEBUG 2024-10-07 11:10:12,477 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] Grabbed Job(s): 17505
galaxy.jobs.mapper DEBUG 2024-10-07 11:10:12,500 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] (17505) Mapped job to destination id: local_dest
galaxy.jobs.handler DEBUG 2024-10-07 11:10:12,510 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] (17505) Dispatching to local_runner runner
galaxy.jobs DEBUG 2024-10-07 11:10:12,524 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] (17505) Persisting job destination (destination id: local_dest)
galaxy.jobs DEBUG 2024-10-07 11:10:12,538 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] (17505) Working directory for job is: /mnt/nfs/data/job_work_dir/017/17505
galaxy.jobs.runners DEBUG 2024-10-07 11:10:12,545 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] Job [17505] queued (34.395 ms)
galaxy.jobs.handler INFO 2024-10-07 11:10:12,549 [pN:main.2,p:3238883,tN:JobHandlerQueue.monitor_thread] (17505) Job dispatched
galaxy.jobs DEBUG 2024-10-07 11:10:12,616 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] Job wrapper for Job [17505] prepared (58.727 ms)
galaxy.jobs.command_factory INFO 2024-10-07 11:10:12,656 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] Built script [/mnt/nfs/data/job_work_dir/017/17505/tool_script.sh] for tool command [python '/srv/galaxy/server/lib/galaxy/tools/imp_exp/unpack_tar_gz_archive.py' 'L3Nydi9nYWxheHkvdmFyL3RtcC90bXB4cDh1c3JrZi56aXA=' 'L3Nydi9nYWxheHkvdmFyL3RtcC90bXBpYzd5cW8zZg==' --file --file-sources '/mnt/nfs/data/job_work_dir/017/17505/configs/tmpfahhef6b' --encoded]
galaxy.jobs.runners DEBUG 2024-10-07 11:10:12,720 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] (17505) command is: mkdir -p working outputs configs
if [ -d _working ]; then
    rm -rf working/ outputs/ configs/; cp -R _working working; cp -R _outputs outputs; cp -R _configs configs
else
    cp -R working _working; cp -R outputs _outputs; cp -R configs _configs
fi
cd working; /bin/bash /mnt/nfs/data/job_work_dir/017/17505/tool_script.sh > '../outputs/tool_stdout' 2> '../outputs/tool_stderr'; return_code=$?; echo $return_code > /mnt/nfs/data/job_work_dir/017/17505/galaxy_17505.ec; cd '/mnt/nfs/data/job_work_dir/017/17505'; 
[ "$GALAXY_VIRTUAL_ENV" = "None" ] && GALAXY_VIRTUAL_ENV="$_GALAXY_VIRTUAL_ENV"; _galaxy_setup_environment True; python metadata/set.py; sh -c "exit $return_code"
galaxy.jobs.runners.local DEBUG 2024-10-07 11:10:12,762 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] (17505) executing job script: /mnt/nfs/data/job_work_dir/017/17505/galaxy_17505.sh
galaxy.jobs.runners.util.process_groups DEBUG 2024-10-07 11:10:21,271 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] check_pg(): No process found in process group 380764
galaxy.jobs.runners.local DEBUG 2024-10-07 11:10:21,273 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] execution finished: /mnt/nfs/data/job_work_dir/017/17505/galaxy_17505.sh
/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py:89: SAWarning: Object of type <WorkflowInvocationStep> not in session, add operation along 'WorkflowInvocation.steps' will not proceed
  self.sa_session.commit()
galaxy.jobs ERROR 2024-10-07 11:10:21,493 [pN:main.2,p:3238883,tN:LocalRunner.work_thread-0] exec_after_process hook failed for job 17505
Traceback (most recent call last):
  File "/srv/galaxy/server/lib/galaxy/jobs/__init__.py", line 2039, in finish
    task_wrapper = self.tool.exec_after_process(
  File "/srv/galaxy/server/lib/galaxy/tools/__init__.py", line 3100, in exec_after_process
    JobImportHistoryArchiveWrapper(self.app, job.id).cleanup_after_job()
  File "/srv/galaxy/server/lib/galaxy/tools/imp_exp/__init__.py", line 81, in cleanup_after_job
    model_store.perform_import(new_history, job=job, new_history=True)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 420, in perform_import
    self._import_workflow_invocations(object_import_tracker, history)
  File "/srv/galaxy/server/lib/galaxy/model/store/__init__.py", line 1084, in _import_workflow_invocations
    job = object_import_tracker.jobs_by_key[step_attrs["job"][object_key]]
KeyError: '040106f9b0b0dbd5'

@patrick-austin patrick-austin mentioned this issue Oct 15, 2024
Closed
This was referenced Oct 16, 2024
Closed
Closed
mvdbeek added a commit to mvdbeek/galaxy that referenced this issue Nov 27, 2024
@mvdbeek
Add regression test for galaxyproject#18927
8881c9d
mvdbeek added a commit to mvdbeek/galaxy that referenced this issue Nov 27, 2024
@mvdbeek
Add regression test for galaxyproject#18927
ac44482
mvdbeek added a commit to mvdbeek/galaxy that referenced this issue Nov 27, 2024
@mvdbeek
Add regression test for galaxyproject#18927
2d7667d
@mvdbeek mvdbeek mentioned this issue Nov 27, 2024
Merged
4 tasks
@davelopez
Copy link
Contributor

It should be fixed in #19215, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davelopez davelopez

Labels
area/backend kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bgruening @mvdbeek @davelopez @patrick-austin