Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace sample Celery result_backend in config #17949

Merged
merged 8 commits into from
Apr 11, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions doc/source/admin/galaxy_options.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5154,7 +5154,10 @@
only if you have setup a Celery worker for Galaxy and you have
configured the `celery_conf` option below. Specifically, you need
to set the `result_backend` option in the `celery_conf` option to
a valid Celery result backend URL. For details, see
a valid Celery result backend URL. By default, Galaxy uses an
SQLite database at '<data_dir>/results.sqlite' for storing task
results. Please use a more robust backend for production setups
like Redis. For details, see
https://docs.galaxyproject.org/en/master/admin/production.html#use-celery-for-asynchronous-tasks
:Default: ``false``
:Type: bool
Expand All @@ -5177,7 +5180,7 @@
disabled on a per-task basis at this time.)
For details, see Celery documentation at
https://docs.celeryq.dev/en/stable/userguide/configuration.html.
:Default: ``{'result_backend': 'redis://127.0.0.1:6379/0', 'task_routes': {'galaxy.fetch_data': 'galaxy.external', 'galaxy.set_job_metadata': 'galaxy.external'}}``
:Default: ``{'result_backend': 'db+sqlite:///./database/results.sqlite?isolation_level=IMMEDIATE', 'task_routes': {'galaxy.fetch_data': 'galaxy.external', 'galaxy.set_job_metadata': 'galaxy.external'}}``
:Type: any


Expand Down
13 changes: 13 additions & 0 deletions lib/galaxy/config/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -1082,6 +1082,9 @@ def _process_config(self, kwargs: Dict[str, Any]) -> None:
self.amqp_internal_connection = (
f"sqlalchemy+sqlite:///{self._in_data_dir('control.sqlite')}?isolation_level=IMMEDIATE"
)

self._process_celery_config()

self.pretty_datetime_format = expand_pretty_datetime_format(self.pretty_datetime_format)
try:
with open(self.user_preferences_extra_conf_path) as stream:
Expand Down Expand Up @@ -1203,6 +1206,16 @@ def _load_theme(path: str, theme_dict: dict):
else:
_load_theme(self.themes_config_file, self.themes)

def _process_celery_config(self):
if self.celery_conf:
result_backend = self.celery_conf.get("result_backend")
if result_backend:
# If the result_backend is the default SQLite database, we need to
# ensure that the correct data directory is used.
if "results.sqlite" in result_backend:
davelopez marked this conversation as resolved.
Show resolved Hide resolved
result_backend = f"db+sqlite:///{self._in_data_dir('results.sqlite')}?isolation_level=IMMEDIATE"
self.celery_conf["result_backend"] = result_backend

def _check_database_connection_strings(self):
"""
Verify connection URI strings in galaxy's configuration are parseable with urllib.
Expand Down
7 changes: 5 additions & 2 deletions lib/galaxy/config/sample/galaxy.yml.sample
Original file line number Diff line number Diff line change
Expand Up @@ -2761,7 +2761,10 @@ galaxy:
# only if you have setup a Celery worker for Galaxy and you have
# configured the `celery_conf` option below. Specifically, you need to
# set the `result_backend` option in the `celery_conf` option to a
# valid Celery result backend URL. For details, see
# valid Celery result backend URL. By default, Galaxy uses an SQLite
# database at '<data_dir>/results.sqlite' for storing task results.
# Please use a more robust backend for production setups like Redis.
# For details, see
# https://docs.galaxyproject.org/en/master/admin/production.html#use-celery-for-asynchronous-tasks
#enable_celery_tasks: false

Expand All @@ -2778,7 +2781,7 @@ galaxy:
# For details, see Celery documentation at
# https://docs.celeryq.dev/en/stable/userguide/configuration.html.
#celery_conf:
# result_backend: redis://127.0.0.1:6379/0
# result_backend: db+sqlite:///./database/results.sqlite?isolation_level=IMMEDIATE
# task_routes:
# galaxy.fetch_data: galaxy.external
# galaxy.set_job_metadata: galaxy.external
Expand Down
5 changes: 3 additions & 2 deletions lib/galaxy/config/schemas/config_schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3764,14 +3764,15 @@ mapping:
Activate this only if you have setup a Celery worker for Galaxy and you have
configured the `celery_conf` option below. Specifically, you need to set the
`result_backend` option in the `celery_conf` option to a valid Celery result
backend URL.
backend URL. By default, Galaxy uses an SQLite database at '<data_dir>/results.sqlite' for storing task results.
Please use a more robust backend for production setups like Redis.
davelopez marked this conversation as resolved.
Show resolved Hide resolved
For details, see https://docs.galaxyproject.org/en/master/admin/production.html#use-celery-for-asynchronous-tasks

celery_conf:
type: any
required: false
default:
result_backend: redis://127.0.0.1:6379/0
nsoranzo marked this conversation as resolved.
Show resolved Hide resolved
result_backend: db+sqlite:///./database/results.sqlite?isolation_level=IMMEDIATE
davelopez marked this conversation as resolved.
Show resolved Hide resolved
task_routes:
'galaxy.fetch_data': 'galaxy.external'
'galaxy.set_job_metadata': 'galaxy.external'
Expand Down
Loading