Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace sample Celery result_backend in config #17949

Merged
merged 8 commits into from
Apr 11, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions doc/source/admin/galaxy_options.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5154,7 +5154,10 @@
only if you have setup a Celery worker for Galaxy and you have
configured the `celery_conf` option below. Specifically, you need
to set the `result_backend` option in the `celery_conf` option to
a valid Celery result backend URL. For details, see
a valid Celery result backend URL. By default, Galaxy uses an
SQLite database at '<data_dir>/results.sqlite' for storing task
results. Please use a more robust backend (e.g. Redis) for
production setups. For details, see
https://docs.galaxyproject.org/en/master/admin/production.html#use-celery-for-asynchronous-tasks
:Default: ``false``
:Type: bool
Expand All @@ -5169,15 +5172,16 @@
To refer to a task by name, use the template `galaxy.foo` where
`foo` is the function name of the task defined in the
galaxy.celery.tasks module.
The `broker_url` option, if unset, defaults to the value of
`amqp_internal_connection`. The `result_backend` option must be
set if the `enable_celery_tasks` option is set.
The `broker_url` option, if unset or null, defaults to the value
of `amqp_internal_connection`. The `result_backend` option, if
unset or null, defaults to an SQLite database at
'<data_dir>/results.sqlite' for storing task results.
The galaxy.fetch_data task can be disabled by setting its route to
"disabled": `galaxy.fetch_data: disabled`. (Other tasks cannot be
disabled on a per-task basis at this time.)
For details, see Celery documentation at
https://docs.celeryq.dev/en/stable/userguide/configuration.html.
:Default: ``{'result_backend': 'redis://127.0.0.1:6379/0', 'task_routes': {'galaxy.fetch_data': 'galaxy.external', 'galaxy.set_job_metadata': 'galaxy.external'}}``
:Default: ``{'broker_url': None, 'result_backend': None, 'task_routes': {'galaxy.fetch_data': 'galaxy.external', 'galaxy.set_job_metadata': 'galaxy.external'}}``
:Type: any


Expand Down
9 changes: 9 additions & 0 deletions lib/galaxy/config/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -1082,6 +1082,9 @@ def _process_config(self, kwargs: Dict[str, Any]) -> None:
self.amqp_internal_connection = (
f"sqlalchemy+sqlite:///{self._in_data_dir('control.sqlite')}?isolation_level=IMMEDIATE"
)

self._process_celery_config()

self.pretty_datetime_format = expand_pretty_datetime_format(self.pretty_datetime_format)
try:
with open(self.user_preferences_extra_conf_path) as stream:
Expand Down Expand Up @@ -1203,6 +1206,12 @@ def _load_theme(path: str, theme_dict: dict):
else:
_load_theme(self.themes_config_file, self.themes)

def _process_celery_config(self):
if self.celery_conf and self.celery_conf.get("result_backend") is None:
# If the result_backend is not set, use a SQLite database in the data directory
result_backend = f"db+sqlite:///{self._in_data_dir('results.sqlite')}?isolation_level=IMMEDIATE"
self.celery_conf["result_backend"] = result_backend

def _check_database_connection_strings(self):
"""
Verify connection URI strings in galaxy's configuration are parseable with urllib.
Expand Down
15 changes: 10 additions & 5 deletions lib/galaxy/config/sample/galaxy.yml.sample
Original file line number Diff line number Diff line change
Expand Up @@ -2761,24 +2761,29 @@ galaxy:
# only if you have setup a Celery worker for Galaxy and you have
# configured the `celery_conf` option below. Specifically, you need to
# set the `result_backend` option in the `celery_conf` option to a
# valid Celery result backend URL. For details, see
# valid Celery result backend URL. By default, Galaxy uses an SQLite
# database at '<data_dir>/results.sqlite' for storing task results.
# Please use a more robust backend (e.g. Redis) for production setups.
davelopez marked this conversation as resolved.
Show resolved Hide resolved
# For details, see
# https://docs.galaxyproject.org/en/master/admin/production.html#use-celery-for-asynchronous-tasks
#enable_celery_tasks: false

# Configuration options passed to Celery.
# To refer to a task by name, use the template `galaxy.foo` where
# `foo` is the function name of the task defined in the
# galaxy.celery.tasks module.
# The `broker_url` option, if unset, defaults to the value of
# `amqp_internal_connection`. The `result_backend` option must be set
# if the `enable_celery_tasks` option is set.
# The `broker_url` option, if unset or null, defaults to the value of
# `amqp_internal_connection`. The `result_backend` option, if unset or
# null, defaults to an SQLite database at '<data_dir>/results.sqlite'
# for storing task results.
# The galaxy.fetch_data task can be disabled by setting its route to
# "disabled": `galaxy.fetch_data: disabled`. (Other tasks cannot be
# disabled on a per-task basis at this time.)
# For details, see Celery documentation at
# https://docs.celeryq.dev/en/stable/userguide/configuration.html.
#celery_conf:
# result_backend: redis://127.0.0.1:6379/0
# broker_url: null
# result_backend: null
# task_routes:
# galaxy.fetch_data: galaxy.external
# galaxy.set_job_metadata: galaxy.external
Expand Down
11 changes: 7 additions & 4 deletions lib/galaxy/config/schemas/config_schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3764,14 +3764,16 @@ mapping:
Activate this only if you have setup a Celery worker for Galaxy and you have
configured the `celery_conf` option below. Specifically, you need to set the
`result_backend` option in the `celery_conf` option to a valid Celery result
backend URL.
backend URL. By default, Galaxy uses an SQLite database at '<data_dir>/results.sqlite' for storing task results.
Please use a more robust backend (e.g. Redis) for production setups.
For details, see https://docs.galaxyproject.org/en/master/admin/production.html#use-celery-for-asynchronous-tasks

celery_conf:
type: any
required: false
default:
result_backend: redis://127.0.0.1:6379/0
nsoranzo marked this conversation as resolved.
Show resolved Hide resolved
broker_url: null
result_backend: null
task_routes:
'galaxy.fetch_data': 'galaxy.external'
'galaxy.set_job_metadata': 'galaxy.external'
Expand All @@ -3781,8 +3783,9 @@ mapping:
To refer to a task by name, use the template `galaxy.foo` where `foo` is the function name
of the task defined in the galaxy.celery.tasks module.

The `broker_url` option, if unset, defaults to the value of `amqp_internal_connection`.
The `result_backend` option must be set if the `enable_celery_tasks` option is set.
The `broker_url` option, if unset or null, defaults to the value of `amqp_internal_connection`.
The `result_backend` option, if unset or null, defaults to an SQLite database at '<data_dir>/results.sqlite'
for storing task results.

The galaxy.fetch_data task can be disabled by setting its route to "disabled": `galaxy.fetch_data: disabled`.
(Other tasks cannot be disabled on a per-task basis at this time.)
Expand Down
Loading