forked from apache/airflow
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add pre-Airflow-2-7 hardcoded defaults for config for older providers (…
…apache#32775) * Quarantine test_backfill_integration in dask executor The test has been recently failing with deadlock (see apache#32778) and needs thorough looking at if we want to find the root cause/remedium. In the meantime it looks like a niche case connected with Dask Executor that is rather obsure and we have no expertise in solving problems with and diagnosing, so until the problem is diagnosed it might be a long time (and maybe even we decide not to care about it and let Dask community take a look and either fix or ignore it. We aim to have a very low number of those Quarantined tests (currently we have 1 and we have not run it for a while as this was a mysql test run on Postgres) but we have now the opportunity to also improve the quarantined tests framework. This test will be run together with other (1) quarantined test and: * they will not be run in our regular tests * they are run sequentially not in parallel with all other tests * they are run for all 4 backends but only for the default versions of those backends * failure of the quarantined tests will not cause failure of the whole job or limit constraints from being generated and updated * Add pre-Airflow-2-7 hardcoded defaults for config for older providers During thorough testing and review of moving configuration to provoders I realised that there was a case that was not handled properly. In some cases some providers and DAGs could rely on some default values being available as default, but when we move them from core, and use older version of provider those defaults were not available: * they were remove as defaults in core * the old providers did not have "config" section to contribute the defaults This would be a breaking change and old providers (Celery, K8s) could fail - as it happened in some tests. This PR implements a nice solution to that, also allowing to remove some manual fallbacks in Celery and Kubernetes executor code. The solution is to add a hard-coded "pre-2.7" configuration which would only contain "provider" pre-2.7 hard-coded defaults and make it a fallback option if the values are neither set nor defaults contributed by the providers. We do not have to maintain those - the defaults are "frozen" effectively at the values available just before 2.7. The nice side effect is that we can remove a number of fallbacks, because this hard-coded configuration becomes the fallback automatically, That entirely solves the case where you want to install older providers on 2.7 where config.yml does not contain those provider values. * Update airflow/configuration.py Co-authored-by: Jed Cunningham <[email protected]> --------- Co-authored-by: Jed Cunningham <[email protected]>
- Loading branch information
1 parent
031e394
commit 2ac237b
Showing
10 changed files
with
135 additions
and
38 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
# | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
|
||
# This file contains pre Airflow 2.7, provider defaults for Airflow configuration. | ||
# They are provided as fallback option to older version of the | ||
# providers that might expect them to be present. | ||
# | ||
# NOTE !!!! Please DO NOT modify values in the file even if they change in corresponding | ||
# providers. The values here should be treated as "read only" and should not be modified | ||
# even if defaults in newer versions of corresponding Providers change. | ||
# They are only here so that backwards compatible behaviour for old provider | ||
# versions can be maintained. | ||
|
||
[atlas] | ||
sasl_enabled = False | ||
host = | ||
port = 21000 | ||
username = | ||
password = | ||
|
||
[hive] | ||
default_hive_mapred_queue = | ||
|
||
[local_kubernetes_executor] | ||
kubernetes_queue = kubernetes | ||
|
||
[celery_kubernetes_executor] | ||
kubernetes_queue = kubernetes | ||
|
||
[celery] | ||
celery_app_name = airflow.executors.celery_executor | ||
worker_concurrency = 16 | ||
worker_prefetch_multiplier = 1 | ||
worker_enable_remote_control = true | ||
broker_url = redis://redis:6379/0 | ||
result_backend_sqlalchemy_engine_options = | ||
flower_host = 0.0.0.0 | ||
flower_url_prefix = | ||
flower_port = 5555 | ||
flower_basic_auth = | ||
sync_parallelism = 0 | ||
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG | ||
ssl_active = False | ||
ssl_key = | ||
ssl_cert = | ||
ssl_cacert = | ||
pool = prefork | ||
operation_timeout = 1.0 | ||
task_track_started = True | ||
task_publish_max_retries = 3 | ||
worker_precheck = False | ||
|
||
[elasticsearch_configs] | ||
use_ssl = False | ||
verify_certs = True | ||
|
||
[kubernetes_executor] | ||
api_client_retry_configuration = | ||
logs_task_metadata = False | ||
pod_template_file = | ||
worker_container_repository = | ||
worker_container_tag = | ||
namespace = default | ||
delete_worker_pods = True | ||
delete_worker_pods_on_failure = False | ||
worker_pods_creation_batch_size = 1 | ||
multi_namespace_mode = False | ||
multi_namespace_mode_namespace_list = | ||
in_cluster = True | ||
kube_client_request_args = | ||
delete_option_kwargs = | ||
enable_tcp_keepalive = True | ||
tcp_keep_idle = 120 | ||
tcp_keep_intvl = 30 | ||
tcp_keep_cnt = 6 | ||
verify_ssl = True | ||
worker_pods_queued_check_interval = 60 | ||
ssl_ca_cert = |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters