Airflow 2 to 3 auto migration rules #41641
Comments
Lee-W
added
kind:feature
|
The |
|
I pinned the issue - this way it will show up at the top of "Issues" list in the repo |
|
|
|
Thanks! |
|
We can just go over all the significant newsfragments and create a rule for them or keep some reasoning why it doesn't require one |
|
We should add something for the public API change too. API v1 won't work anymore. Those are being changed as part of AIP-84 to a new FastApi based app. GitHub project for it: https://github.com/orgs/apache/projects/414 |
|
Issue here to regroup Rest API breaking changes #43378 |
|
I have started prototyping a small package based on LibCST to build a Python 2to3 like tool for Airflow 2to3 that does simple and straight forward replacements. My main motivation was around lot of our users in our Airflow instance using Dags
Operators
Sample file import datetime
from airflow import DAG
from airflow.decorators import dag, task
from airflow.operators.empty import EmptyOperator
from airflow.timetables.events import EventsTimetable
with DAG(
dag_id="my_dag_name",
default_view="tree",
start_date=datetime.datetime(2021, 1, 1),
schedule_interval="@daily",
concurrency=2,
):
op = EmptyOperator(
task_id="task", task_concurrency=1, trigger_rule="none_failed_or_skipped"
)
@dag(
default_view="graph",
start_date=datetime.datetime(2021, 1, 1),
schedule_interval=EventsTimetable(event_dates=[datetime.datetime(2022, 4, 5)]),
max_active_tasks=2,
full_filepath="/tmp/test_dag.py"
)
def my_decorated_dag():
op = EmptyOperator(task_id="task")
my_decorated_dag()Sample usage python -m libcst.tool codemod dag_fixer.DagFixerCommand -u 1 tests/test_dag.py
Calculating full-repo metadata...
Executing codemod...
reformatted -
All done! ✨ 🍰 ✨
1 file reformatted.
--- /home/karthikeyan/stuff/python/libcst-tut/tests/test_dag.py
+++ /home/karthikeyan/stuff/python/libcst-tut/tests/test_dag.py
@@ -10,6 +10,6 @@
dag_id="my_dag_name",
- default_view="tree",
+ default_view="grid",
start_date=datetime.datetime(2021, 1, 1),
- schedule_interval="@daily",
- concurrency=2,
+ schedule="@daily",
+ max_active_tasks=2,
):
@@ -23,5 +23,4 @@
start_date=datetime.datetime(2021, 1, 1),
- schedule_interval=EventsTimetable(event_dates=[datetime.datetime(2022, 4, 5)]),
+ schedule=EventsTimetable(event_dates=[datetime.datetime(2022, 4, 5)]),
max_active_tasks=2,
- full_filepath="/tmp/test_dag.py"
)
Finished codemodding 1 files!
- Transformed 1 files successfully.
- Skipped 0 files.
- Failed to codemod 0 files.
- 0 warnings were generated. |
|
NICE! @tirkarthi -> you should start a thread about it at devlist and propose adding it to the repo. The sooner we start working on it and let poeple test it, the better it will be. And we can already start adding not only the newsfragments but also rules to the migration tools (cc: @vikramkoka @kaxil ) - we can even think about keeping a database of old-way-dags and running such migration tool on them and letting airflow scheduler from Airflow 3 process them (and maybe even execute) as part of our CI. This would tremendously help with maintaining and updating such a tool if we will make it a part of our CI pipeline. |
|
BTW. I like it a lot how simple it is with libCST - we previously used quite a bit more complex tool from Facebook that allowed to do refactoring at scale in parallell (https://github.com/facebookincubator/Bowler) , but it was rather brittle to develop rules for it and it had some weird problems and missing features. One thing that was vere useful - is that it had a nice "parallelism" features - which allowed to refactor 1000s of files in seconds (but also made it difficult to debug). I think if we get it working with libCST - it will be way more generic and maintainable, also we can easily add parallelism later on when/if we see it is slow. |
|
One small watchout though - such a tool should have a way to isolate rules - so that they are not in a single big method - some abstraction that will allow us to easily develop and selectively apply (or skip) different rules - see https://github.com/apache/airflow/tree/v1-10-test/airflow/upgrade where we have documentation and information about the upgrade check we've done in Airflow 1 -> 2 migration. Also we have to discuss, whether it should be a separate repo or whether it should be in airflow's monorepo. Both have pros and cons - in 1.10 we chose to keep it 1.10 branch of airflow, because it imported some of the airflow code and it was easier, but we could likely create a new repo for it, add CI there and keep it there. We even have this archived repo https://github.com/apache/airflow-upgrade-check which we never used and archived, we could re-open it. We also have https://pypi.org/project/apache-airflow-upgrade-check/ - package in PyPI - and we could release new upgrade check versions (2.* ?) with "apache-airflow>=2.11.0" as dependency. All that should likely be discussed at devlist :) |
|
Thanks @potiuk for the details. I will start a discussion on this at the devlist and continue there. Bowler looks interesting. Using |
Don't be deceived by it :). It was helpful for Provider's migration at some point in time, but I had many rough edges - like debugging a problem was a nightmare until we learned how to do it properly, also it had some annoying limitations - you had to learn a completely new non-standard abstractions (an SQLAlchemy-like DSL to perform modifications) - which did not cover all the refactorings we wanted to do. We had to really dig-deep into the code an find some workarounds for things we wanted to do, when authors of Bowler have not thoght about them. And sometimes those were nasty workarounds. query = (
Query(<paths to modify>)
.select_function("old_name")
.rename("new_name")
.diff(interactive=True)
)Example that I remember above is that we could not rename some of the object types easily because it was not "foreseen" (can't remember exactly) - we had a few surprises there. Also Bowler seems to be not maintained for > 3 years and it means that it's unlikely to handle some constructs even in 3.9+ Airflow. What I like about libcst is that it is really "low-level" interface that you have to program in Python rather than in abstract DSL - similar to "ast". You write actual python code to perform what you want to perform rather than rely on incomplete abstractions, even if you have to copy&paste rename code between different "rules" (for example) (which you can then abstract away as 'common` util if you need, so no big deal). |
|
BTW. Codemod .... is also 5 years not maintained. Not that it is disqualification - but they list |
|
I tried to use libcst in airflow as a tiny POC of this issue here . It mostly works great except for its speed. I was also thinking about whether to add these migrations thing info ruff airflow linter but not yet explore much on the rust/ruff side. |
|
👀 👀 Me ❤️ it (but I doubt we want to invest in it as it might be difficult to maintain, unless we find quite a few committers who are somewhat ruff profficient to at least be able to review the code) . But it's tempting I must admit. But to be honest - while I'd love to finally get a serious rust project, it's not worth it I think we are talking of one-time migration for even a 10.000 dags it will take at most single minutes and we can turn it maybe in under one minute with rust - so not a big gain for a lot of pain :) . Or at lest this is what my intuition tells me. I think parallelism will do the job nicely. My intuition tells me (but this is just intuition and understanding on some limits ans speed of certain operation) - that we will get from multiple 10s of minutes (when running such migration sequentially) to single minutes when we allow to run migration in parallel using multiple processors and processes - even with Python and libcst. This task is really suitable for such parallelisation because each file is complete, independent task that can be run in complete isolation from all other tasks - so spawning multiple paralllel interpreters, ideally forking them right after all the imports and common code is loaded so that they use shared memory for those - this should do the job nicely (at least intuitively). Using RUST for that might be classic premature optimisation - we might likely not need it :). But would be worth to make some calculations and get some "numbers" for big installation - i.e. how many dags of what size are out there, and how long it will be to parse them all with libcst and write back (even unmodified or with a simple modification). I presume that parsing and writing back will be the bulk of the job - and modifications will add very little overhead as they will be mostly operating on in memory data structures. |
Yep, totally agree. I just want to raise this idea which might be interesting. 👀
Yep, I think you're right. My previous default deferrable script took around 10 sec to process ~400 operators. Using ast for checking took around 1 sec |
Yep, not that convinced either. but it is always good to have an alternative we could consider 🤔 |
|
My best idea right now is to split this into two tools. We don’t really want to invest too much time into building a very rich CLI tool to show users what need to be changed—we’ll effectively be rebuilding the error reporting interface in ruff (or flake8). Those squiggle lines, colors, error codes, and code context things are not easy to build. It is probably easiest to tack the linter part on Ruff—it is Rust, but the code to implement a lint rule isn’t that hard if you know Python AST and just a general idea about C-like languages. The rewrite part is a lot more difficult, so it’s probably better to implement this as a different tool in Python with libcst. I’m thinking something like $ ruff check --select AIR
This spits out lint errors with codes like AIR005 AIR123...
$ airflow2to3 --select AIR005 -- path/to/dag/file.py
This fixes the given error(s) in given file(s) in-place with a minimal CLI...I plan to start experiementing some rules in Ruff to see how easy the first part actually is. We should be able to save a lot of effort if it is viable. |
|
I tried to change the format a bit and list the rules in the following format. * [ ] link to the pr with breaking change
* [ ] things to doOnce the I also updated the format for #41366, #41367, #41368, #41391, #41393 If anyone has anything to add but do not have permission to update the description. Please just tag me and I'll take a look |
Actually I am convinced too - I quite like this one after a bit of thought. This is not something that might be maintained by a lot of people and a number of contributors, and even for them, this is so far from the main "airflow code" - it's really a "one-time" tool - that it might be worth treating it as our first "rust experiment". And I quite agree that, the AST code on it's own is not really that "pythonic" and if you know what you want, and have already existing examples, adding a new rule in RUST, should not be difficult even if you do not know it (and AI driven development here might be even pretty cool exercise). I'd myself be happy to add a few rules at some point of time and maybe even take part in implementing the tooling for rust for our CI environment. |
|
The things we'll need to migrate for 41348
|
|
I tried my hands on implementing a rule in Ruff. This one checks if a DAG uses the Does this look reasonable enough for people to build on? I’ll produce a more detailed writeup of what to do if we feel this is the way to go. |
Which models? But no, the plan is to not have/"allow" users to import anything from airflow.models at all. Exact details and new names are to be determined yet though |
Pretty much every model 👀 Sounds good. Just want to confirm I'm not misunderstanding anything. I'll just mark it as model change and not going to migrate for now till we have anything decided |
|
@Lee-W There are few rules that we should add for Airflow configs too since we changed /removed them: and some imports which won't work:
|
Thanks for reminding me! I'm still halfway to completing reading all the PRs. Will continue work on updating the list |
|
Awesome @Lee-W , I hadn't seen this issue, so great to see the progress here. |
Looks great! I'll try to update both places |
|
I also played around the fixer implementation a bit: https://github.com/uranusjr/airflow2to3 Still a lot of room for improvement, but I think it is a good starting point. FWIW I also tried to create a Flake8 plugin for comparison, but got stuck resolving imports (we need to check whether a thing is actually from Airflow first to decide whether to emit errors for it). Flake8 does not seem to provide this out of the box (Ruff and LibCST each has an easy solution). I’m pretty sure there must be a solution, but the effort looking into this is probably not worthwhile unless we feel Ruff is too big a hurdle. |
|
So what do people think about using Ruff for linting? This is somewhat important since we want to encourage community help to implement rules. If people don’t have strong opinions, I’ll start a lazy consensus thread in the mailing list to get a resolution. |
|
Thanks for preparing this Lee-W! For #43102, we can only detect the changes with |
But one of the questions is how we will detect it 🤔 there are plenty of ways to send a request to an API. |
|
There is no straightforward way to do it. Indeed, it can be even a standalone application communicating with the Airflow API without other interactions. It could be even in different programming languages such as |
|
I am not sure #42042. I moved out one property from |
I think we can only do it reasonably well if we assume the user uses Python Cliant of ours and then we should be able to say the users they could run their custom code through Ruff with the new python client installed to detect wrong parameters. Not sure if we need to have custom ruff rules for those changes, or maybe it's "mypy" kind of check for types ? I know astral works on a For the rest of the clients, I think what we could also do - potentially - is to have a custom error code in fast API - where we wil handle errors generated when "known old-style requests" are issued to the new API and return pretty descriptive error " You are using old-style API ble, ble, ble ... you need to change your paremeters to .... ." - maybe we can generalize it in our API in the way to handle "typical" mistakes from multiple APIs by the same error handler? |
I agree, we should make this assumption and limit the check to a reasonable scope. I like the idea of restricting it so that only our Python Client will be affected. Otherwise, it could turn into a project of its own. :)
I was considering a similar approach, returning an error response if the old request is provided but I wasn’t entirely sure about the scope. If the goal is to catch these issues before upgrading the version, I am unsure how we can easily provide that. Simply reading the API changes documentation seems easier than creating a transition API and asking users to route calls through it to catch errors. Otherwise, such responses would indicate that their processes have already failed with an error. |
Yeah. Maybe we can do something in Airlfow 2.11 ? Since our goal is that 2.11 should be the "bridge" release - maybe we could do a variation of my original proposal - see if the message coming is in "old" format and raise a deprecation and also manually implement "new" format there (that would likely require some manual modificaiton of the openapi specification there and some conditional code in 2.11 (if possible). That could follow our pattern of "Make sure 2.11 raises no warnings and then migration to 3 should be smooth". |
Indeed upgrading the client will automatically highlights type errors in users code. For people bypassing the python client and making direct request to the API (or any other non-python system), we can indeed catch errors of such breaking change and return a clear message that this is not accepted anymore, and maybe even better give them the new way of how to achieve this. It's just more work but possible. Otherwise reading the significant newsfragment for the |
Just to repeat above - yes. I think it's more work, and I think it might require some nasty workarounds in FAST API that we will have to keep forever, but maybe we can do a 2.11-only change that will raise warnings if the old way is used instead (and allow to use new way) ? Not sure if possible and how many such breaking channges we will have, but it would really be nice to tell the users "if you have no warnings on 2.11, you are good to go". |
After reading it again, I don't think users are using it either. 🤔 Then I'll just remove it. Thanks for checking it! |
Description
Why
As we're introducing breaking changes to the main branch, it would be better to begin recording the things we could use migration tools to help our users migrate from Airflow 2 to 3.
The breaking changes can be found at https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aairflow3.0%3Abreaking.
What
List of significant news fragments and rules (by Nov 26)
allow_raw_html_descriptionsscheduler.processor_poll_interval→scheduler.scheduler_idle_sleep_timeairflow.datasets→airflow.sdk.definitions.assetDatasetAlias→AssetAliasDatasetAll→AssetAllDatasetAny→AssetAnyexpand_alias_to_datasets→expand_alias_to_assetsDatasetAliasEvent→AssetAliasEventdest_dataset_uri→BaseAssetBaseDataset→BaseAssetiter_datasets→iter_assetsiter_dataset_aliases→iter_asset_aliasesDataset→Assetiter_datasets→iter_assetsiter_dataset_aliases→iter_asset_aliases_DatasetBooleanCondition→_AssetBooleanConditioniter_datasets→iter_assetsiter_dataset_aliases→iter_asset_aliasesairflow.datasets.manager→airflow.assets.managerdataset_manager→asset_managerresolve_dataset_manager→resolve_asset_managerDatasetManager→AssetManagerregister_dataset_change→register_asset_changecreate_datasets→create_assetsregister_dataset_change→notify_asset_creatednotify_dataset_changed→notify_asset_changednotify_dataset_alias_created→notify_asset_alias_createdairflow.listeners.spec.dataset→airflow.listeners.spec.asseton_dataset_created→on_asset_createdon_dataset_changed→on_asset_changedairflow.timetables.datasets→airflow.timetables.assetsDatasetOrTimeSchedule→AssetOrTimeScheduleairflow.datasets.metadata→airflow.sdk.definitions.asset.metadataairflow.listeners.spec.dataset→airflow.listeners.spec.asseton_dataset_created→on_asset_createdon_dataset_changed→on_asset_changedairflow.timetables.datasets.DatasetOrTimeSchedule→airflow.timetables.assets.AssetOrTimeScheduleairflow.api_connexion.security.requires_access_dataset→airflow.api_connexion.security.requires_access_dataset.requires_access_assetairflow.auth.managers.models.resource_details.DatasetDetails→airflow.auth.managers.models.resource_details.AssetDetailsairflow.auth.managers.base_auth_manager.is_authorized_dataset→airflow.auth.managers.base_auth_manager.is_authorized_assetairflow.timetables.simple.DatasetTriggeredTimetable→airflow.timetables.simple.AssetTriggeredTimetableairflow.providers_manager.ProvidersManagerinitialize_providers_dataset_uri_resources→initialize_providers_asset_uri_resourcesdataset_factories→asset_factoriesdataset_uri_handlers→asset_uri_handlersdataset_to_openlineage_converters→asset_to_openlineage_convertersairflow.security.permissions.RESOURCE_DATASET→airflow.security.permissions.RESOURCE_ASSETairflow.www.auth.has_access_dataset→airflow.www.auth.has_access_dataset.has_access_assetairflow.lineage.hook.DatasetLineageInfo→airflow.lineage.hook.AssetLineageInfodataset→assetairflow.lineage.hook.HookLineageCollectorcreate_dataset→create_assetadd_input_dataset→add_input_assetadd_output_dataset→add_output_assetcollected_datasets→collected_assetstriggering_dataset_events→triggering_asset_eventsdataset-uris→asset-uris(for providers amazon, common.io, mysql, fab, postgres, trino)airflow.providers.amazon.aws.datasets→airflow.providers.amazon.aws.assetss3create_dataset→create_assetconvert_dataset_to_openlineage→convert_asset_to_openlineageairflow.providers.amazon.auth_manager.avp.entities.AvpEntities.DATASET→airflow.providers.amazon.auth_manager.avp.entities.AvpEntities.ASSETairflow.providers.amazon.auth_manager.aws_auth_manager.AwsAuthManager.is_authorized_dataset→airflow.providers.amazon.auth_manager.aws_auth_manager.AwsAuthManager.is_authorized_assetairflow.providers.common.io.datasets→airflow.providers.common.io.assetsfilecreate_dataset→create_assetconvert_dataset_to_openlineage→convert_asset_to_openlineageairflow.providers.fab.auth_manager.fab_auth_manager.is_authorized_dataset→airflow.providers.fab.auth_manager.fab_auth_manager.is_authorized_assetairflow.providers.openlineage.utils.utilsDatasetInfo→AssetInfotranslate_airflow_dataset→translate_airflow_assetairflow.providers.postgres.datasets→airflow.providers.postgres.assetsairflow.providers.mysql.datasets→airflow.providers.mysql.assetsairflow.providers.trino.datasets→airflow.providers.trino.assetsairflow.api_connexion.schemas.dataset_schemaairflow.api_ui.views.datasetsairflow.serialization.pydantic.datasetairflow.serialization.pydantic.taskinstanceairflow.serialization.enums.DagAttributeTypesairflow.serialization.serialized_objectsairflow.utils.contextcontrib#41366airflow.contrib.*ImportErrorfromairflow.models#41367airflow.models.ImportError→airflow.models.errors.ParseImportErrorairflow.executors.*airflow.hooks.*airflow.macros.*airflow.operators.*airflow.sensors.*airflow.operators.subdagExternalTaskSensorLink#41391airflow.sensors.external_task.ExternalTaskSensorLink→airflow.sensors.external_task.ExternalDagLinDayOfWeekSensor#41393DayOfWeekSensoruse_task_execution_day→use_task_execution_dayTaskMixinclass #41394airflow.models.taskMixin.TaskMixin→airflow.models.taskMixin.DependencyMixinairflow.executors&airflow.utils#41395airflow.executors.executor_loader.UNPICKLEABLE_EXECUTORSairflow.utils.dag_cycle_tester.test_cycleairflow.utils.file.TemporaryDirectoryairflow.utils.file.mkdirsairflow.utils.state.SHUTDOWNairflow.utils.state.terminating_statessetinstead oflistfor dag's tags #41420DAGschedule_intervaltimetableairflow.utils.dates.date_rangeairflow.utils.dates.days_ago→ ❓ do we need to change it topendulum.today('UTC').add(days=-N, ...)airflow.utils.helpers.chain→airflow.models.baseoperator.chainairflow.utils.helpers.chain→airflow.models.baseoperator.cross_downstreamairflow.secrets.local_filesystem.load_connections→airflow.secrets.local_filesystem.load_connections_dictairflow.secrets.local_filesystem.get_connection→airflow.secrets.local_filesystem.load_connections_dictsmtp.smtp_usersmtp.smtp_passwordwebserver.session_lifetime_days→ usewebserver.session_lifetime_minuteswebserver.force_log_out_after→ usewebserver.session_lifetime_minutespolicy→task_policyairflow.utils.log.file_task_handler.FileTaskHandlerfilename_templateairflow.utils.decorators.apply_defaultsscheduler.dependency_detectorairflow.secrets.base_secrets.BaseSecretsBackend.get_conn_uri→airflow.secrets.base_secrets.BaseSecretsBackend.get_conn_valueairflow.secrets.base_secrets.BaseSecretsBackend.get_connections→airflow.secrets.base_secrets.BaseSecretsBackend.get_connectionairflow.api.auth.backend.basic_auth→airflow.providers.fab.auth_manager.api.auth.backend.basic_authairflow.api.auth.backend.kerberos_auth→airflow.providers.fab.auth_manager.api.auth.backend.kerberos_authairflow.auth.managers.fab.api.auth.backend.kerberos_auth→airflow.providers.fab.auth_manager.api.auth.backend.kerberos_authairflow.auth.managers.fab.fab_auth_manager→airflow.providers.fab.auth_manager.security_manager.overrideairflow.auth.managers.fab.security_manager.override→airflow.providers.fab.auth_manager.security_manager.overrideairflow.hooks.base.BaseHook.get_connections(❓ related to 41642)airflow.kubernetesairflow.operators.datetime.BranchDateTimeOperatoruse_task_execution_day→use_task_logical_dateairflow.operators.trigger_dagrun.TriggerDagRunOperatorparameter→logical_dateairflow.operators.weekday.BranchDayOfWeekOperatoruse_task_execution_day→use_task_logical_dateairflow.triggers.external_task.TaskStateTriggerairflow.hooks.dbapi→airflow.providers.common.sql.hooks.sqlairflow.www.auth.has_access→ useairflow.www.auth.has_access_*airflow.www.security→airflow.providers.fab.auth_manager.security_manager.override.FabAirflowSecurityManagerOverrideairflow.www.utils.get_sensitive_variables_fields→airflow.utils.log.secrets_masker.get_sensitive_variables_fieldsairflow.www.utils.should_hide_value_for_key→airflow.utils.log.secrets_masker.should_hide_value_for_keyBaseOperatortask_concurrency→max_active_tis_per_dagdummynone_failed_or_skippedoperators.ALLOW_ILLEGAL_ARGUMENTSairflow.models.baseoperator.BaseOperatorLink→airflow.models.baseoperatorlink.BaseOperatorLinkairflow.models.connection.parse_netloc_to_hostnameairflow.models.connection.Connection.parse_from_uriairflow.models.connection.Connection.log_infoairflow.models.connection.Connection.debug_infoairflow.api_connexion.security.requires_access→ userequires_access_*--treeflag fromairflow tasks listcommand #41964metrics.metrics_use_pattern_matchairflow.metrics.validators.AllowListValidator→ suggest usingairflow.metrics.validators.PatternAllowListValidator(not direct mapping)airflow.metrics.validators.BlockListValidator→ suggest usingairflow.metrics.validators.PatternBlockListValidator(not direct mapping)is_activeuser property to FAB auth manager #42042[ ] Remove property❌ Users are not likely to use itairflow.auth.managers.models.base_user.is_activecelery.stalled_task_timeoutkubernetes_executor.worker_pods_pending_timeout→scheduler.task_queued_timeoutmetrics.statsd_allow_list→metrics.metrics_allow_listmetrics.statsd_block_list→metrics.metrics_block_listscheduler.statsd_on→metrics.statsd_onscheduler.statsd_host→metrics.statsd_hostscheduler.statsd_port→metrics.statsd_portscheduler.statsd_prefix→metrics.statsd_prefixscheduler.statsd_allow_list→metrics.statsd_allow_listscheduler.stat_name_handler→metrics.stat_name_handlerscheduler.statsd_datadog_enabled→metrics.statsd_datadog_enabledscheduler.statsd_datadog_tags→metrics.statsd_datadog_tagsscheduler.statsd_datadog_metrics_tags→metrics.statsd_datadog_metrics_tagsscheduler.statsd_custom_client_path→metrics.statsd_custom_client_pathcore.interleave_timestamp_parser→logging.interleave_timestamp_parsercore.base_log_folder→logging.base_log_foldercore.remote_logging→logging.remote_loggingcore.remote_log_conn_id→logging.remote_log_conn_idcore.remote_base_log_folder→logging.remote_base_log_foldercore.encrypt_s3_logs→logging.encrypt_s3_logscore.logging_level→logging.logging_levelcore.fab_logging_level→logging.fab_logging_levelcore.logging_config_class→logging.logging_config_classcore.colored_console_log→logging.colored_console_logcore.colored_log_format→logging.colored_log_formatcore.colored_formatter_class→logging.colored_formatter_classcore.log_format→logging.log_formatcore.simple_log_format→logging.simple_log_formatcore.task_log_prefix_template→logging.task_log_prefix_templatecore.log_filename_template→logging.log_filename_templatecore.log_processor_filename_template→logging.log_processor_filename_templatecore.dag_processor_manager_log_location→logging.dag_processor_manager_log_locationcore.task_log_reader→logging.task_log_readercore.sql_alchemy_conn→database.sql_alchemy_conncore.sql_engine_encoding→database.sql_engine_encodingcore.sql_engine_collation_for_ids→database.sql_engine_collation_for_idscore.sql_alchemy_pool_enabled→database.sql_alchemy_pool_enabledcore.sql_alchemy_pool_size→database.sql_alchemy_pool_sizecore.sql_alchemy_max_overflow→database.sql_alchemy_max_overflowcore.sql_alchemy_pool_recycle→database.sql_alchemy_pool_recyclecore.sql_alchemy_pool_pre_ping→database.sql_alchemy_pool_pre_pingcore.sql_alchemy_schema→database.sql_alchemy_schemacore.sql_alchemy_connect_args→database.sql_alchemy_connect_argscore.load_default_connections→database.load_default_connectionscore.max_db_retries→database.max_db_retriescore.worker_precheck→celery.worker_precheckscheduler.max_threads→scheduler.parsing_processescelery.default_queue→operators.default_queueadmin.hide_sensitive_variable_fields→core.hide_sensitive_var_conn_fieldsadmin.sensitive_variable_fields→core.sensitive_var_conn_namescore.non_pooled_task_slot_count→core.default_pool_task_slot_countcore.dag_concurrency→core.max_active_tasks_per_dagapi.access_control_allow_origin→api.access_control_allow_originsapi.auth_backend→api.auth_backendsscheduler.deactivate_stale_dags_interval→scheduler.parsing_cleanup_intervalkubernetes_executor.worker_pods_pending_timeout_check_interval→scheduler.task_queued_timeout_check_intervalwebserver.update_fab_perms→fab.update_fab_permswebserver.auth_rate_limited→fab.auth_rate_limitedwebserver.auth_rate_limit→fab.auth_rate_limitkubernetes→kubernetes_executorAUTH_ROLE_PUBLICin FAB auth manager #42280core.check_slasDAGsla_miss_callbackBaseOperatorsladag_ignore_file_syntaxhas changedDAG.max_active_runsbehavior has been changeddefaultas auth backend #43096airflow.api.auth.backend.default→airflow.providers.fab.auth_manager.api.auth.backend.sessionlogging.enable_task_context_loggerairflow.executors.*airflow.hook.*trigger_rule=TriggerRule.ALWAYSin a dynamic mapped task #43368trigger_rule=TriggerRule.ALWAYSis blocked in a dynamic mapped tasktask_failtable #43490airflow/configuration.py#43530airflow.config.get→airflow.config.conf.getairflow.config.getboolean→airflow.config.conf.getbooleanairflow.config.getfloat→airflow.config.conf.getfloatairflow.config.getint→airflow.config.conf.getintairflow.config.has_option→airflow.config.conf.has_optionairflow.config.remove_option→airflow.config.conf.remove_optionairflow.config.as_dict→airflow.config.conf.as_dictairflow.config.set→airflow.config.conf.setairflow.config.→airflow.config.conf.airflow.utils.dates#43533airflow.utils.dates.parse_execution_dateairflow.utils.dates.round_timeairflow.utils.dates.scale_time_unitsairflow.utils.dates.infer_time_unitairflow.PY36→if sys.version_info >== (3, 6)airflow.PY37→if sys.version_info >== (3, 7)core.strict_dataset_uri_validationtraces.otel_task_log_eventmetrics.timer_unit_consistencyRelated issues
No response
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: