Feature/add slas #359

edualvess · 2024-05-08T22:50:17Z

Overview

This PR addresses the development of notification callbacks and the enabling of SLA checks for tasks that could potentially take longer to finish.

Key Changes

Defined alert_sla_miss function to send notifications to designed Slack channels leveraging Sentry integration
Setup DAG-level parameter sla_miss_callback with the newly defined callback function for the required DAGs
Setup appropriate SLA for each of the desired tasks
Extra: Increased task_timeout value for the get_ledger_range_from_times in order to avoid task retries while processing XCOM

…lerts

dags/stellar_etl_airflow/build_dbt_task.py

chowbao · 2024-05-15T00:30:08Z

airflow_variables_dev.json

+  },
+  "task_sla": {
+    "get_ledger_range_from_times": 240,
+    "export_task": 240,


I think we need to adjust some of these sla times. Like for instance this export_task sla 240 will never be hit because the task_timeout for export_task is 180.

Also the current sla times are too aggressive from all the alerts we are getting in the slack channel #alerts-hubble-testnet

I think we need to adjust some of these sla times. Like for instance this export_task sla 240 will never be hit because the task_timeout for export_task is 180.

There's an interesting behavior going on for the export_task. Even though the task_timeout is set for 180 seconds, the tasks usually run for around 220 seconds without triggering the timeout. Differently from time_task, where the timeout triggers in the middle of the job.

I'll push the commits where I increase the timeout for export_task too, which makes more sense to the actual behavior of the pipeline.

Also the current sla times are too aggressive from all the alerts we are getting in the slack channel #alerts-hubble-testnet

Those have been reviewed through the suggested buffer and will be updated

chowbao · 2024-05-15T00:37:11Z

I believe you are missing the sla= parameter for some more the the build_* functions like for BigQueryInsertJobOperator.

I'd do a grep/search for all the operators/build functions in dags/stellar_etl_airflow/ and see where sla= can be added. And then update the task_sla airflow variable. Or include a default value in task_sla that faster running tasks can be set to.

chowbao · 2024-05-24T18:02:27Z

airflow_variables_prod.json

@@ -238,9 +238,33 @@
    "build_copy_table": 180,
    "build_dbt_task": 6000,


Can we lower the build_dbt_task timeout value? I don't think there is anything currently that should run over 1.5 hours unless it is a backfill

chowbao · 2024-05-24T18:03:49Z

airflow_variables_prod.json

+    "snapshot_state": 840,
+    "elementary_dbt_sdf_marts": 120,
+    "build_bq_insert_job": 120,
+    "sandbox_create": 120


I think the sandbox update task_sla is missing

I think they are also missing for like build_delete_data_task.

I think there should be a task_sla for every task in airflow

chowbao · 2024-05-29T19:37:51Z

dags/stellar_etl_airflow/build_time_task.py

+    #    args.extend(
+    #        ["--cloud-storage-bucket", Variable.get("gcs_exported_data_bucket_name")]
+    #    )
+    #    args.extend(["--cloud-provider", "gcp"])


I think we should remove the code that is currently from the dev branch PR #371 it is still in development and has a bunch of commented out code in it for testing

* HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]>

* HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]>

* [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]>

* [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: sydneynotthecity <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]>

* HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) * Updated SLA values for dbt eho related tasks (#384) * Patch/merge conflict resolution (#386) * [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * Help (#388) * [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * Patch/fix release conflicts (#390) * [PRODUCTION] Update production Airflow environment (#370) * requirements updated * sqlfluff * Use the correct gcs bucket (#372) --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#375) * requirements updated * Update READMEs --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> --------- Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> * [PRODUCTION] Update production Airflow environment (#378) * HUBBLE-409 Update/delete airflow variable names (#377) * Updates for testnet-reset (#380) * Feature/add slas (#359) --------- Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> * [PRODUCTION] Update production Airflow environment (#382) * HUBBLE-409 Update/delete airflow variable names (#377) * test time task without affinity * delete affinity from repo * dbt_ * testing config related to stellar_marts * documented * test without var dbt_dataset * with dbt_dataset again * dbt_dataset_for_test * comment excluded * Updates for testnet-reset (#380) * Feature/add slas (#359) * Created alert_sla_miss callback funcion * Defined test value for export task SLA * Defined sla_miss_callback for DAG level parameter * Increased time task timeout value * Added dev Airflow variable for tasks SLA values * Added slack notification logic for the SLA miss callback * Added SLA parameter to the relavant task building functions * Removed SLA miss callback function and related logic * Added refactored logic for sla miss callback function * Added sla miss callback to desired dags * Adjusted callback logic for sentry integration * Added task SLA values for prod Airflow variables * Changed alert message to better distinguish SLA miss from task fail alerts * Update dev tasks sla and timeout values * Update prod tasks sla and timeout values * Added default sla value to dag default args * Added missing SLA param to build task functions * Standardizing logic for fetching SLA param value * Added sla miss callback reference * Remove default SLA from default args * Set missing sla params to tasks * Updated values for sla variables * Updated build_dbt_task timeout value * Changed variable names to match new naming reference --------- Co-authored-by: chowbao <[email protected]> * Update stellar-etl image (#381) * Update stellar-etl image (#383) --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> --------- Co-authored-by: sydneynotthecity <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Laysa de Sousa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> --------- Co-authored-by: Laysa Bitencourt <[email protected]> Co-authored-by: chowbao <[email protected]> Co-authored-by: Eduardo Alves <[email protected]> Co-authored-by: sydneynotthecity <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

edualvess added 14 commits May 8, 2024 16:46

Created alert_sla_miss callback funcion

8257b56

Defined test value for export task SLA

99f6204

Defined sla_miss_callback for DAG level parameter

e1acb56

Merge branch 'master' into feature/add_slas

9a0852f

Increased time task timeout value

6f85078

Added dev Airflow variable for tasks SLA values

6d01a09

Added slack notification logic for the SLA miss callback

9911c3b

Added SLA parameter to the relavant task building functions

8982c76

Removed SLA miss callback function and related logic

e0e0ea7

Added refactored logic for sla miss callback function

88c9da2

Added sla miss callback to desired dags

accb888

Adjusted callback logic for sentry integration

bc45c40

Added task SLA values for prod Airflow variables

ad5a1b4

Changed alert message to better distinguish SLA miss from task fail a…

f15c2d6

…lerts

edualvess marked this pull request as ready for review May 15, 2024 00:15

edualvess requested a review from a team as a code owner May 15, 2024 00:15

chowbao reviewed May 15, 2024

View reviewed changes

dags/stellar_etl_airflow/build_dbt_task.py Show resolved Hide resolved

chowbao reviewed May 15, 2024

View reviewed changes

chowbao and others added 10 commits May 14, 2024 20:37

Merge branch 'master' into feature/add_slas

c5fa61e

Merge branch 'master' into feature/add_slas

e8c7ca2

Merge branch 'master' into feature/add_slas

7660175

Update dev tasks sla and timeout values

9d5b32a

Update prod tasks sla and timeout values

fb1d4e8

Added default sla value to dag default args

efd34e2

Added missing SLA param to build task functions

ffc003b

Standardizing logic for fetching SLA param value

1dfdd23

Added sla miss callback reference

7beb0aa

Remove default SLA from default args

7f0f421

chowbao reviewed May 24, 2024

View reviewed changes

chowbao reviewed May 29, 2024

View reviewed changes

edualvess added 2 commits June 4, 2024 15:53

Set missing sla params to tasks

6ef6453

Updated values for sla variables

48d2e69

edualvess force-pushed the feature/add_slas branch from 62f859a to 48d2e69 Compare June 5, 2024 11:31

edualvess added 2 commits June 5, 2024 08:34

Merge branch 'master' into feature/add_slas

b91062d

Updated build_dbt_task timeout value

f28457d

chowbao approved these changes Jun 5, 2024

View reviewed changes

edualvess added 2 commits June 12, 2024 13:30

Merge branch 'master' into feature/add_slas

ee2b3ff

Changed variable names to match new naming reference

e85dba9

edualvess merged commit b7156c3 into master Jun 12, 2024
4 checks passed

edualvess deleted the feature/add_slas branch June 19, 2024 01:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/add slas #359

Feature/add slas #359

edualvess commented May 8, 2024 •

edited

Loading

chowbao May 15, 2024

chowbao May 15, 2024

edualvess May 15, 2024

edualvess May 15, 2024

chowbao commented May 15, 2024

chowbao May 24, 2024

chowbao May 24, 2024

chowbao May 24, 2024 •

edited

Loading

chowbao May 29, 2024 •

edited

Loading

		@@ -238,9 +238,33 @@
		"build_copy_table": 180,
		"build_dbt_task": 6000,

Feature/add slas #359

Feature/add slas #359

Conversation

edualvess commented May 8, 2024 • edited Loading

Overview

Key Changes

chowbao May 15, 2024

Choose a reason for hiding this comment

chowbao May 15, 2024

Choose a reason for hiding this comment

edualvess May 15, 2024

Choose a reason for hiding this comment

edualvess May 15, 2024

Choose a reason for hiding this comment

chowbao commented May 15, 2024

chowbao May 24, 2024

Choose a reason for hiding this comment

chowbao May 24, 2024

Choose a reason for hiding this comment

chowbao May 24, 2024 • edited Loading

Choose a reason for hiding this comment

chowbao May 29, 2024 • edited Loading

Choose a reason for hiding this comment

edualvess commented May 8, 2024 •

edited

Loading

chowbao May 24, 2024 •

edited

Loading

chowbao May 29, 2024 •

edited

Loading