[CAPPL-403] Workflow Engine Rate Limiter #15939

justinkaseman · 2025-01-15T16:35:05Z

Description

Adds rate limiting on the workflow executions (when the trigger event is being read) of the workflow engine.

There are two buckets of rate limits on every workflow execution:

Globally across all workflows on the node (whether started from the syncer or delegate)
Per User (workflow owner)

Within this there are two types of rate limit:

RPS - allowed executions per seconds
Burst - allowed concurrent executions

When the rate limit has been reached for either bucket, the workflow execution will be skipped.
In this scenario, a Beholder custom message is emitted and a newly added counter metric is incremented.

Configuration

Adds new config to the node .toml config with the following defaults:

[Capabilities.RateLimit]
# GlobalRPS is the global rate limit for the dispatcher.
GlobalRPS = 800 # Default
# GlobalBurst is the global burst limit for the dispatcher.
GlobalBurst = 1000 # Default
# PerSenderRPS is the per-sender rate limit for the dispatcher.
PerSenderRPS = 10 # Default
# PerSenderBurst is the per-sender burst limit for the dispatcher.
PerSenderBurst = 50 # Default

github-actions · 2025-01-15T16:46:16Z

AER Report: CI Core ran successfully ✅

aer_workflow , commit

AER Report: Operator UI CI ran successfully ✅

aer_workflow , commit

core/services/workflows/engine.go

github-actions · 2025-01-21T23:52:35Z

Flakeguard Summary

Ran new or updated tests between develop and 721c7f6 (CAPPL-403).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name	Pass Ratio	Panicked?	Timed Out?	Race?	Runs	Failures	Package	Package Panicked?	Avg Duration	Code Owners
TestConfig_Marshal	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	40ms	Unknown
TestConfig_Marshal/full	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	13.333333ms	Unknown
TestConfig_full	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	3.333333ms	Unknown
TestConfig_setDefaults	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	0s	Unknown

Artifacts

For detailed logs of the failed tests, please refer to the artifact failed-test-results-with-logs.json.

github-actions · 2025-01-22T01:34:47Z

Flakeguard Summary

Ran new or updated tests between develop and ee6ebb9 (CAPPL-403).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name	Pass Ratio	Panicked?	Timed Out?	Race?	Runs	Failures	Package	Package Panicked?	Avg Duration	Code Owners
TestConfig_Marshal	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	40ms	Unknown
TestConfig_Marshal/full	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	13.333333ms	Unknown
TestConfig_full	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	3.333333ms	Unknown
Test_generalConfig_LogConfiguration	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	26.666666ms	Unknown
Test_generalConfig_LogConfiguration/empty	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	0s	Unknown
Test_generalConfig_LogConfiguration/full	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	10ms	Unknown
Test_generalConfig_LogConfiguration/multi-chain	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	10ms	Unknown

Artifacts

For detailed logs of the failed tests, please refer to the artifact failed-test-results-with-logs.json.

…obal vs. per user allowed

github-actions · 2025-01-22T01:55:32Z

Flakeguard Summary

Ran new or updated tests between develop and 70dcac1 (CAPPL-403).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name	Pass Ratio	Panicked?	Timed Out?	Race?	Runs	Failures	Package	Package Panicked?	Avg Duration	Code Owners
TestConfig_Marshal	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	40ms	Unknown
TestConfig_Marshal/full	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	10ms	Unknown
TestConfig_full	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	0s	Unknown
Test_generalConfig_LogConfiguration	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	23.333333ms	Unknown
Test_generalConfig_LogConfiguration/empty	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	0s	Unknown
Test_generalConfig_LogConfiguration/full	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	10ms	Unknown
Test_generalConfig_LogConfiguration/multi-chain	0.00%	false	false	false	3	3	github.com/smartcontractkit/chainlink/v2/core/services/chainlink	false	10ms	Unknown

Artifacts

For detailed logs of the failed tests, please refer to the artifact failed-test-results-with-logs.json.

github-actions · 2025-01-22T06:21:43Z

Flakeguard Summary

Ran new or updated tests between develop and 8aeb0e0 (CAPPL-403).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name	Pass Ratio	Panicked?	Timed Out?	Race?	Runs	Successes	Failures	Skips	Package	Package Panicked?	Avg Duration	Code Owners
TestEngineWithHardcodedWorkflow	0.00%	true	false	false	3	0	3	0	github.com/smartcontractkit/chainlink/v2/core/services/workflows	true	0s	@smartcontractkit/keystone

Artifacts

For detailed logs of the failed tests, please refer to the artifact failed-test-results-with-logs.json.

github-actions · 2025-01-22T06:36:29Z

Flakeguard Summary

Ran new or updated tests between develop and b8c4baa (CAPPL-403).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name	Pass Ratio	Panicked?	Timed Out?	Race?	Runs	Successes	Failures	Skips	Package	Package Panicked?	Avg Duration	Code Owners
TestEngine_CustomComputePropagatesBreaks	0.00%	false	false	false	3	0	3	0	github.com/smartcontractkit/chainlink/v2/core/services/workflows	false	32.303333333s	@smartcontractkit/keystone
TestEngine_WithCustomComputeStep	0.00%	false	false	false	3	0	3	0	github.com/smartcontractkit/chainlink/v2/core/services/workflows	false	39.2s	@smartcontractkit/keystone

Artifacts

For detailed logs of the failed tests, please refer to the artifact failed-test-results-with-logs.json.

github-actions · 2025-01-22T06:59:13Z

Flakeguard Summary

Ran new or updated tests between develop and 5896f80 (CAPPL-403).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name	Pass Ratio	Panicked?	Timed Out?	Race?	Runs	Successes	Failures	Skips	Package	Package Panicked?	Avg Duration	Code Owners
TestEngine_CustomComputePropagatesBreaks	0.00%	false	false	false	3	0	3	0	github.com/smartcontractkit/chainlink/v2/core/services/workflows	false	29.966666666s	@smartcontractkit/keystone
TestEngine_WithCustomComputeStep	0.00%	false	false	false	3	0	3	0	github.com/smartcontractkit/chainlink/v2/core/services/workflows	false	35.896666666s	@smartcontractkit/keystone

Artifacts

For detailed logs of the failed tests, please refer to the artifact failed-test-results-with-logs.json.

core/config/toml/types.go

core/services/chainlink/application.go

docs/CONFIG.md

github-actions · 2025-01-22T12:11:42Z

Flakeguard Summary

Ran new or updated tests between develop and ca9fe40 (CAPPL-403).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name	Pass Ratio	Panicked?	Timed Out?	Race?	Runs	Successes	Failures	Skips	Package	Package Panicked?	Avg Duration	Code Owners
TestEngine_CustomComputePropagatesBreaks	0.00%	false	false	false	3	0	3	0	github.com/smartcontractkit/chainlink/v2/core/services/workflows	false	32.766666666s	@smartcontractkit/keystone
TestEngine_WithCustomComputeStep	0.00%	false	false	false	3	0	3	0	github.com/smartcontractkit/chainlink/v2/core/services/workflows	false	39.3s	@smartcontractkit/keystone

Artifacts

For detailed logs of the failed tests, please refer to the artifact failed-test-results-with-logs.json.

github-actions · 2025-01-22T13:19:49Z

Flakeguard Summary

Ran new or updated tests between develop and aceb93d (CAPPL-403).

View Flaky Detector Details | Compare Changes

Found Flaky Tests ❌

Name	Pass Ratio	Panicked?	Timed Out?	Race?	Runs	Successes	Failures	Skips	Package	Package Panicked?	Avg Duration	Code Owners
TestEngine_CustomComputePropagatesBreaks	0.00%	false	false	false	3	0	3	0	github.com/smartcontractkit/chainlink/v2/core/services/workflows	false	30.523333333s	@smartcontractkit/keystone
TestEngine_WithCustomComputeStep	0.00%	false	false	false	3	0	3	0	github.com/smartcontractkit/chainlink/v2/core/services/workflows	false	37.013333333s	@smartcontractkit/keystone

Artifacts

For detailed logs of the failed tests, please refer to the artifact failed-test-results-with-logs.json.

cl-sonarqube-production · 2025-01-22T14:19:12Z

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

Draft

fc727e5

justinkaseman force-pushed the CAPPL-403 branch from 748b086 to fc727e5 Compare January 15, 2025 18:14

cedric-cordenier reviewed Jan 16, 2025

View reviewed changes

core/services/workflows/engine.go Outdated Show resolved Hide resolved

Move engine rate limit to trigger executions, rather than steps

d551d0b

Engine RateLimiter config from node config

ee6ebb9

justinkaseman force-pushed the CAPPL-403 branch from 721c7f6 to ee6ebb9 Compare January 22, 2025 01:19

justinkaseman added 2 commits January 21, 2025 17:37

Clone gateway ratelimiter to workflow ratelimiter to differentiate gl…

78e0fcc

…obal vs. per user allowed

Merge branch 'develop' into CAPPL-403

70dcac1

justinkaseman added 4 commits January 21, 2025 18:06

RateLimiterConfig -> Config

755542c

(test): Add capability.ratelimit to TOML testdata

1d2addb

make config docs

9ae0bc7

(test): Add engine rate limiter unit tests

8aeb0e0

justinkaseman changed the title ~~[DRAFT] Workflow Engine Rate Limiter~~ Workflow Engine Rate Limiter Jan 22, 2025

justinkaseman requested a review from cedric-cordenier January 22, 2025 06:09

justinkaseman marked this pull request as ready for review January 22, 2025 06:09

justinkaseman requested review from a team as code owners January 22, 2025 06:09

justinkaseman added 2 commits January 21, 2025 22:20

Soothe linter

8ab9214

Fix stray nil in test

b8c4baa

justinkaseman changed the title ~~Workflow Engine Rate Limiter~~ [CAPPL-403] Workflow Engine Rate Limiter Jan 22, 2025

(test): Set chan timeout to 1s

5896f80

cedric-cordenier reviewed Jan 22, 2025

View reviewed changes

core/config/toml/types.go Show resolved Hide resolved

cedric-cordenier reviewed Jan 22, 2025

View reviewed changes

core/services/chainlink/application.go Outdated Show resolved Hide resolved

cedric-cordenier reviewed Jan 22, 2025

View reviewed changes

docs/CONFIG.md Show resolved Hide resolved

cedric-cordenier previously approved these changes Jan 22, 2025

View reviewed changes

Small fixes

078e8c1

cedric-cordenier dismissed their stale review via 078e8c1 January 22, 2025 11:05

Add logging

0284768

cedric-cordenier force-pushed the CAPPL-403 branch from ca9fe40 to 377049c Compare January 22, 2025 13:02

Merge branch 'develop' into CAPPL-403

aceb93d

cedric-cordenier force-pushed the CAPPL-403 branch from 377049c to aceb93d Compare January 22, 2025 13:04

Fix tests

acf356e

cedric-cordenier approved these changes Jan 22, 2025

View reviewed changes

ettec approved these changes Jan 22, 2025

View reviewed changes

cedric-cordenier added this pull request to the merge queue Jan 22, 2025

Merged via the queue into develop with commit c54d23d Jan 22, 2025
175 of 177 checks passed

cedric-cordenier deleted the CAPPL-403 branch January 22, 2025 15:05

agparadiso approved these changes Jan 22, 2025

View reviewed changes

github-actions bot mentioned this pull request Jan 22, 2025

[DO NOT MERGE] Changeset Release Preview - v2.20.0 #13148

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CAPPL-403] Workflow Engine Rate Limiter #15939

[CAPPL-403] Workflow Engine Rate Limiter #15939

justinkaseman commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 22, 2025

github-actions bot commented Jan 22, 2025

github-actions bot commented Jan 22, 2025

github-actions bot commented Jan 22, 2025

github-actions bot commented Jan 22, 2025

github-actions bot commented Jan 22, 2025

github-actions bot commented Jan 22, 2025

cl-sonarqube-production bot commented Jan 22, 2025

[CAPPL-403] Workflow Engine Rate Limiter #15939

[CAPPL-403] Workflow Engine Rate Limiter #15939

Conversation

justinkaseman commented Jan 15, 2025 • edited Loading

Description

Configuration

github-actions bot commented Jan 15, 2025 • edited Loading

AER Report: CI Core ran successfully ✅

AER Report: Operator UI CI ran successfully ✅

github-actions bot commented Jan 21, 2025

Flakeguard Summary

Found Flaky Tests ❌

Artifacts

github-actions bot commented Jan 22, 2025

Flakeguard Summary

Found Flaky Tests ❌

Artifacts

github-actions bot commented Jan 22, 2025

Flakeguard Summary

Found Flaky Tests ❌

Artifacts

github-actions bot commented Jan 22, 2025

Flakeguard Summary

Found Flaky Tests ❌

Artifacts

github-actions bot commented Jan 22, 2025

Flakeguard Summary

Found Flaky Tests ❌

Artifacts

github-actions bot commented Jan 22, 2025

Flakeguard Summary

Found Flaky Tests ❌

Artifacts

github-actions bot commented Jan 22, 2025

Flakeguard Summary

Found Flaky Tests ❌

Artifacts

github-actions bot commented Jan 22, 2025

Flakeguard Summary

Found Flaky Tests ❌

Artifacts

cl-sonarqube-production bot commented Jan 22, 2025

Quality Gate passed

justinkaseman commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading