Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dss] Reduce contention on OI creation/mutation endpoint #1004

Merged

Conversation

barroco
Copy link
Contributor

@barroco barroco commented Feb 23, 2024

This PR adds a lock at the beginning of the transactions creating or mutating operational intents.
Contention is mainly due to the ability of the transaction to increment the subscription's notification index.
The lock will ensure transactions with contention on subscriptions on the same cells to wait before starting preventing multiple retries.

Compared to the result reported in the issue #1002, we can identify the following overall test durations (single run) and number of transaction retries:

test step v0.11.0-rc1 #1004
test_create_ops_concurrent 4s 1.4 s
test_mutate_ops_concurrent 5.2s 1.5 s
overall retries 150 17
overall CI run ~ 24s ~ 19s

The following diagram shows the result of scd/test_operation_simple_heavy_traffic_concurrent.py. Test was run on a GCP deployment (3 core-services, 3 cockroach db nodes):

Screenshot 2024-02-23 at 18 26 20

This improvement should address #742.

Note that due to the sensitivity of the change, I tried to keep it as focused as possible. Some opportunities for refactoring will be addressed separately.

@barroco barroco changed the title [dss] Performance improvement of OI creation/mutation endpoint (#1002) [dss] Reduce contention on OI creation/mutation endpoint (#1002) Feb 23, 2024
@barroco barroco force-pushed the 1002-scd-lock-subscription-on-cell branch from 01827ac to c1e2819 Compare February 23, 2024 17:10
@barroco barroco force-pushed the 1002-scd-lock-subscription-on-cell branch from c1e2819 to fe227e2 Compare February 23, 2024 17:15
@barroco barroco marked this pull request as ready for review February 23, 2024 17:19
Copy link
Member

@BenjaminPelletier BenjaminPelletier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's great we can sufficiently alleviate the problem with a solution this simple -- thanks for doing this!

After this P0 is resolved, we can (at a more normal priority) take a look at those follow-up items like refactoring and additional improvements. Thanks again!

@barroco barroco changed the title [dss] Reduce contention on OI creation/mutation endpoint (#1002) [dss] Reduce contention on OI creation/mutation endpoint Feb 23, 2024
@barroco barroco merged commit 9147b9b into interuss:master Feb 23, 2024
6 checks passed
@barroco barroco deleted the 1002-scd-lock-subscription-on-cell branch February 23, 2024 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants