Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mitigate contention with dense, high-tempo operations #742

Open
BenjaminPelletier opened this issue Mar 16, 2022 · 0 comments
Open

Mitigate contention with dense, high-tempo operations #742

BenjaminPelletier opened this issue Mar 16, 2022 · 0 comments
Labels
dss Relating to one of the DSS implementations feature Issue would improve software P2 Normal priority

Comments

@BenjaminPelletier
Copy link
Member

When prober's scd/test_operation_simple_heavy_traffic_concurrent.py test runs on a real-world cross-data-center distributed CRDB cluster with 100 concurrent operations, often 1 operation mutation will fail (as many as 3 failures observed) with one of the contention-type errors, usually ABORT_REASON_PUSHER_ABORTED.

Even when the number of concurrent operations is reduced to 40, it appears that a failure is sometimes observed even with 50 retries. This is not yet verified as the failure observation was made before #740 was merged, and #740 is what confirms the 50 retries.

While even 10-20 concurrency support should serve all foreseeable medium-term deployments, we should understand this failure better and identify a mitigation to enable future scaling. In the short term, we should reduce the number of concurrent operations to better align test acceptance criteria to current product needs (reducing concurrent operation count). In the long term, we should mitigate the issue and restore the higher concurrency limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dss Relating to one of the DSS implementations feature Issue would improve software P2 Normal priority
Projects
None yet
Development

No branches or pull requests

1 participant