Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document Data Index Horizontal Scalling #682

Open
ricardozanini opened this issue Oct 22, 2024 · 0 comments
Open

Document Data Index Horizontal Scalling #682

ricardozanini opened this issue Oct 22, 2024 · 0 comments
Assignees

Comments

@ricardozanini
Copy link
Member

Currently, Data Index is deployed as a vanilla Kubernetes Deployments. This approach leads administrators and users to assume that it's safe to scale the .spec.dataIndex.podTemplate.replicas to a number greater than 1.

This is not necessarily true in all cases. The data index is not designed for multiple writers unless the new "Eventing Grouping" feature is enabled. In this situation, it might work.

So, the documentation must reflect the following:

  1. The data index should be deployed with sharding in mind. For example, a namespace for Finance workflows should have a Data Index specifically for that data cluster.
  2. The cluster-wide deployment can be used and only scale vertically (Hardware Resources); since that can be a single point of failure, the database layer must be robust and designed for the expected load for the whole workflows in the cluster.

We need more information to document the horizontal scale with grouping events. So, in the near future, when @gmunozfe and @fjtirado come up with the solution and the benchmark, we can document and recommend this architectural deployment approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants