Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Discover Cassandra nodes via Kubernetes DNS during replication #33

Open
c-w opened this issue Feb 26, 2018 · 2 comments
Open

Discover Cassandra nodes via Kubernetes DNS during replication #33

c-w opened this issue Feb 26, 2018 · 2 comments
Labels
project-fortis-pipeline

Comments

@c-w
Copy link
Contributor

c-w commented Feb 26, 2018

Currently, the helm chart that sets up Cassandra has nodes discovering each other by IP during data replication. In rare situations we've seen this cause issues if Cassandra and Kubernetes get out of sync as to what are the correct IP addresses for the Cassandra workers. To prevent this issue and shield Cassandra from IP changes, we want to front each Cassandra node with a DNS name of a kubernetes service. @xtophs is the expert in this area for further questions.
@c-w
Copy link
Contributor Author

c-w commented Apr 3, 2018

Error screenshots from yet another outage caused by this:

error1
error2
error3

@c-w
Copy link
Contributor Author

c-w commented Apr 3, 2018

Tried to fix the issue with kubectl -n cassandra exec -it cassandra-cluster-cassan-0 -- nodetool removenode 7d0841ec-09fc-4c5c-b67b-412d0d8a0afb and kubectl -n cassandra exec -it cassandra-cluster-cassan-0 -- nodetool removenode 860d09c0-1571-4771-9574-6d537fee57c1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
project-fortis-pipeline
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@c-w