Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DB stuck in read-only mode after upgrading v0.6.2 to v0.6.3 #915

Open
Dimcon opened this issue Feb 6, 2024 · 0 comments
Open

DB stuck in read-only mode after upgrading v0.6.2 to v0.6.3 #915

Dimcon opened this issue Feb 6, 2024 · 0 comments

Comments

@Dimcon
Copy link

Dimcon commented Feb 6, 2024

I've managed to resolve the issue, but just wanted to document it here.

We were running Bitpoke mysql v0.6.2. GKE auto-upgraded our k8s cluster from v1.24 to v1.25 while we weren't looking.
The databases went down, and then back up promptly once we had upgraded the operator to v0.6.3. However all of the databases were stuck in read-only mode.

After checking all the logs (The orchestrator had the offending log lines), I found the service for the operators orchestrator had an additional label (The label was "app: mysql") that wasn't found on the pods. So the service didn't have any endpoints defined.
Removing the additional label resolved the databases to read-write capable almost immediately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant