Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Notebook Controller reconcileHandler incorrectly reports ERRORs when reconciling #62

Open
nishant-dash opened this issue Oct 1, 2024 · 0 comments

Comments

@nishant-dash
Copy link

I see that on various notebook reconciliations, the reconcile handler reports the same error, for example

2024-10-01T13:07:25.847Z [jupyter-controller] 1.7277880458479142e+09	ERROR	controller.notebook	Reconciler error	{"reconciler group": "kubeflow.org", "reconciler kind": "Notebook", "name": "mini-notebook-gpu", "namespace": "<REDACTED>", "error": "Operation cannot be fulfilled on notebooks.kubeflow.org \"mini-notebook-gpu\": the object has been modified; please apply your changes to the latest version and try again"}

However the error is not critical, since it eventually reconciles it ok. This red herring in the logs messes with the alerts from JupyterControllerRuntimeReconciliationErrorsExceedThreshold.

@github-project-automation github-project-automation bot moved this to Needs Triage in Kubeflow Notebooks Oct 1, 2024
@nishant-dash nishant-dash changed the title Jupyter Controller reconcileHandler incorrectly reports ERRORs when reconciling Notebook Controller reconcileHandler incorrectly reports ERRORs when reconciling Oct 25, 2024
orfeas-k added a commit to canonical/notebook-operators that referenced this issue Nov 6, 2024
This comments out an alert rule that's always `Firing` due to kubeflow/notebooks#62

Ref #409
orfeas-k added a commit to canonical/notebook-operators that referenced this issue Nov 6, 2024
This comments out an alert rule that's always `Firing` due to kubeflow/notebooks#62

Ref #409
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Needs Triage
Development

No branches or pull requests

1 participant