[Core]: Error while creating, editing or deleting SI and SIB

[anurag-rajawat]

Description

Nimbus logs error messages when we create, update or delete SIB, however, the resources including the NP get created, updated or deleted.

Some of these operations are hard to reproduce.

Steps to reproduce

1. Run nimbus and keep an eye on its log

make run

2. Apply SI and SIB

kubectl apply -f examples/namespaced/pkg-mgr-exec-si-sib.yaml

3. Edit or delete the SI and SIB created in the previous step. If you're not getting error logs edit it again to reproduce

and check logs

...
...
2024-02-14T21:40:37+05:30       ERROR   failed to update NimbusPolicy   {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"pkg-mgr-execution-binding","namespace":"default"}, "namespace": "default", "name": "pkg-mgr-execution-binding", "reconcileID": "c2becafc-3fa9-4fdb-8574-040eec46dd38", "error": "NimbusPolicy.intent.security.nimbus.com \"pkg-mgr-execution-binding\" is invalid: spec.rules: Required value"}
github.com/5GSEC/nimbus/internal/controller.(*SecurityIntentBindingReconciler).Reconcile
        /Users/anurag/workspace/nimbus/internal/controller/securityintentbinding_controller.go:99
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-02-14T21:40:37+05:30       ERROR   Reconciler error        {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"pkg-mgr-execution-binding","namespace":"default"}, "namespace": "default", "name": "pkg-mgr-execution-binding", "reconcileID": "c2becafc-3fa9-4fdb-8574-040eec46dd38", "error": "NimbusPolicy.intent.security.nimbus.com \"pkg-mgr-execution-binding\" is invalid: spec.rules: Required value"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-02-14T21:40:37+05:30       INFO    SecurityIntentBinding not found. Ignoring since object must be deleted  {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"pkg-mgr-execution-binding","namespace":"default"}, "namespace": "default", "name": "pkg-mgr-execution-binding", "reconcileID": "e79a294b-e307-45db-a099-cdb27ccc7256"}
2024-02-14T21:40:37+05:30       INFO    SecurityIntentBinding not found. Ignoring since object must be deleted  {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"pkg-mgr-execution-binding","namespace":"default"}, "namespace": "default", "name": "pkg-mgr-execution-binding", "reconcileID": "0efc000c-51e9-4394-ace0-961eaa515e18"}
2024-02-14T21:40:37+05:30       INFO    SecurityIntentBinding not found. Ignoring since object must be deleted  {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"pkg-mgr-execution-binding","namespace":"default"}, "namespace": "default", "name": "pkg-mgr-execution-binding", "reconcileID": "d0ae5415-c5a2-41b9-8dc6-cfd821c6ece1"}

or

...
...
 ERROR   failed to update SecurityIntentBinding status after NimbusPolicy operation      {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"pkg-mgr-execution-binding","namespace":"default"}, "namespace": "default", "name": "pkg-mgr-execution-binding", "reconcileID": "034cc08c-708a-4bb9-b98f-50fcd627a875", "SecurityIntentBinding.Name": "pkg-mgr-execution-binding", "SecurityIntentBinding.Namespace": "default", "error": "Operation cannot be fulfilled on securityintentbindings.intent.security.nimbus.com \"pkg-mgr-execution-binding\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/5GSEC/nimbus/internal/controller.(*SecurityIntentBindingReconciler).Reconcile
        /Users/anurag/workspace/nimbus/internal/controller/securityintentbinding_controller.go:93
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-02-14T21:50:44+05:30       ERROR   Reconciler error        {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"pkg-mgr-execution-binding","namespace":"default"}, "namespace": "default", "name": "pkg-mgr-execution-binding", "reconcileID": "034cc08c-708a-4bb9-b98f-50fcd627a875", "error": "Operation cannot be fulfilled on securityintentbindings.intent.security.nimbus.com \"pkg-mgr-execution-binding\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227

or sometimes you may get similar logs as follows:

...
...
2024-02-14T22:03:21+05:30       INFO    SecurityIntentBinding not found. Ignoring since object must be deleted  {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"pkg-mgr-execution-binding","namespace":"default"}, "namespace": "default", "name": "pkg-mgr-execution-binding", "reconcileID": "8e5e0b5a-50c2-4fea-8f64-a0d01ccef662"}
2024-02-14T22:03:21+05:30       ERROR   Failed to update SecurityIntentBinding after SI deletion/update {"controller": "securityintent", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntent", "SecurityIntent": {"name":"pkg-mgr-execution"}, "namespace": "", "name": "pkg-mgr-execution", "reconcileID": "b9c9b50b-601e-4e60-9a5e-a28311d9a495", "SecurityIntentBinding.Name": "pkg-mgr-execution-binding", "error": "Operation cannot be fulfilled on securityintentbindings.intent.security.nimbus.com \"pkg-mgr-execution-binding\": StorageError: invalid object, Code: 4, Key: /registry/intent.security.nimbus.com/securityintentbindings/default/pkg-mgr-execution-binding, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 3baceb47-0c84-48d4-9a46-efdfcb9c21b1, UID in object meta: "}
github.com/5GSEC/nimbus/internal/controller.(*SecurityIntentReconciler).updateRelatedSIBs
        /Users/anurag/workspace/nimbus/internal/controller/securityintent_controller.go:113
github.com/5GSEC/nimbus/internal/controller.(*SecurityIntentReconciler).Reconcile
        /Users/anurag/workspace/nimbus/internal/controller/securityintent_controller.go:39
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-02-14T22:03:21+05:30       ERROR   failed to update related SecurityIntentBindings after SI deletion       {"controller": "securityintent", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntent", "SecurityIntent": {"name":"pkg-mgr-execution"}, "namespace": "", "name": "pkg-mgr-execution", "reconcileID": "b9c9b50b-601e-4e60-9a5e-a28311d9a495", "SecurityIntent.Name": "pkg-mgr-execution", "error": "Operation cannot be fulfilled on securityintentbindings.intent.security.nimbus.com \"pkg-mgr-execution-binding\": StorageError: invalid object, Code: 4, Key: /registry/intent.security.nimbus.com/securityintentbindings/default/pkg-mgr-execution-binding, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 3baceb47-0c84-48d4-9a46-efdfcb9c21b1, UID in object meta: "}
github.com/5GSEC/nimbus/internal/controller.(*SecurityIntentReconciler).Reconcile
        /Users/anurag/workspace/nimbus/internal/controller/securityintent_controller.go:40
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-02-14T22:03:21+05:30       ERROR   Reconciler error        {"controller": "securityintent", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntent", "SecurityIntent": {"name":"pkg-mgr-execution"}, "namespace": "", "name": "pkg-mgr-execution", "reconcileID": "b9c9b50b-601e-4e60-9a5e-a28311d9a495", "error": "Operation cannot be fulfilled on securityintentbindings.intent.security.nimbus.com \"pkg-mgr-execution-binding\": StorageError: invalid object, Code: 4, Key: /registry/intent.security.nimbus.com/securityintentbindings/default/pkg-mgr-execution-binding, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 3baceb47-0c84-48d4-9a46-efdfcb9c21b1, UID in object meta: "}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /Users/anurag/.local/share/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-02-14T22:03:21+05:30       INFO    SecurityIntent not found. Ignoring since object must be deleted {"controller": "securityintent", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntent", "SecurityIntent": {"name":"pkg-mgr-execution"}, "namespace": "", "name": "pkg-mgr-execution", "reconcileID": "f7e2d218-8325-4dff-aaeb-ac194531c58e"}
2024-02-14T22:03:21+05:30       INFO    SecurityIntentBinding not found. Ignoring since object must be deleted  {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"pkg-mgr-execution-binding","namespace":"default"}, "namespace": "default", "name": "pkg-mgr-execution-binding", "reconcileID": "84467062-486d-4123-97f9-6a774e1a7618"}

Expected behaviour

Nimbus should not log error messages while performing create, update or delete operations on SIB or NP.

[anurag-rajawat]

I couldn't reproduce the error logs similar to first and second type