[controller] State-of-the-world reconciliation

[guicassolato]

Add support for state-of-the-world reconciliation, based on controller-runtime.

Closes #15.

---

• 1. Replace cache with a watchable map
• 2. Define a Runnable interface: { Run(stopCh <-chan struct{}) }
• 3. Modify the list of informers of the controller to be a list of generic runnables
• 4. Modify the controller builder option WithInformer to WithRunnable
• 5. Add an informer builder option InformerBuilder(builder)
  • IncrementalInformer → returns a cache.SharedInformer that adds incremental changes directly to the cache (default informer builder)
  • StateReconciler → returns a runnable that simply adds a ListFunc to a slice of state reconciler ListFuncs of the controller (default informer builder)
• 6. Modify informer builder func controller.For to controller.Watch
  • Build the watcher using the builder specified in the options
• 7. Add a controller builder option ManagedBy(manager)
• 8. Implement controller-runtime’s Reconciler interface
  • List from the cluster all resources whose kinds the controller is watching, by triggering all the state reconciler ListFuncs registered with the controller
  • Update the cache atomically with the newly fetched list of resources
• 9. Modify controller.Start():
  • Run all the runnables
  • If a manager was provided, create a controller-runtime controller, passing the manager and the PM controller as the reconciler:

    controller.New("gatewayapi", c.manager, controller.Options{Reconciler: c})
    c.manager.Start()

• 10. Rename CallbackFunc as ReconcileFunc
• 11. Modify the ReconcileFunc to receive a list of ResourceEvents
• 12. Modify controller builder option WithCallback to WithReconciler(ReconcileFunc)
  • Subscribe to the cache map
  • On messages published to the subscription channel:

    if snapshot, ok := <-subscription; ok {
      events := lo.Map(snapshot.Updates, …)
      topology := c.topology.Build(snapshot.State)
      reconcileFunc(ctx, events, topology)
    }

Out of scope of the PR but still addressed here (maybe extract to a separate PR)

• Fix go workspaces for resolving dependencies

TODOs

• Support for CreateEvent UpdateEvent in the state-of-the-world reconciler
• Old object supplied with the UpdateEvents generated by the state-of-the-world reconciler
• Field selector predicate for state-of-the-world reconciler
• Bugfix: topology not thread-safe for read access - crashing occasionally when accessed concurrently by multiple goroutines
• Bugfix: cannot compare Istio AuthorizationPolicies with github.com/google/go-cmp/cmp.Equal:

  cannot handle unexported field at {*v1.AuthorizationPolicy}.Spec.state:
          "istio.io/api/security/v1".AuthorizationPolicy
  consider using a custom Comparer; if you control the implementation of type, you can also consider using an Exporter, AllowUnexported, or cmpopts.IgnoreUnexported
  

[Boomatang]

I was looking over this and there was something that came to mind which I am not sure how we would stand on the issue. Being able to run the controller with or with out the need for the controller runtime is nice and I like the concept. But if I choice not use the controller runtime, it and all it's dependence's are included. It seems wrong to force this dependency, but for now I am not sure how we would approach the issue. Or even if it is an issue.

[guicassolato]

I was looking over this and there was something that came to mind which I am not sure how we would stand on the issue. Being able to run the controller with or with out the need for the controller runtime is nice and I like the concept. But if I choice not use the controller runtime, it and all it's dependence's are included. It seems wrong to force this dependency, but for now I am not sure how we would approach the issue. Or even if it is an issue.

Maybe I'm not seeing it all, but it doesn't seem an issue to me, I guess. As long as the controller-runtime layer is not loaded, it's "only" unreachable code, no?

[KevFan]

Verified envoy-gateway.md and multiple-gateway-providers.md for both delta and state reconciliation mode, and both are working as expected 💯

Changes looks good to me! 🥇