[OFO/flagd] Research and document flagd client support

[toddbaert]

With client spec changes and SDKs on the way, we need to decide what sort of support we want for the client/web in OFO (if any). We already have the required bulk-resolution functionality in flagd, as well as CORs options to support local development

@james-milligan already did some research here with a PoC deployment, I believe. That may feed into this discussion.

If we were to support this, we'd want enhancements to the operator that could facilitate the following:

• Flagd must be exposed via a public ingress (handled by OFO or just documented)
• Flagd must support a bi-directional HTTP2 connections
• A client must be able to access flagd
• Flagd must be horizontally scalable based on load or number of connections

Non functionally, we should consider:

• updates to the threat model
• load considerations: How many persistent connection can be supported?

[skyerus]

POC

I have put together a POC using kubernetes gateway API (KGA) as means to ingest external traffic (intended as successor to an ingress).

The deployment pattern this POC is catered to:

• Deployment of an application exposed by a Service
• Gateway exposes the app's Service via a HTTPRoute and listener (with hostname)

Assuming that the above is true (application is readily deployed and ready to receive external traffic), one can create a ClientSideConfiguration CR (naming TBD).
The creation of a ClientSideConfiguration CR triggers its reconciliation loop which does the following:

• Creates a deployment of flagd
• Creates a service exposing the flagd deployment
• Appends a listener to the Gateway resource (name, port defined by the CR)
• Creates a HTTPRoute exposing the flagd deployment service to the gateway (hostname defined by the CR)

The end result is two exposed applications: the client application and flagd. Both are exposed by the Gateway, which routes the request to the appropriate service based on the matching of the Host header (this is checked against the hostname defined in the Gateway's listeners). The following diagram depicts this.

Note: To get this to work locally I had to add hostnames to my hosts file to route them to localhost. Without this the browser doesn't send the appropriate Host header to the Gateway, preventing it from being able to route the request.

How to run the POC

Prerequisites:

• Kind
• Checkout OFO at the branch in the linked PR
• istioctl (this can be substituted for any other KGA implementor but make sure to replace references to istio with the substitute)

# build OFO
IMG=ghcr.io/open-feature/open-feature-operator:local ARCH=arm64 make docker-build
# create kind cluster
kind create cluster --config ./test/e2e/kind-cluster.yml
# load OFO into cluster
kind load docker-image ghcr.io/open-feature/open-feature-operator:local
# pull ofo-client-app
docker pull ghcr.io/skyerus/ofo-client-app:latest
# load ofo-client-app into cluster 
kind load docker-image ghcr.io/skyerus/ofo-client-app:latest
# deploy OFO
IMG=ghcr.io/open-feature/open-feature-operator:local make deploy-operator
# install KGA CRDs
kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
  { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.6.1" | kubectl apply -f -; }
# install istio gateway
istioctl install --set profile=minimal -y
# deploy resources
kubectl apply -f config/samples/dev.yaml
# add hostnames to hosts file (location of hosts file is macOS specific in this example)
sudo sh -c 'echo "127.0.0.1 ofoclientapp.com" >> /private/etc/hosts'
sudo sh -c 'echo "127.0.0.1 flagd.ofoclientapp.com" >> /private/etc/hosts'
# port forward 8080 to the gateway pod
kubectl port-forward $(kubectl get pods --selector=istio.io/gateway-name=clientsidegateway -o \
  jsonpath='{.items[*].metadata.name}') 8080:80

Go to http://ofoclientapp.com:8080/ in your browser and click the Update color button to verify that the traffic is routed to flagd correctly.

Questions/Concerns

• KGA and its implementors are currently in beta.
• Can KGA and Ingress coexist or should we enforce anyone who wishes to use OFO to deploy flagd to use KGA?
  KGA and ingress are different means of achieving the same result (routing inbound traffic to the correct service). Using both would mean one sits behind the other with duplicated rules, this seems wrong. IMO, if we want to support ingress we could introduce a networkType field to the CRD and split code paths for KGA/ingress. The ingress path would mutate the existing ingress resource in similar fashion to what has been done for KGA
• Do we want to support creating a Gateway if it doesn't already exist or just return an error?
  Initially, for simplicity, return an error if the resource doesn't exist.
• Can local development be improved to not require adding hostnames to hosts file?
  This drawback is only impactful to end-to-end development, which this configuration is not suitable nor intended for. There exists other means of making requests which allow headers (Host header) to be set directly (e.g. curl). These means are more appropriate for assessing the validity of the configured networking (no hosts file meddling required).
• Does KGA support bi-directional HTTP2 connections?
  KGA supports GRPC routes (experimentally for now). This implies support of bi-directional HTTP2 connections.
• Can scalability be demonstrated?
  Kubernetes typical notions of scaling (e.g. Horizontal Pod Autoscaling) are applicable.

Further Steps

• Address the above questions/concerns
• Decide on naming of the ClientSideConfiguration CR
• Decide whether ingress support is required
• Move shared code into their own packages and import to existing invokers (a lot of duplicated code for POC)
• Flesh out POC to include other sync provider deployments (currently only kubernetes provider is available)

[AlexsJones]

g that the above is true (application is readily deployed

Hi, it's me again, playing devil's advocate here.

What's the need to connect to the client side to flagd, what do we gain doing this over using the client side SDKs elsewhere?

If you think about the story behind this; someone is going to be using a browser-based experience but the flags are completely unexposed and only accessible through a custom resource; do you have a use case in mind for this? Are we thinking of also having a Web API accessible through the gateway to program flags?

[skyerus]

What's the need to connect to the client side to flagd, what do we gain doing this over using the client side SDKs elsewhere?

I'm not sure I'm following, are you suggesting that the client connects to flagd indirectly via some existing server?

someone is going to be using a browser-based experience but the flags are completely unexposed and only accessible through a custom resource

The flags are exposed in the demonstrated POC, or are you referring to the mutation of them being unexposed?

do you have a use case in mind for this?

It could enable user interfaces to react dynamically to user context based on flags by going straight to the flag source (flagd).

Are we thinking of also having a Web API accessible through the gateway to program flags?

To alter flagd's internal state you mean? Interesting idea but I'd argue it's out of scope for the topic at hand

[AlexsJones]

I am querying whether there is a desire to have flagD as a provider the client SDK's can connect to is the goal.
I was curious whether there are people interested in this pattern.

For example, the in K8s pattern makes sense when its workloads a backend dev might be controlling or a website you're running from a pod. It gets a bit more complex when you're an iOS developer that hits a KGA endpoint for flagD as a provider. That's where I'm trying to really identify what we're talking about.

[skyerus]

Just so I'm clear, are you suggesting that there might not be enough value in exposing flagd for use by client side applications? For those use-cases, defer to a vendor built for this?

[toddbaert]

I am querying whether there is a desire to have flagD as a provider the client SDK's can connect to is the goal.

There's a definite interest internally at Dynatrace. Specifically usage in web applications (directly from the browser) is desired.

Anecdotally, in the vendor interviews I've been party to with @moredip, we typically heard that client use-cases represent about half of the usage most vendors see.

Some examples of this in partner vendors:

https://docs.flagsmith.com/clients/javascript
https://help.split.io/hc/en-us/sections/12619122908557-Client-side-SDKs
https://docs.cloudbees.com/docs/cloudbees-feature-management-api/latest/api-reference/javascript-browser-api
https://docs.launchdarkly.com/sdk/client-side

I think as a reference implementation, we more or less need to support this important paradigm in flagd.

[skyerus]

See this OFEP comment for the current plan for flagd client support.

[skyerus]

I've created a new OFEP outlining an alternative solution avoiding the previously discussed concerns & limitations.

[beeme1mr]

Addressed by #633