Include fault filter in cilium envoy for service-mesh? (or: use a provided envoy?)

[sethp-verica]

Hello! As mentioned on slack, I was experimenting with using the service-mesh beta's CilumEnovyConfig CRD as a way to do some low-level programming of the envoy proxy, specifically looking to turn on the fault injection filter.

It looks like the envoy in question (this one) is intentionally pretty tailored to the specific use-cases "known" to the cilium mesh, and as part of that it disables a lot of the upstream filters (including the fault filter). That makes sense to me, but also seems somewhat in tension with exposing the raw config as a CRD, as that lead me to believe it was an "escape hatch" for doing things that the mesh didn't yet support.

So, the very specific question here is: would you be amenable to updating the cilium distribution of envoy to include Just One More filter?

Alternatively (and unfortunately this may be a better discussion for cilium/service-mesh-beta), what are your thoughts on a "Bring Your Own Envoy" kind of model? It seems easy enough to override the envoy binary with a volume populated by an init container as a way to try that, but that doesn't get very far unless the cilium agent learns to pass portions of the config it doesn't recognize through unmodified by "preserving unknown fields" (admittedly, I'm way out of my depth with protobuf/go & how envoy's using it, so I don't know the level of effort for what I'm describing).

[sethp-verica]

Possibly related to the second suggestion: #47

[jrajahalme]

We'll be adding Envoy extensions to our build but currently we do not have a set process/policy on which extensions to add. Enabling them all makes the build process painful and also increases the binary size considerably. That said, I don't see any reason why we could not add the fault injection filter.

[alibo]

I found this old PR that implemented the required feature as a proxylib go plugin, but I don't know why it's not merged, it has a really nice and easy-to-use API (comparing to fault envoy filter, however, I'm not sure about its performance):

• Envoy: Add golang based chaos-testing plugin cilium#8061
• https://github.com/cilium/chaos-testing-examples/blob/master/examples/