[WIP] use traefik for the proxy

[minrk]

Once this is ready, we'll be able to have multiple proxy replicas, enabling restart/upgrade without any downtime while the proxy restarts.

Tasks:

• remove autohttps, internal ingress, kube-lego
• start etcd (as single container in hub pod, using hub-db-dir for storage)
• replace configurable-http-proxy with traefik
• enable auth for etcd (use existing proxy.secretToken)
• enable auth for traefik api (also use existing proxy.secretToken)
• test with multiple traefik replicas
• test automatic provisioning of certificates by meeting letsencrypt challenges
• test TLS-termination with use of provisioned certificates
• run hubtraf with https, multiple traefik replicas

cc @GeorgianaElena who's done all the great work implementing https://github.com/jupyterhub/traefik-proxy

closes #1226
closes #1364

[consideRatio]

@minrk @GeorgianaElena woooooo I'm excited about this!

[minrk]

This is super close to working (it works for me). The biggest headache is getting authentication setup, because both etcd and traefik work pretty hard to prevent you launching them with simple passwords/token auth set up, which is what we want.

1. etcd requires launching insecurely and then enabling auth via insecure etcdctl calls (this appears to be working now), and
2. traefik uses Apache basic or digest auth and there doesn't appear to be a way to put the token into the template in the right format without requiring the user to specify them in their config file after calling htpasswd on them. This is not insurmountable, but it's pretty inconvenient. One hurdle is that these use base64-encoded digests and the sprig digest functions output hex (and base64-encoded hex is not the same as base64-encoded original bytes). The Apache Basic auth spec does define a 'plain' unencrypted format that isn't supported everywhere, and traefik is indeed one of the places that it's not supported.

I'm trying to avoid forcing the user to generate more secrets to set up one thing, but I may punt and say that you need to call htpasswd to paste it into the form traefik wants. That's pretty annoying, though. The alternative is to rely on network policies to protect the proxy (maybe not the best).

[minrk]

I gave up trying to figure out how to enable traefik auth, given only a single token. That doesn't seem to be possible. Instead, users are just going to have to call htpassword to get the chart started. I really dislike this, but I don't see any way around it.

[yuvipanda]

How about:

1. Make our own traefik image
2. CMD is a Python Script that does the things we need it to do. We can ship htpasswd here.
3. Then it execs to traefik

This should help avoid requiring users to run htpasswd on their own. What do you think?

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/server-becomes-stale-in-a-matter-of-days-even-hours/1920/7

[consideRatio]

Review notes

ACME certificate provisioning (Let's encrypt)

It seems like the traefik docs on ACME indicate that the use of acme.storageFile as used in the configmap.yaml file is deprecated, and that the certificates could be stored in the key value store instead. This would perhaps also remove the need for us to have a PVC?

Use a K8s Secret instead of a ConfigMap

We add proxy.secretToken into the configmap.yaml, so it should be made a secret instead I figure.

acme volume / acme.json file

We are trying to persist the acme.json? Hmmm... But it sais we use an emptyDir volume mount for the /acme folder. And, it seems like we are not using the PVC created perhaps with purpose to persist the acme.json file. If we would use a small PVC for this file, we end up in trouble whenever we get two replicas of this pod as they cannot both mount it in a ReadWriteOnce fasion, and ReadWriteMany would require NFS files etc. In short, this is not a viable option I think.

UPDATE: Action points

• Remove the Acme PVC
• Remove the storageFile reference, we already have the storage entry towards the KV store
• Make the ConfigMap a Secret
• Make the proxy's container relating to traefik startup using config in the KV store using basic command line arguments with details on how to communicate with the KV store
  There was no reason to do this as we understand it.
• Make the traefik prefix be "/like-this" instead of "/like-this/"
• Perhaps mount our trafik.toml into /etc/traefik/ where traefik expects to find such file anyhow, in the initContainer.
• Pass all etcd config through the command line options and strip all etcd config from the configmap. Then, we don't need to make the configmap a secret!
• Investigate what happens with the communication between Etcd and Traefik when Etcd pod restarts (along with the Hub ~ ET connection problem)
• Transition to use Consul instead of ETCD
  The reason for doing is relates to performance, currently Traefik cannot work performant with ETCD or Consul when there are plenty of routes, like there are in mybinder.org for example. But, there is work done towards making Traefik performant with Consul.
• Use a passthrough configuration for traefik.toml where we rely on the toToml command and some modification of the provided yaml that is to be written toToml based on other configuration where needed, just like i did in mattermost-team-edition. For reference on traefik.toml, see the treafik configuration docs.

[manics]

@consideRatio Unfortunately it sounds like Traefik can't save the ACME certificate as a K8S secret, it needs a separately deployed K-V store: traefik/traefik#2542

According to traefik/traefik#2542 (comment) the best suggestion if you don't want to setup your own K-V store is to use cert-manager.

[GeorgianaElena]

I think there are a few things that changed since first opening this PR and also some questions we need to answer before moving forward with this.

What changed

• We replaced the kube-lego + nginx with Traefik v2 for autohttps (more info about this here: Replace kube-lego + nginx ingress with traefik #1539)
• Traefik v2
  • offers support for consul and etcd backends since version 2.2 (Support for Traefik V2 traefik-proxy#97 (comment))
  • introduced IngressRoute provider for managing the routing table (https://docs.traefik.io/providers/kubernetes-crd/)

Questions

• Should we switch to Traefik V2 in the jupyterhub-traefik-proxy implementation now that it offers consul/etcd support and we already use it for autohttps?
  Possible downside: version 2 drops support for storing certificates in the key-value stores. Let's Encrypt in an HA setup is still possible, but only though third-party software, like CertManager or potentially by using Traefik Enterprise Edition. Refs:
  * v2 & Let's Encrypt in an HA setup traefik/traefik#5792
  * https://docs.traefik.io/providers/kubernetes-crd/#letsencrypt-support-with-the-custom-resource-definition-provider
  * https://community.containo.us/t/confusion-about-tls-certficates-stores-and-how-they-fit-together/3026/3

• Should we add a TraefikIngressRouteProxy to jupyterhub-traefik-proxy and use that in Z2JH instead of a key-value store TraefikProxy?
  There are a couple of issues where pros and cons of this were discussed:

  • IngressRoute provider proxy traefik-proxy#85
  • Considerations of traefik-proxy, z2jh, and traefik v2 traefik-proxy#88

  I'll try to summarize them here:
  Pros:
  - Kubernetes-native (routing table as CRD objects)
  - No need to manage any extra external services (etcd/consul)
  - High availability support

  Cons:
  - The highly available mode can't be used with Let's Encrypt unless Traefik enterprise version is used
  - Create services/endpoints to route to instead of pods

[mofanke]

@minrk I have a question about this ,when use traefik for the proxy as HA mode, can i disiable check_routes when jupyterhub started or other situations?

[srggrs]

What's the status of this? My need is just with the simple toeml implementation, no containers, no consul or etcd. Just want to know if this is ready with simple toeml. Thanks

[minrk]

@srggrs the traefik proxy is working great with toml config. This PR is only about integrating it as the default proxy in the helm chart. You can check out the littlest jupyterhub for an example of deploying traefik proxy with toml config on a single VM.

@mofanke you shouldn't disable check_routes, since that is still a recovery/reconciliation step in case of state changes on JupyterHub's side. You can increase the interval, though, if you think the issues are less likely to occur.

[srggrs]

@minrk thank you! Sorry I meant if was ready for traefik v2, but I realised I'm in the wrong issue. I got lost in chasing issues...

[lxm]

any progress here?

[chancez]

What if we already run consul? Seems redundant to have this configured, especially since it doesn't have TLS, etc.

[chancez]

I feel somewhat mixed about this. I don't really want to run another loadbalancer at all and I already have ingress controllers in my cluster which work just fine, is it possible to instead just use my existing ingress? How does that play into internal SSL?

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/large-notebooks-save-fails-with-failed-to-fetch/9022/8

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/replace-the-configurable-http-proxy-chp-with-the-traefik-proxy-as-a-cluster-ip-service-using-z2jk/12587/2

[Vaibhav1919]

Hi everyone , Is there any update on this PR , i.e. is traefik supported in z2jh? Can i use this PR to create multiple replicas of proxy.

[consideRatio]

Hey @minrk! In this PR I recall that we tried replacing autohttps (traefik) + proxy (chp) for a single traefik proxy deployment with X replicas, but concluded that they didn't support multiple replicas in conjunction with integration to acquire certs from Let's encrypt.

Due to this significant realization we arrived at part way through this PR, I'll close this PR as I see no clear path forward. We could go for traefik to replace chp and the jupyterhub proxy class representing traefik to replace the jupyterhub proxy class representing chp, but we wouldn't get the gains we hoped for. If we want to pursue this anyhow, I'd like to open a discussion in an issue at this point I think to overview the updated motivations more clearly.

[minrk]

Yes, I think that's fine. I still think we should try to get traefik going, but it's so weird and frustrating that the two biggest selling points for us - native ACME support and multiple replicas, seem to be mutually exclusive!