Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

netpol: allowedIngressPorts and interNamespaceAccessLabels config added with defaults retaining 0.9.1 current behavior #1842

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
09e7486
netpol: allow internamespace communication
consideRatio Oct 16, 2020
e9541a6
netpol: add inline comment to help us avoid future confusion
consideRatio Oct 17, 2020
cf6914d
netpol: rework - allow labelled namespaces, and labelled pods in anot…
consideRatio Oct 17, 2020
129957e
netpol: add lots of configuration for default ingress rule
consideRatio Oct 19, 2020
081980f
netpol: simplify config of interNamespace ingress
consideRatio Oct 19, 2020
bed3486
netpol: add autohttps netpol, mimicing proxy netpol
consideRatio Oct 19, 2020
fdc0eec
netpol: deprecate proxy.networkPolicy
consideRatio Oct 19, 2020
99d9b26
netpol: add proxy.chp / proxy.traefik allowHttpIngressUnconditionally…
consideRatio Oct 19, 2020
83c3d56
netpol: ci update
consideRatio Oct 19, 2020
380b674
docs: update example to be aligned with best practice
consideRatio Oct 20, 2020
30e9b53
netpol: replace allowHttpIngressUnconditionally with allowedIngressPo…
consideRatio Oct 20, 2020
c0f66cd
docs: fix typo
consideRatio Oct 20, 2020
ec2aabd
netpol: rename option to interNamespaceAccessLabels
consideRatio Oct 20, 2020
d1dcb4c
netpol: default to ignore all interNamespaceAccessLabels
consideRatio Oct 20, 2020
8a9d608
docs: add schema.yaml entries for networkPolicy configuration
consideRatio Oct 20, 2020
02f1de4
netpol: add changelog entry about breaking change
consideRatio Oct 17, 2020
4e20a60
Apply suggestions from code review
consideRatio Oct 20, 2020
0369b8e
docs: smaller adjustments in a self-review
consideRatio Oct 20, 2020
b67732a
Apply suggestions from code review
consideRatio Oct 22, 2020
16e140a
docs: update all netpol comments to reflect suggestions
consideRatio Oct 22, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Here you can find upgrade changes in between releases and upgrade instructions.

## [0.10]

### [0.10.0]
### [0.10.0-beta.1] (UNRELEASED)

#### Breaking changes:

Expand All @@ -16,17 +16,17 @@ Here you can find upgrade changes in between releases and upgrade instructions.
known to be needed, with the exception of the user pods outgoing
communication, where all outgoing communication is allowed by default.

Note that these network policies only influence network communication in a
Note that these NetworkPolicies only influence network communication in a
Kubernetes cluster if a NetworkPolicy controller enforce them, such as Calico.

With network policies enabled, you may require additional configuration,
especially for deployments that include additional components that access
JupyterHub pods directly (i.e. not through the `proxy-public` service).

See the [security
documentation](https://zero-to-jupyterhub.readthedocs.io/en/latest/administrator/security.html#kubernetes-network-policies)
for more details on this.

- The Helm chart configuration `proxy.networkPolicy` has been removed,
Copy link
Member

@minrk minrk Oct 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is my only concern here. I don't love it because it seems to be baking too much of the implementation into the configuration. When we finally manage to land the traefik proxy implementation (#1162), this configuration structure will change again when nothing relevant to users should really need to. What do users need to know about the separation of autohttps and chp that we couldn't represent without separating them?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will still have a dedicated proxy for TLS termination i think, given that traefik doesnt support HA with lets encrypt.

This helm chart has a separate pod for TLS termination atm, and combining the config of two pods into one has caused it to be confusion in general. I'm think im for doing this even if it was a standalone refactoring without any other purpose to reduce that confusion.

If you feel strongly about it, i can revert it back, locating chp netpol config under proxy.netpol again.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, this change seems like it would increase confusion to have two network policies applying to what users should see as one resource (and indeed will be one resource if we manage to land the traefik proxy). I am thinking of the example of

Q: I set proxy.chp.networkPolicy, why is it not having any effect?
A: that's because you have enabled autohttps, which means that proxy.traefik.networkPolicy governs proxy-public, not proxy.chp.networkPolicy. If you change how you configure https, your network policy will do what you expect, or you need to move the configuration to proxy.traefik.networkPolicy

Or is that not an accurate description of the behavior here?

`proxy.chp.networkPolicy` (proxy pod) and `proxy.traefik.networkPolicy`
(autohttps pod) must be used instead.

## [0.9]

### [0.9.0] - 2020-04-15
Expand Down
4 changes: 2 additions & 2 deletions dev-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,13 @@ proxy:
extraEnv:
- name: LEGO_CA_CERTIFICATES
value: /etc/pebble/root-cert.pem
networkPolicy:
egress: [] # overrides allowance of 0.0.0.0/0
chp:
resources:
requests:
memory: 0
cpu: 0
networkPolicy:
egress: [] # overrides allowance of 0.0.0.0/0

hub:
cookieSecret: cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
Expand Down
54 changes: 54 additions & 0 deletions jupyterhub/schema.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,57 @@ properties:
pullSecrets:
- my-k8s-secret-with-image-registry-credentials
```
networkPolicy: &networkPolicy-spec
type: object
description: |
This configuration regards the creation and configuration of a k8s
_NetworkPolicy resource_.
properties:
enabled:
type: bool
description: |
Toggle the creation of the NetworkPolicy resource for this pod.
ingress:
type: list
description: |
Additional ingress rules to add except those that is known to be needed by the respective pods in the Helm chart.
egress:
type: list
description: |
Additional egress rules to add except those that is known to be needed by the respective pods in the Helm chart.

The default value of this egress is to allow all traffic, except for the `singleuser.networkPolicy.egress`, which is also limiting access to a metadata server that can be exploited.

If you want to restrict egress, you can override this permissive default to be an empty list.
interNamespaceAccessLabels:
type: string
enum:
- accept
- ignore
description: |
This configuration option determines if both namespaces and pods
in other namespaces, that have specific access labels, should be
accepted to allow ingress (set to `accept`), or, if the labels are
to be ignored when applied outside the local namespace (set to
`ignore`).

The available access labels for respective NetworkPolicy resources
are:

- `hub.jupyter.org/network-access-hub: "true"` (hub)
- `hub.jupyter.org/network-access-proxy-http: "true"` (proxy.chp, proxy.traefik)
- `hub.jupyter.org/network-access-proxy-api: "true"` (proxy.chp)
- `hub.jupyter.org/network-access-singleuser: "true"` (singleuser)
allowedIngressPorts:
type: list
description: |
A rule to allow ingress on these ports will be added no matter
what the origin of the request is. The default setting for
`proxy.chp` and `proxy.traefik`'s networkPolicy configuration is
`[http, https]`, while it is `[]` for other networkPolicies.
consideRatio marked this conversation as resolved.
Show resolved Hide resolved

Note that these port names or numbers target a Pod's port name or
number, not a k8s Service's port name or number.
db:
type: object
properties:
Expand Down Expand Up @@ -535,6 +586,7 @@ properties:
Configure the configurable-http-proxy (chp) pod managed by jupyterhub to route traffic
both to itself and to user pods.
properties:
networkPolicy: *networkPolicy-spec
extraCommandLineFlags:
type: list
description: |
Expand Down Expand Up @@ -806,6 +858,7 @@ properties:

See the [Kubernetes docs](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
to learn more about labels.
networkPolicy: *networkPolicy-spec
extraEnv:
type: object
description: |
Expand Down Expand Up @@ -880,6 +933,7 @@ properties:
description: |
Options for customizing the environment that is provided to the users after they log in.
properties:
networkPolicy: *networkPolicy-spec
podNameTemplate:
type: string
description: |
Expand Down
5 changes: 5 additions & 0 deletions jupyterhub/templates/NOTES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@ WARNING: Configuring proxy.https without setting proxy.https.enabled to true is
{{- end }}


{{- if .Values.proxy.networkPolicy }}
{{- fail "DEPRECATION: proxy.networkPolicy has been renamed to proxy.chp.networkPolicy" }}
{{- end }}


{{- if .Values.hub.extraConfigMap }}{{ println }}
DEPRECATION: hub.extraConfigMap is deprecated in jupyterhub chart 0.8.
Use top-level `custom` instead:
Expand Down
38 changes: 35 additions & 3 deletions jupyterhub/templates/hub/netpol.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,49 @@ spec:
- Ingress
- Egress

# IMPORTANT:
# NetworkPolicy's ingress "from" and egress "to" rule specifications require
# great attention to detail. A quick summary is:
#
# 1. You can provide "from"/"to" rules that provide access either ports or a
# subset of ports.
# 2. You can for each "from"/"to" rule provide any number of
# "sources"/"destinations" of four different kinds.
# - podSelector - targets pods with a certain label in the same namespace as the NetworkPolicy
# - namespaceSelector - targets all pods running in namespaces with a certain label
# - namespaceSelector and podSelector - targets pods with a certain label running in namespaces with a certain label
# - ipBlock - targets network traffic from/to a set of IP address ranges
#
# Read more at: https://kubernetes.io/docs/concepts/services-networking/network-policies/#behavior-of-to-and-from-selectors
#
ingress:
{{- with .Values.hub.networkPolicy.allowedIngressPorts }}
# allow incoming traffic to these ports independent of source
- ports:
{{- range $port := . }}
- port: {{ $port }}
{{- end }}
{{- end }}

# allowed pods (hub.jupyter.org/network-access-hub) --> hub
- ports:
- port: http
from:
# source 1 - labeled pods
- podSelector:
matchLabels:
hub.jupyter.org/network-access-hub: "true"
{{- if eq .Values.hub.networkPolicy.interNamespaceAccessLabels "accept" }}
namespaceSelector:
matchLabels: {} # without this, the podSelector would only consider pods in the local namespace
# source 2 - pods in labeled namespaces
- namespaceSelector:
matchLabels:
hub.jupyter.org/network-access-hub: "true"
consideRatio marked this conversation as resolved.
Show resolved Hide resolved
{{- end }}

{{- with .Values.hub.networkPolicy.ingress }}
# default: nothing --> hub
# depends, but default is nothing --> hub
{{- . | toYaml | trimSuffix "\n" | nindent 4 }}
{{- end }}

Expand All @@ -45,15 +77,15 @@ spec:
{{- $_ := merge (dict "componentLabel" "singleuser-server") . }}
{{- include "jupyterhub.matchLabels" $_ | nindent 14 }}

# hub -> Kubernetes internal DNS
# hub --> Kubernetes internal DNS
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53

{{- with .Values.hub.networkPolicy.egress }}
# hub --> default: everything
# hub --> depends, but the default is everything
{{- . | toYaml | trimSuffix "\n" | nindent 4 }}
{{- end }}
{{- end }}
86 changes: 86 additions & 0 deletions jupyterhub/templates/proxy/autohttps/netpol.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
{{- $HTTPS := .Values.proxy.https.enabled -}}
{{- $autoHTTPS := and $HTTPS (and (eq .Values.proxy.https.type "letsencrypt") .Values.proxy.https.hosts) -}}
{{- if and $autoHTTPS .Values.proxy.traefik.networkPolicy.enabled -}}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: autohttps
labels:
{{- include "jupyterhub.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels:
{{- include "jupyterhub.matchLabels" . | nindent 6 }}
policyTypes:
- Ingress
- Egress

# IMPORTANT:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there anything we can do to avoid duplication of this comment in multiple places, to prevent them inadvertently getting out of sync?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicating them breaks the DRY principle, but it also ensures this information is made available to everyone working with the resource which I felt was the most important part.

NetworkPolicy resources are sooooooo hard to get right, and our tests doesn't test all our rules etc.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated all comments to reflect your suggestions in the hub/netpol.yaml file

# NetworkPolicy's ingress "from" and egress "to" rule specifications require
# great attention to detail. A quick summary is:
#
# 1. You can provide "from"/"to" rules that provide access either ports or a
# subset of ports.
# 2. You can for each "from"/"to" rule provide any number of
# "sources"/"destinations" of four different kinds.
# - podSelector - targets pods with a certain label in the same namespace as the NetworkPolicy
# - namespaceSelector - targets all pods running in namespaces with a certain label
# - namespaceSelector and podSelector - targets pods with a certain label running in namespaces with a certain label
# - ipBlock - targets network traffic from/to a set of IP address ranges
#
# Read more at: https://kubernetes.io/docs/concepts/services-networking/network-policies/#behavior-of-to-and-from-selectors
#
ingress:
{{- with .Values.proxy.traefik.networkPolicy.allowedIngressPorts }}
# allow incoming traffic to these ports independent of source
- ports:
{{- range $port := . }}
- port: {{ $port }}
{{- end }}
{{- end }}

# allowed pods (hub.jupyter.org/network-access-proxy-http) --> proxy (http/https port)
- ports:
- port: http
- port: https
from:
# source 1 - labeled pods
- podSelector:
matchLabels:
hub.jupyter.org/network-access-proxy-http: "true"
{{- if eq .Values.proxy.traefik.networkPolicy.interNamespaceAccessLabels "accept" }}
namespaceSelector:
matchLabels: {} # without this, the podSelector would only consider pods in the local namespace
# source 2 - pods in labeled namespaces
- namespaceSelector:
matchLabels:
hub.jupyter.org/network-access-proxy-http: "true"
{{- end }}

{{- with .Values.proxy.traefik.networkPolicy.ingress}}
# depends, but default is nothing --> proxy
{{- . | toYaml | trimSuffix "\n" | nindent 4 }}
{{- end }}

egress:
# autohttps --> proxy (http port)
- ports:
- port: 8000
to:
- podSelector:
matchLabels:
{{- $_ := merge (dict "componentLabel" "proxy") . }}
{{- include "jupyterhub.matchLabels" $_ | nindent 14 }}

# autohttps --> Kubernetes internal DNS
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53

{{- with .Values.proxy.traefik.networkPolicy.egress }}
# autohttps --> depends, but the default is everything
{{- . | toYaml | trimSuffix "\n" | nindent 4 }}
{{- end }}
{{- end }}
53 changes: 47 additions & 6 deletions jupyterhub/templates/proxy/netpol.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
{{- $autoHTTPS := and $HTTPS (and (eq .Values.proxy.https.type "letsencrypt") .Values.proxy.https.hosts) -}}
{{- $manualHTTPS := and $HTTPS (eq .Values.proxy.https.type "manual") -}}
{{- $manualHTTPSwithsecret := and $HTTPS (eq .Values.proxy.https.type "secret") -}}
{{- if .Values.proxy.networkPolicy.enabled -}}
{{- if .Values.proxy.chp.networkPolicy.enabled -}}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
Expand All @@ -17,28 +17,69 @@ spec:
- Ingress
- Egress

# IMPORTANT:
# NetworkPolicy's ingress "from" and egress "to" rule specifications require
# great attention to detail. A quick summary is:
#
# 1. You can provide "from"/"to" rules that provide access either ports or a
# subset of ports.
# 2. You can for each "from"/"to" rule provide any number of
# "sources"/"destinations" of four different kinds.
# - podSelector - targets pods with a certain label in the same namespace as the NetworkPolicy
# - namespaceSelector - targets all pods running in namespaces with a certain label
# - namespaceSelector and podSelector - targets pods with a certain label running in namespaces with a certain label
# - ipBlock - targets network traffic from/to a set of IP address ranges
#
# Read more at: https://kubernetes.io/docs/concepts/services-networking/network-policies/#behavior-of-to-and-from-selectors
#
ingress:
{{- with .Values.proxy.chp.networkPolicy.allowedIngressPorts }}
# allow incoming traffic to these ports independent of source
- ports:
{{- range $port := . }}
- port: {{ $port }}
{{- end }}
{{- end }}

# allowed pods (hub.jupyter.org/network-access-proxy-http) --> proxy (http/https port)
- ports:
- port: http
{{- if or $manualHTTPS $manualHTTPSwithsecret }}
- port: https
{{- end }}
from:
# source 1 - labeled pods
- podSelector:
matchLabels:
hub.jupyter.org/network-access-proxy-http: "true"
{{- if eq .Values.proxy.chp.networkPolicy.interNamespaceAccessLabels "accept" }}
namespaceSelector:
matchLabels: {} # without this, the podSelector would only consider pods in the local namespace
# source 2 - pods in labeled namespaces
- namespaceSelector:
matchLabels:
hub.jupyter.org/network-access-proxy-http: "true"
{{- end }}

# allowed pods (hub.jupyter.org/network-access-proxy-api) --> proxy (api port)
- ports:
- port: api
from:
# source 1 - labeled pods
- podSelector:
matchLabels:
hub.jupyter.org/network-access-proxy-api: "true"
{{- if eq .Values.proxy.chp.networkPolicy.interNamespaceAccessLabels "accept" }}
namespaceSelector:
matchLabels: {} # without this, the podSelector would only consider pods in the local namespace
# source 2 - pods in labeled namespaces
- namespaceSelector:
matchLabels:
hub.jupyter.org/network-access-proxy-api: "true"
{{- end }}

{{- with .Values.proxy.networkPolicy.ingress}}
# default: nothing --> proxy
{{- with .Values.proxy.chp.networkPolicy.ingress}}
# depends, but default is nothing --> proxy
{{- . | toYaml | trimSuffix "\n" | nindent 4 }}
{{- end }}

Expand All @@ -61,15 +102,15 @@ spec:
{{- $_ := merge (dict "componentLabel" "singleuser-server") . }}
{{- include "jupyterhub.matchLabels" $_ | nindent 14 }}

# proxy -> Kubernetes internal DNS
# proxy --> Kubernetes internal DNS
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53

{{- with .Values.proxy.networkPolicy.egress }}
# proxy --> default: everything
{{- with .Values.proxy.chp.networkPolicy.egress }}
# proxy --> depends, but the default is everything
{{- . | toYaml | trimSuffix "\n" | nindent 4 }}
{{- end }}
{{- end }}
Loading