Harold/mitm support

[harold-stripe]

This is the same PR than #225, had to re-open because the build process is broken when sending test coverage.

Description

This PR adds MITM support:

• New DSL for config and ACL which:
  • allows adding headers on the fly
  • detailed HTTP logging (URL, method and headers with values redacted)
  • allowlist of headers that should not be redacted
• YAML support for this DSL

I've also:

• Updated the stripe/goproxy dependency as this PR depends on it
• Created a Development.md which now includes instructions on how to run locally for multiple scenarios:
  • HTTP Proxy
  • HTTP CONNECT Proxy
  • Monitor metrics Smokescreen emits
  • HTTP CONNECT Proxy over TL
  • MITM (Man in the middle) Proxy
  • MITM (Man in the middle) Proxy over TLS
• Added .vscode/launch.json to allow debugging easily in Vscode
• Removed config *Config argument from logProxy as it was un-used
• Changes the way extractContextLogFields are added to accommodate the MITM code

pctx.RoundTripper which is normally used for http proxy request is now also used by the MITM outbound request. By default it pools requests and keep them idle for a short period of time to potentially re-use. (even when Response.Body.Close() is run) This doesn't work well with InstrumentedConn that logs CANONICAL-PROXY-CN-CLOSE once the connection is closed.

There are multiple ways to go around this but I chose to run proxy.Tr.CloseIdleConnections as this closes any connections which were previously connected from previous requests but are now sitting idle in a "keep-alive" state. This proxy is not primarily intended to support browser traffic and the performance gain of keeping this connection is negligible for our use-case.

Alternatives considered:

• req.Header.Set("Connection", "close") would work but the header is wiped in goproxy which calls removeProxyHeaders which deletes the Connection header.
• Create a custom dialer and close the connection manually but this added unnecessary complexity.

Testing

I have added automated tests for ACL and the whole MITM flow.

HTTP Proxy (happy path) ✅

See set-up from Development.md HTTP Proxy)

go run . --config-file config.yaml --egress-acl-file acl.yaml
{"level":"info","msg":"Loading egress ACL from acl.yaml","time":"2024-09-09T17:16:34.234136+02:00"}
{"level":"info","msg":"starting","time":"2024-09-09T17:16:34+02:00"}
{"allow":true,"conn_establish_time_ms":98,"content_length":-1,"decision_reason":"host matched allowed domain in rule","dns_lookup_time_ms":1,"enforce_would_deny":false,"id":"crfn7lut29fjr92l4g70","inbound_remote_addr":"[::1]:52470","level":"info","msg":"CANONICAL-PROXY-DECISION","outbound_local_addr":"[2a02:8428:5ae6:a601:854:83b0:cc2a:fc61]:52471","outbound_remote_addr":"[2606:2800:21f:cb07:6820:80da:af6b:8b2c]:80","proxy_type":"http","requested_host":"example.com","start_time":"2024-09-09T15:16:55.303862Z","time":"2024-09-09T17:16:55+02:00","trace_id":""}

HTTP CONNECT Proxy (happy path) ✅

See set-up from Development.md HTTP CONNECT Proxy)

go run . --config-file config.yaml --egress-acl-file acl.yaml
{"level":"info","msg":"Loading egress ACL from acl.yaml","time":"2024-09-09T17:42:55.710681+02:00"}
{"level":"info","msg":"starting","time":"2024-09-09T17:42:55+02:00"}
{"allow":true,"decision_reason":"host matched allowed domain in rule","dns_lookup_time_ms":6,"enforce_would_deny":false,"id":"crflrnut29fufb87e880","inbound_remote_addr":"[::1]:64026","level":"info","msg":"CANONICAL-PROXY-DECISION","project":"security","proxy_type":"connect","requested_host":"api.github.com:443","role":"","start_time":"2024-09-09T15:43:11.728056Z","time":"2024-09-09T17:43:11+02:00","trace_id":""}
{"bytes_in":4706,"bytes_out":594,"conn_establish_time_ms":92,"duration":0.468236125,"end_time":"2024-09-09T15:43:12.295369Z","error":"","id":"crflrnut29fufb87e880","inbound_remote_addr":"[::1]:64026","last_activity":"2024-09-09T15:43:12.294314Z","level":"info","msg":"CANONICAL-PROXY-CN-CLOSE","outbound_local_addr":"192.168.1.65:64027","outbound_remote_addr":"140.82.121.5:443","project":"security","proxy_type":"connect","requested_host":"api.github.com:443","role":"","start_time":"2024-09-09T15:43:11.728056Z","time":"2024-09-09T17:43:12+02:00","trace_id":""}

HTTP CONNECT Proxy over TLS (happy path) ✅

See set-up from Development.md HTTP CONNECT Proxy over TLS)

go run . --config-file config.yaml --egress-acl-file acl.yaml
warn: no statsd addr provided, using noop client
info: Loaded CA with Authority ID 'ffdb815e7ba5132cbd786c176175c6107d26809b'
warn: no CRL loaded for Authority ID 'ffdb815e7ba5132cbd786c176175c6107d26809b'
{"level":"info","msg":"Loading egress ACL from acl.yaml","time":"2024-09-09T17:54:56.824782+02:00"}
{"level":"info","msg":"starting","time":"2024-09-09T17:54:56+02:00"}
{"allow":true,"decision_reason":"host matched allowed domain in rule","dns_lookup_time_ms":593,"enforce_would_deny":false,"id":"crfm1d6t29fgv2nj56n0","inbound_remote_addr":"[::1]:52641","inbound_remote_x509_cn":"localhost","inbound_remote_x509_ou":"Writer","level":"info","msg":"CANONICAL-PROXY-DECISION","project":"github","proxy_type":"connect","requested_host":"api.github.com:443","role":"localhost","start_time":"2024-09-09T15:55:16.019972Z","time":"2024-09-09T17:55:16+02:00","trace_id":""}
{"bytes_in":4680,"bytes_out":594,"conn_establish_time_ms":219,"duration":0.893843,"end_time":"2024-09-09T15:55:17.727861Z","error":"","id":"crfm1d6t29fgv2nj56n0","inbound_remote_addr":"[::1]:52641","inbound_remote_x509_cn":"localhost","inbound_remote_x509_ou":"Writer","last_activity":"2024-09-09T15:55:17.726988Z","level":"info","msg":"CANONICAL-PROXY-CN-CLOSE","outbound_local_addr":"192.168.1.65:52648","outbound_remote_addr":"140.82.121.6:443","project":"github","proxy_type":"connect","requested_host":"api.github.com:443","role":"localhost","start_time":"2024-09-09T15:55:16.019972Z","time":"2024-09-09T17:55:17+02:00","trace_id":""}

MITM (Man in the middle) Proxy (happy path) ✅

See set-up from Development.md MITM (Man in the middle) Proxy)

go run . --config-file config.yaml --egress-acl-file acl.yaml
warn: no statsd addr provided, using noop client
{"level":"info","msg":"Loading egress ACL from acl.yaml","time":"2024-09-09T18:11:30.835473+02:00"}
{"level":"info","msg":"starting","time":"2024-09-09T18:11:30+02:00"}
{"allow":true,"decision_reason":"host matched allowed domain in rule","dns_lookup_time_ms":87,"enforce_would_deny":false,"id":"crfm93mt29fikttq21bg","inbound_remote_addr":"[::1]:59008","level":"info","msg":"CANONICAL-PROXY-DECISION","project":"security","proxy_type":"connect","requested_host":"wttr.in:443","role":"","start_time":"2024-09-09T16:11:42.641312Z","time":"2024-09-09T18:11:42+02:00","trace_id":""}
{"bytes_in":12109,"bytes_out":479,"conn_establish_time_ms":25,"duration":0.497213875,"end_time":"2024-09-09T16:11:43.461775Z","error":"","id":"crfm93mt29fikttq21bg","inbound_remote_addr":"[::1]:59008","last_activity":"2024-09-09T16:11:43.461714Z","level":"info","mitm_req_headers":{"Accept":["[REDACTED]"],"Accept-Language":["[REDACTED]"],"User-Agent":["curl/8.7.1"]},"mitm_req_method":"GET","mitm_req_url":"https://wttr.in:443/","msg":"CANONICAL-PROXY-CN-CLOSE","outbound_local_addr":"192.168.1.65:59009","outbound_remote_addr":"5.9.243.187:443","project":"security","proxy_type":"connect","requested_host":"wttr.in:443","role":"","start_time":"2024-09-09T16:11:42.641312Z","time":"2024-09-09T18:11:43+02:00","trace_id":""}

Accept-Language: el was correctly sent

Notice the mitm_req_headers, mitm_req_method and mitm_req_url fields

MITM (Man in the middle) Proxy over TLS (happy path) ✅

See set-up from Development.md MITM (Man in the middle) Proxy over TLS)
Accept-Language: el was correctly sent (weather is in Greek)

go run . --config-file config.yaml --egress-acl-file acl.yaml
warn: no statsd addr provided, using noop client
info: Loaded CA with Authority ID 'ffdb815e7ba5132cbd786c176175c6107d26809b'
warn: no CRL loaded for Authority ID 'ffdb815e7ba5132cbd786c176175c6107d26809b'
{"level":"info","msg":"Loading egress ACL from acl.yaml","time":"2024-09-09T18:17:01.764258+02:00"}
{"level":"info","msg":"starting","time":"2024-09-09T18:17:01+02:00"}
{"allow":true,"decision_reason":"host matched allowed domain in rule","dns_lookup_time_ms":15,"enforce_would_deny":false,"id":"crfmbket29fj5e1mvrag","inbound_remote_addr":"[::1]:61252","inbound_remote_x509_cn":"localhost","inbound_remote_x509_ou":"Writer","level":"info","msg":"CANONICAL-PROXY-DECISION","project":"github","proxy_type":"connect","requested_host":"wttr.in:443","role":"localhost","start_time":"2024-09-09T16:17:05.96784Z","time":"2024-09-09T18:17:05+02:00","trace_id":""}
{"bytes_in":12109,"bytes_out":479,"conn_establish_time_ms":36,"duration":0.065963916,"end_time":"2024-09-09T16:17:06.243684Z","error":"","id":"crfmbket29fj5e1mvrag","inbound_remote_addr":"[::1]:61252","inbound_remote_x509_cn":"localhost","inbound_remote_x509_ou":"Writer","last_activity":"2024-09-09T16:17:06.243663Z","level":"info","mitm_req_headers":{"Accept":["[REDACTED]"],"Accept-Language":["[REDACTED]"],"User-Agent":["curl/8.7.1"]},"mitm_req_method":"GET","mitm_req_url":"https://wttr.in:443/","msg":"CANONICAL-PROXY-CN-CLOSE","outbound_local_addr":"192.168.1.44:61255","outbound_remote_addr":"5.9.243.187:443","project":"github","proxy_type":"connect","requested_host":"wttr.in:443","role":"localhost","start_time":"2024-09-09T16:17:05.96784Z","time":"2024-09-09T18:17:06+02:00","trace_id":""}

Notice the mitm_req_headers, mitm_req_method, mitm_req_url (MITM), inbound_remote_x509_cn and inbound_remote_x509_ou (TLS) fields.

MITM config not configured with ACL configured ✅

# config.yaml
---
tls:
  cert_file: "mtls_setup/server.crt"
  key_file: "mtls_setup/server.key"
  client_ca_files:
    - "mtls_setup/client-ca.crt"

# acl.yaml
---
version: v1
services:
  - name: localhost
    project: github
    action: enforce
    allowed_domains: []
    allowed_domains_mitm:
      - domain: wttr.in
        add_headers:
          Accept-Language: el
        detailed_http_logs: true
        detailed_http_logs_full_headers:
          - User-Agent
default:
  name: default
  project: security
  action: enforce
  allowed_domains: []

go run . --config-file config.yaml --egress-acl-file acl.yaml

warn: no statsd addr provided, using noop client
info: Loaded CA with Authority ID 'ffdb815e7ba5132cbd786c176175c6107d26809b'
warn: no CRL loaded for Authority ID 'ffdb815e7ba5132cbd786c176175c6107d26809b'
{"level":"info","msg":"Loading egress ACL from acl.yaml","time":"2024-09-09T18:58:32.027806+02:00"}
{"level":"info","msg":"starting","time":"2024-09-09T18:58:32+02:00"}
{"allow":false,"content_length":119,"decision_reason":"ACLDecision specified MITM but Smokescreen doesn't have MITM enabled","dns_lookup_time_ms":18,"enforce_would_deny":false,"id":"crfmv56t29fhhhmhr5i0","inbound_remote_addr":"[::1]:61235","inbound_remote_x509_cn":"localhost","inbound_remote_x509_ou":"Writer","level":"warning","msg":"CANONICAL-PROXY-DECISION","project":"github","proxy_type":"connect","requested_host":"wttr.in:443","role":"localhost","start_time":"2024-09-09T16:58:44.319077Z","time":"2024-09-09T18:58:44+02:00","trace_id":""}

Miss-configuration fails gracefully