ipvs: protect proxy protocol from source address spoofing attack

Two versions -- v1-insecure and v2-insecure -- ared added for the proxy cascading case where the proxy protocol addresses should remain unchanged in the backend proxy server. Meanwhile, the v1 and v2 versions are always using the addresses from client's ip header of inbound packets. Signed-off-by: ywc689 <[email protected]>
iqiyi · Dec 8, 2023 · 486ed1e · 486ed1e
1 parent 109a491
commit 486ed1e
Show file tree

Hide file tree

Showing 11 changed files with 148 additions and 39 deletions.
diff --git a/doc/client-address-conservation-in-fullnat.md b/doc/client-address-conservation-in-fullnat.md
@@ -1,19 +1,19 @@
 Client Address Conservation in Fullnat
 ---
 
-The original client addresses are substituted by DPVS's local addresses in Fullnat forwarding mode so that auxiliary means is required to pass them to realservers. Three solutions are provided in DPVS to solve the problem -- toa, uoa, and proxy protocol, with each having its own pros and cons. The doc will elaborate on them.
+The original client addresses are substituted with DPVS's local addresses in Fullnat forwarding mode so that auxiliary means is required to pass them to realservers. Three solutions have been developed in DPVS for the problem -- *TOA*, *UOA*, and *Proxy Protocol*, with each having its own pros and cons. The document is to elaborate on them.
 
 * **TOA**
 
 Client address is encapsulated in a private TCP option (opcode 254) by DPVS, and parsed into the connected TCP socket on realserver by a kernel module named [toa.ko](../kmod/toa/). By default, it requires no changes in realserver application programs for fnat44, fnat66. But an extra syscall to `getsockopt` with parameters `IPPROTO_IP` and `TOA_SO_GET_LOOKUP` is required for fnat64 to retrieve the original IPv6 client address from toa.
 
 * **UOA**
 
-UOA is the counterpart in UDP protocol. It supports two mode: IP option mode (ipo) and Private Protocol mode (opp). Client address is encapsulated into a private IPv4 option (opcode 31) in ipo mode, and into a private layer4 protocol named "option protocol" (protocol number 248) in opp mode respectively. Similarly, a kernel module name [uoa.ko](../kmod/uoa/) is required to parse the original client address from raw packets on realserver. Realserver application programs should use `getsockopt` with parameters `IPPROTO_IP` and `UOA_SO_GET_LOOKUP` immediately after user data reception to retrieve the original address from uoa. Note that not all kinds of network switches or routers support private IPv4 options or layer4 private protocols. Be aware of your network restrictions before using UOA.
+UOA is the counterpart in UDP protocol. It supports two modes: *IP Option Mode* (ipo) and *Private Protocol Mode* (opp). Client address is encapsulated into a private IPv4 option (opcode 31) in ipo mode, and into a private layer4 protocol named "option protocol" (protocol number 248) in opp mode respectively. Similarly, a kernel module name [uoa.ko](../kmod/uoa/) is required to parse the original client address from raw packets on realserver. Realserver application programs should use `getsockopt` with parameters `IPPROTO_IP` and `UOA_SO_GET_LOOKUP` immediately after user data reception to retrieve the original address from uoa. Note that not all kinds of network switches or routers support private IPv4 options or layer4 private protocols. Be aware of your network restrictions before using UOA.
 
 * **Proxy Protocol**:
 
-[Proxy Protocol](https://www.haproxy.org/download/2.9/doc/proxy-protocol.txt) is a widely-used protocol for client address conservation on reverse proxy. It's been drafted by haproxy.org and supported two versions up to now. The version 1 is a human-readable format and supports TCP only, while version 2 is a binary format supporting both TCP and UDP. DPVS implements both versions and users can choose which one to use on basis of a per-service configuration. Moreover, DPVS allows for clients that have already carried proxy protocol data, which is often the case when DPVS's virtual IP is used as the realserver for some other reverse proxy such as nginx, envoy, or another DPVS, where DPVS doesn't insert client address by itself, but just retains the client address encapsulated in the packet, and makes protocol version translation if necessary. Proxy protocol has advantages of broad and uniform supports for layer3 and layer4 protocols(including IP, IPv6, TCP, UDP), both source and destination address conveying, no dependency on kernel modules, tolerances of network infrastructure differences. The client addresses are encapsulated into the layer4 payload in the begginning position. Application programs on realservers must receive the data and parse it to obtain the original client addresses immediately on establishment of TCP/UDP connection. The client address data may be taken as application data by mistake if not processed, resulting in unexpected behavior in your application. Fortunately, parsing the client address from proxy protocol in application server is quite straightforward, and a variety of well-known proxy servers have supported it. Actually, proxy protocol is becoming a defato standard in this area.
+[Proxy Protocol](https://www.haproxy.org/download/2.9/doc/proxy-protocol.txt) is a widely-used protocol for client address conservation on reverse proxies. It's been drafted by haproxy.org and supported two versions up to now. The version v1 is a human-readable format which supports TCP only, while version v2 is a binary format supporting both TCP and UDP. DPVS implements both versions and users can choose which one to use on basis of a per-service configuration. Moreover, if configured to the insecure mode, DPVS allows for clients that have already carried proxy protocol data, which is often the case when DPVS's virtual IP is behind of other reverse proxies such as nginx, envoy, or another DPVS, where DPVS doesn't insert client address by itself, but just retains the client address encapsulated in the packet, and makes protocol version translation if necessary. Proxy protocol has advantages of broad and uniform supports for layer3 and layer4 protocols(including IP, IPv6, TCP, UDP), both source and destination addresses conveying, no dependency on kernel modules, tolerance of network infrastructure differences. The client addresses are encapsulated into the very begginning position of layer4 payload. Application programs on realservers must receive the data and parse it to obtain the original client addresses immediately on establishment of TCP/UDP connection. Otherwise, the client address data may be taken as application data by mistake, resulting in unexpected behavior in the application program. Fortunately, parsing the client address from proxy protocol is quite straightforward, and a variety of well-known proxy servers have supported it. Actually, proxy protocol is becoming a defato standard in this area.
 
 Next ,let's compare the three client address conservation solutions in detail in the following two tables.
 

diff --git a/include/conf/service.h b/include/conf/service.h
@@ -62,11 +62,24 @@
 #define DEST_INHIBIT_DURATION_MIN       5       // 5s
 #define DEST_INHIBIT_DURATION_MAX       3600    // 1h
 
+#define PROXY_PROTOCOL_VERSION_MASK     0x0F
+#define PROXY_PROTOCOL_FLAGS_MASK       0xF0
+
+#define PROXY_PROTOCOL_VERSION(verflag)     ((verflag) & PROXY_PROTOCOL_VERSION_MASK)
+#define PROXY_PROTOCOL_FLAGS(verflag)       ((verflag) & PROXY_PROTOCOL_FLAGS_MASK)
+#define PROXY_PROTOCOL_IS_INSECURE(verflag) (!!((verflag) & PROXY_PROTOCOL_F_INSECURE))
+
 enum {
-    PROXY_PROTOCOL_DISABLE = 0,
-    PROXY_PROTOCOL_V1,
-    PROXY_PROTOCOL_V2,
-    PROXY_PROTOCOL_MAX,
+    PROXY_PROTOCOL_DISABLE      = 0x00,
+    PROXY_PROTOCOL_V1           = 0x01,
+    PROXY_PROTOCOL_V2           = 0x02,
+    PROXY_PROTOCOL_MAX          = PROXY_PROTOCOL_VERSION_MASK,
+
+    /* The proxy protocol addresses existing in the received mbuf are passed to backends
+     * in insecure mode, making the service subject to Source Address Spoofing Attack,
+     * but it's useful when multiple proxies exist before the backend. */
+    PROXY_PROTOCOL_F_INSECURE   = 0x10,
+    PROXY_PROTOCOL_F_MAX        = PROXY_PROTOCOL_FLAGS_MASK,
 };
 
 struct dest_check_configs {
@@ -183,17 +196,21 @@ static inline uint8_t proxy_protocol_type(const char *str) {
         return PROXY_PROTOCOL_V1;
     if (!strcasecmp(str, "v2"))
         return PROXY_PROTOCOL_V2;
+    if (!strcasecmp(str, "v1-insecure"))
+        return PROXY_PROTOCOL_V1 | PROXY_PROTOCOL_F_INSECURE;
+    if (!strcasecmp(str, "v2-insecure"))
+        return PROXY_PROTOCOL_V2 | PROXY_PROTOCOL_F_INSECURE;
     return PROXY_PROTOCOL_DISABLE;
 }
 
 static inline const char *proxy_protocol_str(uint8_t type) {
-    switch (type) {
+    switch (PROXY_PROTOCOL_VERSION(type)) {
         case PROXY_PROTOCOL_DISABLE:
             return "disable";
         case PROXY_PROTOCOL_V1:
-            return "v1";
+            return PROXY_PROTOCOL_IS_INSECURE(type) ? "v1-insecure" : "v1";
         case PROXY_PROTOCOL_V2:
-            return "v2";
+            return PROXY_PROTOCOL_IS_INSECURE(type) ? "v2-insecure" : "v2";
     }
     return "unknown";
 }

diff --git a/src/ipvs/ip_vs_proto_tcp.c b/src/ipvs/ip_vs_proto_tcp.c
@@ -371,13 +371,15 @@ static int tcp_in_add_proxy_proto(struct dp_vs_conn *conn, struct rte_mbuf *mbuf
     if (unlikely(EDPVS_OK != proxy_proto_parse(mbuf, offset, &ppinfo)))
         return EDPVS_INVPKT;
 
-    if (ppinfo.datalen > 0 && ppinfo.version == conn->pp_version)
+    if (ppinfo.datalen > 0
+            && ppinfo.version == PROXY_PROTOCOL_VERSION(conn->pp_version)
+            && PROXY_PROTOCOL_IS_INSECURE(conn->pp_version))
         return EDPVS_OK;    // keep intact the orginal proxy protocol data
 
-    if (!ppinfo.datalen) {
+    if (!ppinfo.datalen || !PROXY_PROTOCOL_IS_INSECURE(conn->pp_version)) {
         ppinfo.af = tuplehash_in(conn).af;
         ppinfo.proto = IPPROTO_TCP;
-        ppinfo.version = conn->pp_version;
+        ppinfo.version = PROXY_PROTOCOL_VERSION(conn->pp_version);
         ppinfo.cmd = 1;
         if (AF_INET == ppinfo.af) {
             ppinfo.addr.ip4.src_addr = conn->caddr.in.s_addr;
@@ -828,8 +830,8 @@ static int tcp_fnat_in_handler(struct dp_vs_proto *proto,
         tcp_in_remove_ts(th);
 
         tcp_in_init_seq(conn, mbuf, th);
-        if (PROXY_PROTOCOL_V1 != conn->pp_version &&
-                PROXY_PROTOCOL_V2 != conn->pp_version) {
+        if (PROXY_PROTOCOL_V1 != PROXY_PROTOCOL_VERSION(conn->pp_version)
+                && PROXY_PROTOCOL_V2 != PROXY_PROTOCOL_VERSION(conn->pp_version)) {
             if (unlikely(tcp_in_add_toa(conn, mbuf, th) != EDPVS_OK)) {
                 tcp_in_remove_toa(th, iaf);
             }
@@ -839,8 +841,8 @@ static int tcp_fnat_in_handler(struct dp_vs_proto *proto,
     /* add toa/proxy_proto to first data packet */
     if (ntohl(th->ack_seq) == conn->fnat_seq.fdata_seq
             && !th->syn && !th->rst /*&& !th->fin*/) {
-        if (PROXY_PROTOCOL_V2 == conn->pp_version ||
-                PROXY_PROTOCOL_V1 == conn->pp_version) {
+        if (PROXY_PROTOCOL_V2 == PROXY_PROTOCOL_VERSION(conn->pp_version)
+                || PROXY_PROTOCOL_V1 == PROXY_PROTOCOL_VERSION(conn->pp_version)) {
             if (conn->fnat_seq.isn - conn->fnat_seq.delta + 1 == ntohl(th->seq)) {
                 /* avoid inserting repetitive ppdata when the first rs ack delayed */
                 err = tcp_in_add_proxy_proto(conn, mbuf, th, iphdrlen, &pp_hdr_shift);

diff --git a/src/ipvs/ip_vs_proto_udp.c b/src/ipvs/ip_vs_proto_udp.c
@@ -720,13 +720,15 @@ static int udp_in_add_proxy_proto(struct dp_vs_conn *conn,
     if (unlikely(EDPVS_OK != proxy_proto_parse(mbuf, offset, &ppinfo)))
         return EDPVS_INVPKT;
 
-    if (ppinfo.datalen > 0 && ppinfo.version == conn->pp_version)
+    if (ppinfo.datalen > 0
+            && ppinfo.version == PROXY_PROTOCOL_VERSION(conn->pp_version)
+            && PROXY_PROTOCOL_IS_INSECURE(conn->pp_version))
         return EDPVS_OK;    // keep intact the original proxy protocol data
 
-    if (!ppinfo.datalen) {
+    if (!ppinfo.datalen || !PROXY_PROTOCOL_IS_INSECURE(conn->pp_version)) {
         ppinfo.af = tuplehash_in(conn).af;
         ppinfo.proto = IPPROTO_UDP;
-        ppinfo.version = conn->pp_version;
+        ppinfo.version = PROXY_PROTOCOL_VERSION(conn->pp_version);
         ppinfo.cmd = 1;
         if (AF_INET == ppinfo.af) {
             ppinfo.addr.ip4.src_addr = conn->caddr.in.s_addr;
@@ -786,8 +788,8 @@ static int udp_fnat_in_handler(struct dp_vs_proto *proto,
     if (unlikely(!uh))
         return EDPVS_INVPKT;
 
-    if (!conn->pp_sent && (PROXY_PROTOCOL_V2 == conn->pp_version ||
-                PROXY_PROTOCOL_V2 == conn->pp_version)) {
+    if (!conn->pp_sent &&
+            (PROXY_PROTOCOL_V2 == PROXY_PROTOCOL_VERSION(conn->pp_version))) {
         err = udp_in_add_proxy_proto(conn, mbuf, uh, iphdrlen, &hdr_shift);
         if (unlikely(EDPVS_OK != err))
             RTE_LOG(INFO, IPVS, "%s: insert proxy protocol fail -- %s\n",
@@ -832,8 +834,7 @@ static int udp_fnat_in_pre_handler(struct dp_vs_proto *proto,
 {
     struct conn_uoa *uoa = (struct conn_uoa *)conn->prot_data;
 
-    if (PROXY_PROTOCOL_V2 == conn->pp_version ||
-            PROXY_PROTOCOL_V1 == conn->pp_version)
+    if (PROXY_PROTOCOL_V2 == PROXY_PROTOCOL_VERSION(conn->pp_version))
         return EDPVS_OK;
 
     if (uoa && g_uoa_max_trail > 0)

diff --git a/src/ipvs/ip_vs_proxy_proto.c b/src/ipvs/ip_vs_proxy_proto.c
@@ -312,7 +312,7 @@ static int proxy_proto_send_standalone(struct proxy_info *ppinfo,
         err = EDPVS_NOMEM;
         goto errout;
     }
-    if (PROXY_PROTOCOL_V2 == conn->pp_version) {
+    if (PROXY_PROTOCOL_V2 == PROXY_PROTOCOL_VERSION(conn->pp_version)) {
         pphv2 = (struct proxy_hdr_v2 *)pph;
         rte_memcpy(pphv2->sig, PROXY_PROTO_V2_SIGNATURE, sizeof(pphv2->sig));
         pphv2->cmd     = 1;
@@ -321,7 +321,7 @@ static int proxy_proto_send_standalone(struct proxy_info *ppinfo,
         pphv2->af      = ppv2_af_host2pp(ppinfo->af);
         pphv2->addrlen = ntohs(ppdlen - sizeof(struct proxy_hdr_v2));
         rte_memcpy(pphv2 + 1, &ppinfo->addr, ppdlen - sizeof(struct proxy_hdr_v2));
-    } else if (PROXY_PROTOCOL_V1 == conn->pp_version) {
+    } else if (PROXY_PROTOCOL_V1 == PROXY_PROTOCOL_VERSION(conn->pp_version)) {
         rte_memcpy(pph, ppv1data, ppdlen);
     } else {
         err = EDPVS_NOTSUPP;
@@ -434,7 +434,8 @@ int proxy_proto_insert(struct proxy_info *ppinfo, struct dp_vs_conn *conn,
     if (unlikely(conn->dest->fwdmode != DPVS_FWD_MODE_FNAT))
         return EDPVS_NOTSUPP;
 
-    if (ppinfo->datalen > 0 && ppinfo->version == conn->pp_version)
+    if (ppinfo->datalen > 0 && PROXY_PROTOCOL_IS_INSECURE(conn->pp_version)
+            && ppinfo->version == PROXY_PROTOCOL_VERSION(conn->pp_version))
         return EDPVS_OK; // proxy the existing proxy protocol data directly to rs
 
     oaf = tuplehash_out(conn).af;
@@ -453,7 +454,7 @@ int proxy_proto_insert(struct proxy_info *ppinfo, struct dp_vs_conn *conn,
 
     // calculate required space size in mbuf
     ppdatalen = 0;
-    if (PROXY_PROTOCOL_V2 == conn->pp_version) {
+    if (PROXY_PROTOCOL_V2 == PROXY_PROTOCOL_VERSION(conn->pp_version)) {
         ppdatalen = sizeof(struct proxy_hdr_v2);
         if (ppinfo->cmd == 1) {
             switch (ppinfo->af) {
@@ -471,7 +472,7 @@ int proxy_proto_insert(struct proxy_info *ppinfo, struct dp_vs_conn *conn,
                 return EDPVS_NOTSUPP;
             }
         }
-    } else if (PROXY_PROTOCOL_V1 == conn->pp_version) {
+    } else if (PROXY_PROTOCOL_V1 == PROXY_PROTOCOL_VERSION(conn->pp_version)) {
         if (ppinfo->cmd == 1) {
             if (IPPROTO_TCP != ppinfo->proto)
                 return EDPVS_NOTSUPP; // v1 only supports tcp
@@ -554,7 +555,7 @@ int proxy_proto_insert(struct proxy_info *ppinfo, struct dp_vs_conn *conn,
     }
 
     // fill in proxy protocol data
-    if (PROXY_PROTOCOL_V2 == conn->pp_version) {
+    if (PROXY_PROTOCOL_V2 == PROXY_PROTOCOL_VERSION(conn->pp_version)) {
         pphv2 = (struct proxy_hdr_v2 *)pph;
         rte_memcpy(pphv2->sig, PROXY_PROTO_V2_SIGNATURE, sizeof(pphv2->sig));
         pphv2->cmd = 1;

diff --git a/tools/dpvs-agent/dpvs-agent-api.yaml b/tools/dpvs-agent/dpvs-agent-api.yaml
@@ -490,8 +490,20 @@ definitions:
         type: "integer"
         format: "uint8"
       ProxyProto:
+        description: |
+          0  (0x00): disable
+          1  (0x01): v1
+          2  (0x02): v2
+          17 (0x11): v1-insecure
+          18 (0x12): v2-insecure
         type: "integer"
         format: "uint8"
+        enum:
+          - 0
+          - 1
+          - 2
+          - 17
+          - 18
       Port:
         type: "integer"
         format: "uint16"
@@ -586,7 +598,9 @@ definitions:
         type: "string"
         enum:
           - v2
+          - v2-insecure
           - v1
+          - v1-insecure
           - disable
       SchedName:
         type: "string"

diff --git a/tools/dpvs-agent/models/virtual_server_spec_expand.go b/tools/dpvs-agent/models/virtual_server_spec_expand.go
diff --git a/tools/dpvs-agent/models/virtual_server_spec_tiny.go b/tools/dpvs-agent/models/virtual_server_spec_tiny.go
diff --git a/tools/dpvs-agent/pkg/ipc/types/virtualserver.go b/tools/dpvs-agent/pkg/ipc/types/virtualserver.go
@@ -244,11 +244,16 @@ func (vs *VirtualServerSpec) SetProto(proto uint8) {
 }
 
 func (vs *VirtualServerSpec) SetProxyProto(version string) {
-	if version == models.VirtualServerSpecTinyProxyProtocolV2 {
+	switch version {
+	case models.VirtualServerSpecTinyProxyProtocolV2:
 		vs.proxyProto = 2
-	} else if version == models.VirtualServerSpecTinyProxyProtocolV1 {
+	case models.VirtualServerSpecTinyProxyProtocolV1:
 		vs.proxyProto = 1
-	} else {
+	case models.VirtualServerSpecTinyProxyProtocolV2DashInsecure:
+		vs.proxyProto = 18
+	case models.VirtualServerSpecTinyProxyProtocolV1DashInsecure:
+		vs.proxyProto = 17
+	default:
 		vs.proxyProto = 0
 	}
 }