Compatibility with http2

[PanOS1]

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
[email protected] where the issue will be triaged appropriately.

Title: HTTP2 compatibility

Description:

Describe the issue.

We are trying to verify the HTTP2 compatibility of istio/envoy ingress gateways (Istio v1.22.3). We use the h2spec tool. And our findings are the following:

Hypertext Transfer Protocol Version 2 (HTTP/2)
  3. Starting HTTP/2
    3.5. HTTP/2 Connection Preface
      ✔ 1: Sends client connection preface
      ✔ 2: Sends invalid connection preface

  4. HTTP Frames
    4.1. Frame Format
      ✔ 1: Sends a frame with unknown type
      ✔ 2: Sends a frame with undefined flag
      ✔ 3: Sends a frame with reserved field bit

    4.2. Frame Size
      ✔ 1: Sends a DATA frame with 2^14 octets in length
      ✔ 2: Sends a large size DATA frame that exceeds the SETTINGS_MAX_FRAME_SIZE
      ✔ 3: Sends a large size HEADERS frame that exceeds the SETTINGS_MAX_FRAME_SIZE

    4.3. Header Compression and Decompression
      ✔ 1: Sends invalid header block fragment
      ✔ 2: Sends a PRIORITY frame while sending the header blocks
      ✔ 3: Sends a HEADERS frame to another stream while sending the header blocks

  5. Streams and Multiplexing
    5.1. Stream States
      ✔ 1: idle: Sends a DATA frame
      ✔ 2: idle: Sends a RST_STREAM frame
      ✔ 3: idle: Sends a WINDOW_UPDATE frame
      ✔ 4: idle: Sends a CONTINUATION frame
      ✔ 5: half closed (remote): Sends a DATA frame
      ✔ 6: half closed (remote): Sends a HEADERS frame
      ✔ 7: half closed (remote): Sends a CONTINUATION frame
      using source address 172.16.0.2:50107 sending RST_STREAM frame
      × 8: closed: Sends a DATA frame after sending RST_STREAM frame
        -> The endpoint MUST treat this as a stream error of type STREAM_CLOSED.
           Expected: GOAWAY Frame (Error Code: STREAM_CLOSED)
                     RST_STREAM Frame (Error Code: STREAM_CLOSED)
                     Connection closed
             Actual: Timeout
      using source address 172.16.0.2:50120ter sending RST_STREAM frame
      × 9: closed: Sends a HEADERS frame after sending RST_STREAM frame
        -> The endpoint MUST treat this as a stream error of type STREAM_CLOSED.
           Expected: GOAWAY Frame (Error Code: STREAM_CLOSED)
                     RST_STREAM Frame (Error Code: STREAM_CLOSED)
                     Connection closed
             Actual: WINDOW_UPDATE Frame (length:4, flags:0x00, stream_id:0)
      ✔ 10: closed: Sends a CONTINUATION frame after sending RST_STREAM frame
      using source address 172.16.0.2:50141
      × 11: closed: Sends a DATA frame
        -> The endpoint MUST treat this as a connection error of type STREAM_CLOSED.
           Expected: GOAWAY Frame (Error Code: STREAM_CLOSED)
                     RST_STREAM Frame (Error Code: STREAM_CLOSED)
                     Connection closed
             Actual: Timeout
      using source address 172.16.0.2:50155
      × 12: closed: Sends a HEADERS frame
        -> The endpoint MUST treat this as a connection error of type STREAM_CLOSED.
           Expected: GOAWAY Frame (Error Code: STREAM_CLOSED)
                     Connection closed
             Actual: Timeout
      ✔ 13: closed: Sends a CONTINUATION frame

      5.1.1. Stream Identifiers
        ✔ 1: Sends even-numbered stream identifier
        using source address 172.16.0.2:50189numerically smaller than previous
        × 2: Sends stream identifier that is numerically smaller than previous
          -> The endpoint MUST respond with a connection error of type PROTOCOL_ERROR.
             Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
                       Connection closed
               Actual: DATA Frame (length:0, flags:0x01, stream_id:5)

      5.1.2. Stream Concurrency
        using source address 172.16.0.2:50202 their advertised concurrent stream limit to be exceeded
        × 1: Sends HEADERS frames that causes their advertised concurrent stream limit to be exceeded
          -> The endpoint MUST treat this as a stream error of type PROTOCOL_ERROR or REFUSED_STREAM.
             Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
                       RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
                       GOAWAY Frame (Error Code: REFUSED_STREAM)
                       RST_STREAM Frame (Error Code: REFUSED_STREAM)
                       Connection closed
               Actual: Error: read tcp 172.16.0.2:50202->35.236.252.227:443: read: connection reset by peer

    5.3. Stream Priority
      5.3.1. Stream Dependencies
        ✔ 1: Sends HEADERS frame that depends on itself
        ✔ 2: Sends PRIORITY frame that depend on itself

    5.4. Error Handling
      5.4.1. Connection Error Handling
        ✔ 1: Sends an invalid PING frame for connection close

    5.5. Extending HTTP/2
      ✔ 1: Sends an unknown extension frame
      ✔ 2: Sends an unknown extension frame in the middle of a header block

  6. Frame Definitions
    6.1. DATA
      ✔ 1: Sends a DATA frame with 0x0 stream identifier
      ✔ 2: Sends a DATA frame on the stream that is not in "open" or "half-closed (local)" state
      ✔ 3: Sends a DATA frame with invalid pad length

    6.2. HEADERS
      ✔ 1: Sends a HEADERS frame without the END_HEADERS flag, and a PRIORITY frame
      ✔ 2: Sends a HEADERS frame to another stream while sending a HEADERS frame
      ✔ 3: Sends a HEADERS frame with 0x0 stream identifier
      ✔ 4: Sends a HEADERS frame with invalid pad length

    6.3. PRIORITY
      ✔ 1: Sends a PRIORITY frame with 0x0 stream identifier
      ✔ 2: Sends a PRIORITY frame with a length other than 5 octets

    6.4. RST_STREAM
      ✔ 1: Sends a RST_STREAM frame with 0x0 stream identifier
      ✔ 2: Sends a RST_STREAM frame on a idle stream
      ✔ 3: Sends a RST_STREAM frame with a length other than 4 octets

    6.5. SETTINGS
      ✔ 1: Sends a SETTINGS frame with ACK flag and payload
      ✔ 2: Sends a SETTINGS frame with a stream identifier other than 0x0
      ✔ 3: Sends a SETTINGS frame with a length other than a multiple of 6 octets

      6.5.2. Defined SETTINGS Parameters
        ✔ 1: SETTINGS_ENABLE_PUSH (0x2): Sends the value other than 0 or 1
        ✔ 2: SETTINGS_INITIAL_WINDOW_SIZE (0x4): Sends the value above the maximum flow control window size
        ✔ 3: SETTINGS_MAX_FRAME_SIZE (0x5): Sends the value below the initial value
        ✔ 4: SETTINGS_MAX_FRAME_SIZE (0x5): Sends the value above the maximum allowed frame size
        ✔ 5: Sends a SETTINGS frame with unknown identifier

      6.5.3. Settings Synchronization
        ✔ 1: Sends multiple values of SETTINGS_INITIAL_WINDOW_SIZE
        ✔ 2: Sends a SETTINGS frame without ACK flag

    6.7. PING
      ✔ 1: Sends a PING frame
      ✔ 2: Sends a PING frame with ACK
      ✔ 3: Sends a PING frame with a stream identifier field value other than 0x0
      ✔ 4: Sends a PING frame with a length field value other than 8

    6.8. GOAWAY
      ✔ 1: Sends a GOAWAY frame with a stream identifier other than 0x0

    6.9. WINDOW_UPDATE
      ✔ 1: Sends a WINDOW_UPDATE frame with a flow control window increment of 0
      ✔ 2: Sends a WINDOW_UPDATE frame with a flow control window increment of 0 on a stream
      ✔ 3: Sends a WINDOW_UPDATE frame with a length other than 4 octets

      6.9.1. The Flow-Control Window
        ✔ 1: Sends SETTINGS frame to set the initial window size to 1 and sends HEADERS frame
        using source address 172.16.0.2:51570mes increasing the flow control window to above 2^31-1
        × 2: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1
          -> The endpoint MUST sends a GOAWAY frame with a FLOW_CONTROL_ERROR code.
             Expected: GOAWAY Frame (Error Code: FLOW_CONTROL_ERROR)
               Actual: Connection closed
        using source address 172.16.0.2:51580mes increasing the flow control window to above 2^31-1 on a stream
        × 3: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream
          -> The endpoint MUST sends a RST_STREAM frame with a FLOW_CONTROL_ERROR code.
             Expected: RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR)
               Actual: Connection closed

      6.9.2. Initial Flow-Control Window Size
        ✔ 1: Changes SETTINGS_INITIAL_WINDOW_SIZE after sending HEADERS frame
        ✔ 2: Sends a SETTINGS frame for window size to be negative
        ✔ 3: Sends a SETTINGS_INITIAL_WINDOW_SIZE settings with an exceeded maximum window size value

    6.10. CONTINUATION
      ✔ 1: Sends multiple CONTINUATION frames preceded by a HEADERS frame
      ✔ 2: Sends a CONTINUATION frame followed by any frame other than CONTINUATION
      ✔ 3: Sends a CONTINUATION frame with 0x0 stream identifier
      ✔ 4: Sends a CONTINUATION frame preceded by a HEADERS frame with END_HEADERS flag
      ✔ 5: Sends a CONTINUATION frame preceded by a CONTINUATION frame with END_HEADERS flag
      ✔ 6: Sends a CONTINUATION frame preceded by a DATA frame

  7. Error Codes
    using source address 172.16.0.2:51661n error code
    × 1: Sends a GOAWAY frame with unknown error code
      -> The endpoint MUST NOT trigger any special behavior.
         Expected: Connection closed
                   PING Frame (length:8, flags:0x01, stream_id:0, opaque_data:h2spec)
           Actual: WINDOW_UPDATE Frame (length:4, flags:0x00, stream_id:0)
    ✔ 2: Sends a RST_STREAM frame with unknown error code

  8. HTTP Message Exchanges
    8.1. HTTP Request/Response Exchange
      ✔ 1: Sends a second HEADERS frame without the END_STREAM flag

      8.1.2. HTTP Header Fields
        ✔ 1: Sends a HEADERS frame that contains the header field name in uppercase letters

        8.1.2.1. Pseudo-Header Fields
          ✔ 1: Sends a HEADERS frame that contains a unknown pseudo-header field
          ✔ 2: Sends a HEADERS frame that contains the pseudo-header field defined for response
          ✔ 3: Sends a HEADERS frame that contains a pseudo-header field as trailers
          ✔ 4: Sends a HEADERS frame that contains a pseudo-header field that appears in a header block after a regular header field

        8.1.2.2. Connection-Specific Header Fields
          ✔ 1: Sends a HEADERS frame that contains the connection-specific header field
          ✔ 2: Sends a HEADERS frame that contains the TE header field with any value other than "trailers"

        8.1.2.3. Request Pseudo-Header Fields
          ✔ 1: Sends a HEADERS frame with empty ":path" pseudo-header field
          ✔ 2: Sends a HEADERS frame that omits ":method" pseudo-header field
          ✔ 3: Sends a HEADERS frame that omits ":scheme" pseudo-header field
          ✔ 4: Sends a HEADERS frame that omits ":path" pseudo-header field
          ✔ 5: Sends a HEADERS frame with duplicated ":method" pseudo-header field
          ✔ 6: Sends a HEADERS frame with duplicated ":scheme" pseudo-header field
          ✔ 7: Sends a HEADERS frame with duplicated ":path" pseudo-header field

        8.1.2.6. Malformed Requests and Responses
          ✔ 1: Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length
          ✔ 2: Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length

    8.2. Server Push
      ✔ 1: Sends a PUSH_PROMISE frame

Failures:

Hypertext Transfer Protocol Version 2 (HTTP/2)
  5. Streams and Multiplexing
    5.1. Stream States
      using source address 172.16.0.2:50107
      × 8: closed: Sends a DATA frame after sending RST_STREAM frame
        -> The endpoint MUST treat this as a stream error of type STREAM_CLOSED.
           Expected: GOAWAY Frame (Error Code: STREAM_CLOSED)
                     RST_STREAM Frame (Error Code: STREAM_CLOSED)
                     Connection closed
             Actual: Timeout
      using source address 172.16.0.2:50120
      × 9: closed: Sends a HEADERS frame after sending RST_STREAM frame
        -> The endpoint MUST treat this as a stream error of type STREAM_CLOSED.
           Expected: GOAWAY Frame (Error Code: STREAM_CLOSED)
                     RST_STREAM Frame (Error Code: STREAM_CLOSED)
                     Connection closed
             Actual: WINDOW_UPDATE Frame (length:4, flags:0x00, stream_id:0)
      using source address 172.16.0.2:50141
      × 11: closed: Sends a DATA frame
        -> The endpoint MUST treat this as a connection error of type STREAM_CLOSED.
           Expected: GOAWAY Frame (Error Code: STREAM_CLOSED)
                     RST_STREAM Frame (Error Code: STREAM_CLOSED)
                     Connection closed
             Actual: Timeout
      using source address 172.16.0.2:50155
      × 12: closed: Sends a HEADERS frame
        -> The endpoint MUST treat this as a connection error of type STREAM_CLOSED.
           Expected: GOAWAY Frame (Error Code: STREAM_CLOSED)
                     Connection closed
             Actual: Timeout

      5.1.1. Stream Identifiers
        using source address 172.16.0.2:50189
        × 2: Sends stream identifier that is numerically smaller than previous
          -> The endpoint MUST respond with a connection error of type PROTOCOL_ERROR.
             Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
                       Connection closed
               Actual: DATA Frame (length:0, flags:0x01, stream_id:5)

      5.1.2. Stream Concurrency
        using source address 172.16.0.2:50202
        × 1: Sends HEADERS frames that causes their advertised concurrent stream limit to be exceeded
          -> The endpoint MUST treat this as a stream error of type PROTOCOL_ERROR or REFUSED_STREAM.
             Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
                       RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
                       GOAWAY Frame (Error Code: REFUSED_STREAM)
                       RST_STREAM Frame (Error Code: REFUSED_STREAM)
                       Connection closed
               Actual: Error: read tcp 172.16.0.2:50202->35.236.252.227:443: read: connection reset by peer

  6. Frame Definitions
    6.9. WINDOW_UPDATE
      6.9.1. The Flow-Control Window
        using source address 172.16.0.2:51570
        × 2: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1
          -> The endpoint MUST sends a GOAWAY frame with a FLOW_CONTROL_ERROR code.
             Expected: GOAWAY Frame (Error Code: FLOW_CONTROL_ERROR)
               Actual: Connection closed
        using source address 172.16.0.2:51580
        × 3: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream
          -> The endpoint MUST sends a RST_STREAM frame with a FLOW_CONTROL_ERROR code.
             Expected: RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR)
               Actual: Connection closed

  7. Error Codes
    using source address 172.16.0.2:51661
    × 1: Sends a GOAWAY frame with unknown error code
      -> The endpoint MUST NOT trigger any special behavior.
         Expected: Connection closed
                   PING Frame (length:8, flags:0x01, stream_id:0, opaque_data:h2spec)
           Actual: WINDOW_UPDATE Frame (length:4, flags:0x00, stream_id:0)

Finished in 389.2907 seconds
94 tests, 85 passed, 0 skipped, 9 failed

Are those failures a known issue in compatibility of Envoy with HTTP2 protocol ?

The high level context of these report are the increase of sporadic 520 errors we have with Cloudflare after enabling the Optimization feature HTTP2-to-Origin

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[kyessenov]

Interesting! These look like edge cases with stricter validation, and the tool h2spec looks quite old.

CC @alyssawilk

[alyssawilk]

Did you try running this tool both with nghttp2 and oghttp2 (by flipping the runtime guard)?
I'm curious if they're roughly equivalent or if the migration ends up improving things or causing a regression
@birenroy @ianswett @diannahu

[PanOS1]

@alyssawilk i do not know how to do that. We use Istio v1.22.3, so please do you have an idea how to do it ?

The h2spec tool used after recommendation of the Cloudflare Docs . Since, after enabling the http2-to-origin optimization we observed an increase in 520 errors. With the logs from their side indicating
our origin resetting connection cause: connection reset with following context Upstream ReadError while reading h2 header.

[alyssawilk]

I don't know what Istio config looks like but there's a runtime guard
envoy.reloadable_features.http2_use_oghttp2
which changes the underlying http2 codec being used
There's hopefully examples of setting runtime config over in istio land but if not @kyessenov may be able to help?