Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to handle dispatch_http_call/other timeout #622

Closed
danielphan2003 opened this issue Nov 11, 2024 · 6 comments · Fixed by #625
Closed

How to handle dispatch_http_call/other timeout #622

danielphan2003 opened this issue Nov 11, 2024 · 6 comments · Fixed by #625

Comments

@danielphan2003
Copy link

Even though you can specify timeout in dispatch_http_call Rust SDK, it doesn't seem to return any Status and the function stops instead. Same case for invalid upstream.

@thibaultcha
Copy link
Member

thibaultcha commented Nov 11, 2024

Indeed presently the on_http_call_response does not get invoked on a timeout or bad upstream (you only get an error log). So I tried the case with Envoy and the handler is invoked with on_http_call_response(token_id, 0, 0, 0) which seems correct and would allow you to catch failures, so we will make this change.

However I need to better understand the behavior on both failure conditions (timeout + bad upstream) but I currently don't. When in the call response handler on a timeout, get_http_call_response_header(":status") returns 504 which seems appropriate. Unfortunately, my C++ skills do not let me decipher the maze that is the combination of the 3 codebases proxy-wasm-cpp-host + proxy-wasm-cpp-sdk + envoy.

@hishamhm @casimiro Any help from either of you? I would like to understand how Envoy gets from status = BrokenConnection to returning a 504 status code on get_http_call_response_header(":status") (so that we can see what other kinds of failures return which status code). Amongst other kinds of failures, it would be nice to know the behavior and status code produced on bad upstream specifically, but I have not been able to either follow the code or reproduce such a failure with an envoy config...

@casimiro
Copy link
Contributor

I'm not properly familiarized with Envoy's codebase, but it seems to me that the headers of the response are defined before Context::onHttpCallFailure is called.

I did some investigation on AsyncClientImpl, and some other classes in the Envoy::Http namespace, but could not pinpoint exactly where/when :status header is ever written.

Time is quite scarce this week so I couldn't invest much in this issue.

@hishamhm
Copy link
Contributor

hishamhm commented Nov 13, 2024

Ah, I did look at this yesterday but I ended up not sending the message.

how Envoy gets from status = BrokenConnection to returning a 504 status code on get_http_call_response_header(":status")

From what I could gather, I don't think it does. From looking at the code, the BrokenConnection status (which translates to 11 in its enum), gets saved to the context status_code_ and that (along with status_message_) is what is returned when the client calls proxy_get_status. (See proxy-wasm-cpp-host proxy_get_status proxy-wasm-cpp-sdk getStatus. So, I haven't tested it, but my guess is that on a callout failure, calling proxy_get_status would return 11, "reset".

I'm not properly familiarized with Envoy's codebase, but it seems to me that the headers of the response are defined before Context::onHttpCallFailure is called.

I got the same impression as @casimiro. The :status header itself comes from the response object. The router filter onTimeout sets kDeadlineExceeded, which then translates to 504. The timeout goes into AsyncStreamImpl::sendLocalReply.

The weird thing is that since Context::onHttpCallSuccess seems to be the only path where http_call_response_ is set, it seems that for being able to get a 504 value for get_http_call_response_header(":status"), it would have to go through through onHttpCallSuccess, rather than onHttpCallFailure.

@thibaultcha to clarify, did you get on_http_call_response(token_id, 0, 0, 0) callback and get_http_call_response_header(":status") == "504" within that same callback?

@thibaultcha
Copy link
Member

@hishamhm @casimiro Thanks!

@thibaultcha to clarify, did you get on_http_call_response(token_id, 0, 0, 0) callback and get_http_call_response_header(":status") == "504" within that same callback?

On running the case again this morning, it turns out this isn't the case, I'm not sure how I ended up in the 0,0,0 code path... So with the hostcalls filter on dispatch timeout we get:

[2024-11-13 08:11:48.357][27958][info][wasm] [source/extensions/common/wasm/context.cc:1195] wasm log my_wasm_filter: [hostcalls] testing in "RequestHeaders"
[2024-11-13 08:11:50.359][27958][info][wasm] [source/extensions/common/wasm/context.cc:1195] wasm log my_wasm_filter: [hostcalls] on_http_call_response (id: 4, status: 504, headers: 3, body_bytes: 24, trailers: 0, op: )

So the timeout (after 2s) does indeed enter the Context::onHttpCallSuccess path. That said it seems Envoy does things I wouldn't expect and we'll have to adapt in our codebase: it produces a default response that it gives to on_http_call_response: the 504 + 3 headers + a plain text body that says upstream request timeout (24 bytes).

This helps a lot, thanks.

That said there remains a few unknowns for me:

  • What must happen for the Context::onHttpCallFailure to be called and invoke the on_http_call_response(token_id, 0, 0, 0) callback?
  • What other kind of response status/body does the "bad upstream" case produces (as it invokes Context::onHttpCallSuccess)?

@hishamhm
Copy link
Contributor

What must happen for the Context::onHttpCallFailure to be called and invoke the on_http_call_response(token_id, 0, 0, 0) callback?

I assume it's something like a failure to connect (e.g. trying to reach an upstream that doesn't resolve).

What other kind of response status/body does the "bad upstream" case produces (as it invokes Context::onHttpCallSuccess)?

There are other code paths there which trigger completeAndSendLocalReply with kUnavailable, which translates to 503. They seem to be Envoy-specific as they carry status messages such as "cluster_maintain_mode" and "no_healthy_upstream".

thibaultcha added a commit that referenced this issue Nov 15, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 15, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 15, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 15, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 16, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 19, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 19, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 19, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 19, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 19, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 20, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 20, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 20, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 20, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 22, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 22, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 22, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 23, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 25, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
thibaultcha added a commit that referenced this issue Nov 25, 2024
Invoke `on_http_call_response` on dispatch connection failures. This
allows catching failures such as:

- timeout
- broken connection
- resolver failures
- TLS handshake failures
- Any other failures *after the connection has attempted to be
  established*

Dispatch failures occurring *before* a connection has attempted to be
established will not trigger `on_http_call_response` as they are already
supposed to trigger a Wasm exception (i.e. trap).

When a dispatch connection failure occurs, `on_http_call_response` is
invoked with: `on_http_call_response(call_id, 0, 0, 0)` (i.e. no header,
no body, no trailers). Calls to retrieve the response `:status` will
return `None`. A user may triage the connection failure by querying the
`:dispatch_status` pseudo-header:

```rust
fn on_http_call_response(
    &mut self,
    token_id: u32,
    nheaders: usize,
    body_size: usize,
    ntrailers: usize,
) {
    let dispatch_status = self.get_http_call_response_header(":dispatch_status");

    match dispatch_status.as_deref() {
        Some("timeout") => {},
        Some("broken connection") => {},
        Some("tls handshake failure") => {},
        Some("resolver failure") => {},
        Some("reader failure") => {},
        Some(s) => info!("dispatch_status: {}", s),
        None => {}
    }

    self.resume_http_request()
}
```

The socket status `"bad argument"` also exists but since the connection
has not been attempted yet, it won't show up during
`on_http_call_response` and is for internal use only (i.e. could be
added to the socket error log for example).

Fix #622.
@thibaultcha
Copy link
Member

@danielphan2003 We now have #625 merged which will allow filter authors to catch such failures. It should make it in the 3.9 Kong release (probably 3.9.1 which is I think 5-6 weeks from now). Or if you use the module standalone, then it's already available. We also have a guide on HTTP dispatch calls with the module: https://github.com/Kong/ngx_wasm_module/blob/main/docs/PROXY_WASM.md#http-dispatches

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants