Overriding response body and compression handling

[danielphan2003]

I'm currently overriding response body from origin by first preventing origin from compressing the body via Accept-Encoding (set to an empty string) and process each body chunk like so:

fn on_http_response_body(&mut self, body_size: usize, eof: bool) -> Action {
    warn!("#{} on_http_response_body eof={}", self.request_id, eof);

    self.phase = Phase::ResponseBody;

    let content_type = self
        .get_http_response_header(CONTENT_TYPE.as_str())
        .unwrap_or_default();

    if !content_type.contains("text/html") {
        return Action::Continue;
    }

    if self.is_first_response_body_chunk && !eof {
        self.is_first_response_body_chunk = false;
        return Action::Pause;
    }

    if !eof {
        self.body = self.get_http_response_body(0, body_size);
        self.body_size = body_size;
        if let Some(new_body) = intercept_response_html(self) {
            self.body = Some(new_body.as_bytes().to_vec());
        }

        if let Some(body) = self.body.as_deref() {
            self.set_http_response_body(0, usize::MAX, body);
        }
    } else {
        self.is_first_response_body_chunk = true;
    }

    Action::Continue
}

While this does work for simple html manipulation (i.e. no error is thrown throughout the body process), trying to perform compression would result in a truncated body.

Code snippet using `flate2` crate

if let Some(body) = self.body.as_deref() {
    let mut result = Vec::new();
    let mut z = GzEncoder::new(body, flate2::Compression::default());
    match z.read_to_end(&mut result) {
        Ok(_) => {
            self.set_http_response_body(0, usize::MAX, result.as_slice());
        }
        Err(e) => {
            warn!(
                "#{} on_http_response_body: failed to compress response body: {}",
                self.request_id, e
            );
        }
    }
}

[thibaultcha]

It is unclear to me what the problem is. Care to elaborate? Is the response actually truncated? Do you have a minimally reproducible example? By the way, it seems you are misusing response body buffering (or maybe not misusing, but it looks incomplete?), have you looked at the docs on response body buffering? All you should have to do is return Action::Pause once, and only once, and the next invocation should be the full body. However there is still a possible chunk-by-chunk case if the response body is larger than the configured wasm_response_body_buffers, so you should either handle a possible chunk-by-chunk case in your code (this is unavoidable), or make sure that your upstream never returns bodies larger than the configured buffer size, in which case it is avoidable (but still the user's responsibility to make sure all responses fit in this buffer).