[KNET-12362] Produce V3 API - Improve visibility around produce-record errors

[wcheng92]

Description

A single http requests to produce V3 API can contain many produce-records. A record receipt is returned for each record, each with its own error-code. For example see the produce example to produce a batch of records - Kafka REST APIs for Confluent Cloud | Confluent Documentation

{
"error_code": 401,
"cluster_id": lkc-vo0pz,
"topic_name": "testTopic1",
"partition_id": 0,
"offset": 67217,
"timestamp": "2021-12-13T16:29:10.951Z",
"key": {
"type": "BINARY",
"size": 10
},
"value": {
"type": "JSON",
"size": 26
}
}
{
"error_code": 503,
"cluster_id": lkc-vo0pz,
"topic_name": "testTopic1",
"partition_id": 0,
"offset": 67218,
"timestamp": "2021-12-13T16:29:10.951Z",
"key": {
"type": "BINARY",
"size": 10
},
"value": {
"type": "JSON",
"size": 26
}
}

Using the above example, REST log would like below with status 200(OK) -

10.1.1.158 - - [14/Nov/2023:19:20:40 +0000] "POST /kafka/v3/clusters/lkc-m8j07q/topics/apac.consumer.AddCalculatedAttributes.v1/records HTTP/1.1" 200 196 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36 Hutool" 503 -

Whereas clearly the user’s request had errors(sometimes 5xx). There is no visibility about these record-level error codes, which hinders oncall ability to answer customer’s question like “did the produce API request failed/succeeded for all records, or subset of records?”. This should be improved

Solution

Add a summarised counts of different error-codes in REST request log, Codes=401:1,503:1 as in example section below.

Example

Before - 10.1.1.158 - - [14/Nov/2023:19:20:40 +0000] "POST /kafka/v3/clusters/lkc-m8j07q/topics/apac.consumer.AddCalculatedAttributes.v1/records HTTP/1.1" 200 196 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36 Hutool" 503 -

Now - 10.1.1.158 - - [14/Nov/2023:19:20:40 +0000] "POST /kafka/v3/clusters/lkc-m8j07q/topics/apac.consumer.AddCalculatedAttributes.v1/records HTTP/1.1" 200 196 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36 Hutool" 503 - Codes=401:1,503:1

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[dimitarndimitrov]

It's great that Kafka REST continues to receive attention and improvements, especially for its newest significant API, the Produce v3!

I have one comment about the error counting structure.

[ehumber]

One last question from me - did you test this in our cloud environment @wcheng92 ?

[msn-tldr]

lgtm

@wcheng92

One last question from me - did you test this in our cloud environment @wcheng92 ?

Can you share the findings of ur ccloud testing? Merge once the Ccloud testing looks good.