Feat/5.0 changes #1008
Feat/5.0 changes #1008
Conversation
…ding to an exception raised since the Connect struct for the envelope does not have the "value" field on null. The code change ensures the transformer does not populate the struct unless the filed is present.
…class extending SinkData is introduced. For text storage the safeValue function is overwritten to return the java BigDecimal. The JsonConverter is changed to store decimals as numbers, not base64 (default). While a breaking change, this should not be a cause of concern since BigDecimal was not handled before, so it should have created an error if that was the case. Unit test and integration tests were added to ensure the storage can handle BigDecimal.
| .put("title", "mr") | ||
| .put( | ||
| "salary", | ||
| BigDecimal(100.43).setScale(18).bigDecimal, |
There was a problem hiding this comment.
It's advisable to get into the habit of constructing using BigDecimal(String) instead of relying on floating-point literals. The conversion of floating-point literals, such as 100.43, to BigDecimal using setScale(18) may introduce rounding errors due to the binary representation of these literals. To ensure accurate representation and avoid precision loss, consider using BigDecimal("100.43").setScale(18) instead.
Additionally, it's worth evaluating whether a scale of 18 is necessary for the specific use case, as an excessively high scale might impact performance and memory usage without providing significant benefits.
(This comment applies to various places in the PR.)
There was a problem hiding this comment.
Thanks for raising that. The 18 scale comes from the schema definition. They should align or Avro serialisation will fail.
Brings 5.0 branch changes to master.
The S3 consumer offset sink will have another PR