You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Currently, in DynamoDB source, the DynamoDB partition key and sort key will be stored in event metadata which can then be used as doc id in OpenSearch.
However, for the event comes from export data file, the metadata info will be incorrect if the key is in numeric format.
For example, if I have a table with partition key pk and sort key sk. And for an item like below
{
"pk": 39370,
"sk": 25130455
}
If we set up to use document_id: "${getMetadata(\"primary_key\")}" in the configuration, the result in OpenSearch will be something like:
The pk in the doc id is with E notation (3.937E+4|25130455). The impact is that, if some changes to this item happens, it will create a new doc with id 39370|25130455.
To Reproduce
Steps to reproduce the behavior:
Create a table with numeric partition key or sort key, run the pipeline with doc id mapping and check the output in OpenSearch.
If you want to verify in code, here is the Ion output for that item: " $ion_1_0 {Item:{pk:3937d1,sk:25130455.}}"
Expected behavior
The primary key in metadata should contain the correct numeric format (without E notation) for exports.
Additional context
No such issue in streaming.
The text was updated successfully, but these errors were encountered:
Describe the bug
Currently, in DynamoDB source, the DynamoDB partition key and sort key will be stored in event metadata which can then be used as doc id in OpenSearch.
However, for the event comes from export data file, the metadata info will be incorrect if the key is in numeric format.
For example, if I have a table with partition key
pk
and sort keysk
. And for an item like belowIf we set up to use
document_id: "${getMetadata(\"primary_key\")}"
in the configuration, the result in OpenSearch will be something like:The pk in the doc id is with E notation (
3.937E+4|25130455
). The impact is that, if some changes to this item happens, it will create a new doc with id39370|25130455
.To Reproduce
Steps to reproduce the behavior:
Create a table with numeric partition key or sort key, run the pipeline with doc id mapping and check the output in OpenSearch.
If you want to verify in code, here is the Ion output for that item:
" $ion_1_0 {Item:{pk:3937d1,sk:25130455.}}"
Expected behavior
The primary key in metadata should contain the correct numeric format (without E notation) for exports.
Additional context
No such issue in streaming.
The text was updated successfully, but these errors were encountered: