Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] The primary key in metadata for export data is incorrect for DynamoDB source #3642

Closed
daixba opened this issue Nov 13, 2023 · 0 comments · Fixed by #3650
Closed

[BUG] The primary key in metadata for export data is incorrect for DynamoDB source #3642

daixba opened this issue Nov 13, 2023 · 0 comments · Fixed by #3650
Assignees
Labels
bug Something isn't working
Milestone

Comments

@daixba
Copy link
Contributor

daixba commented Nov 13, 2023

Describe the bug
Currently, in DynamoDB source, the DynamoDB partition key and sort key will be stored in event metadata which can then be used as doc id in OpenSearch.

However, for the event comes from export data file, the metadata info will be incorrect if the key is in numeric format.

For example, if I have a table with partition key pk and sort key sk. And for an item like below

{
 "pk": 39370,
 "sk": 25130455
}

If we set up to use document_id: "${getMetadata(\"primary_key\")}" in the configuration, the result in OpenSearch will be something like:

        "_index": "...",
        "_id": "3.937E+4|25130455",
        "_score": 1,
        "_source": {
          "pk": 39370,
          "sk": 25130455
        }
      }

The pk in the doc id is with E notation (3.937E+4|25130455). The impact is that, if some changes to this item happens, it will create a new doc with id 39370|25130455.

To Reproduce
Steps to reproduce the behavior:
Create a table with numeric partition key or sort key, run the pipeline with doc id mapping and check the output in OpenSearch.

If you want to verify in code, here is the Ion output for that item: " $ion_1_0 {Item:{pk:3937d1,sk:25130455.}}"

Expected behavior
The primary key in metadata should contain the correct numeric format (without E notation) for exports.

Additional context
No such issue in streaming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants