DynamoDB: Add special decoding for varied lists #39
Conversation
| # Define name of the column where typed DynamoDB fields will get materialized into. | ||
| # This column uses the `OBJECT(DYNAMIC)` data type. | ||
| TYPED_COLUMN = "data" | ||
|
|
||
| # Define name of the column where untyped DynamoDB fields will get materialized into. | ||
| # This column uses the `OBJECT(IGNORED)` data type. | ||
| UNTYPED_COLUMN = "aux" |
There was a problem hiding this comment.
Running out of "naming things", please advise if you can come up with better labels. I think data is fine, but maybe we can find a better name for the aux column together.
The implementation idea is from @proddata, so explicitly tagging him about it hereby.
| def test_decode_typed_untyped(): | ||
| assert DynamoDBCDCTranslator(table_name="foo").decode_record( | ||
| {"foo": {"N": "84.84"}, "bar": {"L": [{"N": "1"}, {"S": "foo"}]}} | ||
| ) == DualRecord(typed={"foo": 84.84}, untyped={"bar": [1.0, "foo"]}) |
There was a problem hiding this comment.
The gist, in outcome, is best viewed here.
| @@ -71,6 +72,7 @@ def test_to_sql_list_of_objects(): | |||
| "utm_source": "google", | |||
| } | |||
| ] | |||
| } | |||
| }, | |||
| "untyped": {}, | |||
There was a problem hiding this comment.
Shouldn't we add a test including untyped columns?
There was a problem hiding this comment.
You are right on the spot. Another test case for that is yet missing. Thanks a stack!
A little test case I used to drive my coding procedures is here, but it doesn't do end-to-end including .to_sql() yet, it just exercises .decode_record().
Store them into a separate `OBJECT(IGNORED)` column in CrateDB.
amotl
force-pushed
the
dynamodb-varied-lists
branch
from
d7a1dd3 to
3a31a06
Compare
| def test_to_sql_all_types(): | ||
| assert DynamoDBFullLoadTranslator(table_name="foo").to_sql(RECORD_ALL_TYPES) == SQLOperation( | ||
| statement="INSERT INTO foo (data) VALUES (:record);", | ||
| statement="INSERT INTO foo (data, aux) VALUES (:typed, :untyped);", | ||
| parameters={ | ||
| "record": { | ||
| "typed": { | ||
| "id": "5F9E-Fsadd41C-4C92-A8C1-70BF3FFB9266", | ||
| "data": {"temperature": 42.42, "humidity": 84.84}, | ||
| "meta": {"timestamp": "2024-07-12T01:17:42", "device": "foo"}, | ||
| "string_set": ["location_1"], | ||
| "number_set": [0.34, 1.0, 2.0, 3.0], | ||
| "binary_set": ["U3Vubnk="], | ||
| "somemap": {"test": 1.0, "test2": 2.0}, | ||
| } | ||
| }, | ||
| "untyped": { | ||
| "list_varied": [ | ||
| {"a": 1.0}, | ||
| 2.0, | ||
| "Three", | ||
| ], | ||
| }, | ||
| }, | ||
| ) |
There was a problem hiding this comment.
Missing test case fragment has been added. /cc @seut
About
Because CrateDB can't store varied lists into
OBJECT(DYNAMIC)columns, let's break them apart and store them into a separateOBJECT(IGNORED)column.References
/cc @matriv, @proddata