DynamoDB: Fix serializing OBJECT and ARRAY representations to CrateDB

[amotl]

About

Another attempt to get relaying container type values right, this time for both OBJECTs and ARRAYs, at least a basic variant thereof, in order to unblock people needing it.

Details

The problem with ARRAYs is that they can't be relayed using JSON syntax, supported by a corresponding CAST. Instead, we need to use the native notation of CrateDB, which works without much ado on primitive inner types.

When serializing ARRAYs including OBJECT inner types, we need to actively convert to CrateDB's native syntax, which is close to Python's dict(foo='bar') syntax notation.

References

• DynamoDB: Handle container types correctly #33

[amotl]

Handling it specially, [...] close to Python's dict(foo='bar') syntax notation.

This does not apply any special treatments to the OBJECT keys yet, and I did not perform any further orientation flights yet. For example, what happens if the key is a reserved keyword, or contains special characters like dashes -?

[amotl]

Depending on your spirits, we may want to defer further special handling to a subsequent iteration, and first check if the use case at hand already gets satisfied well.

[seut]

Single quotes must always be escaped as they are used for any text literals at CrateDB.

[amotl]

Thanks. So that serialization will need further improvements to treat OBJECT values the same like above.

[seut]

What is about the other cases where the array element isn't a dict?

[amotl]

They absolutely need to be taken care of, right. Currently, as far as I can see through knowing about data payloads at hand, it might not be of current concerns yet.

[seut]

Looks like the special array(object) case is already support by casting the json string to object[]. See crate/crate#14666 and crate/crate#14840.

[amotl]

Looks like the special array(object) case is already support by casting the json string to object[].

This is absolutely right on the spot, and seems to work well. Thank you so much!

data['list_of_objects'] = '[{"foo": "bar"}, {"baz": "qux"}]'::OBJECT[]

cr> UPDATE "foo" SET data['tags'] = ['foo', 'bar'], data['empty_map'] = '{}'::OBJECT, data['empty_list'] = [], data['string_set'] = ['location_1'], data['number_set'] = [0.34, 1.0, 2.0, 3.0],
     data['binary_set'] = ['U3Vubnk='], data['somemap'] = '{"test": 1.0, "test2": 2.0}'::OBJECT, data['list_of_objects'] = '[{"foo": "bar"}, {"baz": "qux"}]'::OBJECT[] WHERE data['device'] =
    'foo';
UPDATE OK, 1 row affected (0.039 sec)

cr> select * from foo;

| {"binary_set": ["U3Vubnk="], "device": "foo", "empty_list": [], "empty_map": {}, "items": ["foo", "bar"], "list_of_objects": [{"foo": "bar"}, {"baz": "qux"}], "number_set": [0.34, 1.0, 2.0, 3.0], "somemap": {"test": 1.0, "test2": 2.0}, "string_set": ["location_1"], "tags": ["foo", "bar"]} |

[amotl]

I've amended the patch correspondingly.

[seut]

Btw. any reason not to use parameter binding instead of serializing values into the SQL? This would avoid the need of any special escaping I think and would be much safer..

[amotl]

Btw. any reason not to use parameter binding instead of serializing values into the SQL? This would avoid the need of any special escaping I think and would be much safer.

Absolutely not. It is just my fault that I didn't use it from the very beginning. A subsequent iteration certainly will bring it to the table, because it is absolutely the right choice not needing to care about those details manually. Thanks!