Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement store and load for wrapped types (for Hive & Iceberg) #4

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

emk
Copy link
Collaborator

@emk emk commented Dec 9, 2024

This PR contains the work-in-progress for Hive support using the dbcrossbar_trino crate. It was a joint project by @hanakslr and me.

I fixed a few more issues this weekend!

This currently only fails 7 tests, all related to SELECT wildcards in some way. So we're only missing one final fix.

@hanakslr, this was a fun challenge, and I enjoyed working on it!

Current Hive test status
cargo run -- sql-test --database "trino://admin@localhost/hive/default" ./tests/sql/
   Compiling joinery v0.1.0 (/home/emk/src/joinery)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7.02s
     Running `target/debug/joinery sql-test --database 'trino://admin@localhost/hive/default' ./tests/sql/`
.P......3............PP..............................E..........EE.EE.EE...

FAILED 1: tests/sql/queries/from/unnest_struct.sql
error: could not infer types
  ┌─ tests/sql/queries/from/unnest_struct.sql:6:8
  │
6 │ SELECT * FROM UNNEST([STRUCT<a INT64, b INT64>(1, 2)]);
  │        ^^ not yet implemented: InsertStoreExpressions(Wildcard)


FAILED 2: tests/sql/queries/qualify.sql
error: Internal error: cannot store INT64 in a column unless it has been converted to a storage type (at unknown location): missing conversion to storage type


FAILED 3: tests/sql/queries/select_arguments/except.sql
error: could not infer types
  ┌─ tests/sql/queries/select_arguments/except.sql:5:8
  │
5 │ SELECT * EXCEPT (b) FROM t1;
  │        ^^^^^^^^^^^^^ not yet implemented: InsertStoreExpressions(Wildcard)


FAILED 4: tests/sql/queries/select_arguments/struct_star.sql
error: could not infer types
  ┌─ tests/sql/queries/select_arguments/struct_star.sql:4:8
  │
4 │ SELECT STRUCT(1 AS a, 'b' AS b).*;
  │        ^^^^^^^^^^^^^^^^^^^^^^^^^^ not yet implemented: InsertStoreExpressions(TableNameWildcard)


FAILED 5: tests/sql/queries/select_arguments/table_star.sql
error: could not infer types
  ┌─ tests/sql/queries/select_arguments/table_star.sql:5:8
  │
5 │ SELECT t.* FROM table_star AS t;
  │        ^^^^ not yet implemented: InsertStoreExpressions(TableNameWildcard)


FAILED 6: tests/sql/queries/set_operations/intersect_distinct.sql
error: could not infer types
  ┌─ tests/sql/queries/set_operations/intersect_distinct.sql:5:8
  │
5 │ SELECT * FROM t1
  │        ^^ not yet implemented: InsertStoreExpressions(Wildcard)


FAILED 7: tests/sql/queries/set_operations/order_and_limit.sql
error: could not infer types
   ┌─ tests/sql/queries/set_operations/order_and_limit.sql:38:14
   │
38 │       SELECT * FROM names1 AS n
   │              ^^ not yet implemented: InsertStoreExpressions(Wildcard)


Pending tests:
  tests/sql/data_types/cast_types.sql (TESTS ONLY: Rust driver seems to have problems with NULL in certain columns)
  tests/sql/functions/simple/format_datetime.sql (Possible, effort may vary)
  tests/sql/functions/simple/generate_date_array.sql (Possible, effort may vary)

FAIL: 7 tests failed, 64 passed, 3 pending
ERROR: Some tests failed

CC @dave-shirley-faraday @seamusabshere

hanakslr and others added 5 commits December 6, 2024 15:11
This turned out to be surprisingly tricky. The key insight is that LOAD
and STORE aren't really symmetrical, at least not in this version of the
code. Instead, %STORE(expr) gets manually inserted as a phantom
operator, but %LOAD(var_ref) only occurs as part of a variable
reference.

This means that transforms, `sql_quote!` and all our other rewriting
machinery don't even notice that LOADs exist.

This is a weird decision, and it might not be optimal in the long run,
but it works.

This brings us down to `7 tests failed, 64 passed, 3 pending`, and
all the failing test cases are places we need to expand SELECT wildcards
like `*`, `table.*`, etc., into column lists with explicit STORE
operations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants