Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duckdb updates #337

Closed
wants to merge 1 commit into from
Closed

Duckdb updates #337

wants to merge 1 commit into from

Conversation

Swoorup
Copy link
Contributor

@Swoorup Swoorup commented Jun 11, 2024

  1. Strong types for duck db vectors.
  2. Ensure correct size are reserved before writing to duckdb vectors.
  3. Introduce DataChunkHandle for manipulating data chunks, and wrap unsafe functions at bay. There are still few handle like duckdb_appender that is better off being wrap instead of sprinkling unsafe everywhere.
  4. Add union, struct type and nested type without the duplicated code. (It should be simple enough to now add other nested types like Map)
  5. Appender keeps a cached copy of the schema for error handling and insertions.
  6. Fix large amount of record insertion (Respect the duckdb limit)
  7. Remove unsafe code for traits that need not be unsafe. Entire idea for rust is to limit usage of unsafe code when providing as a library.
  8. For unit testing, I also added arrow-convert library which makes it easier to derive arrow arrays directly from rust specific unions/structs than hand writing it.

@Swoorup
Copy link
Contributor Author

Swoorup commented Jun 11, 2024

@era127
Copy link
Contributor

era127 commented Jun 11, 2024

I think the 306 update to the test case could be merged into your changes.

@Swoorup Swoorup force-pushed the sj-merge-duckdb-fork branch 10 times, most recently from 21aa865 to f0c84c5 Compare June 16, 2024 16:37
@Swoorup
Copy link
Contributor Author

Swoorup commented Jun 16, 2024

Union feature is blocked by a suitable release i.e which includes the fix duckdb/duckdb#11326 #336

@Mytherin
Copy link
Contributor

Thanks for the PR! Could you perhaps split this up into smaller chunks so that it is easier to review?

@Swoorup Swoorup force-pushed the sj-merge-duckdb-fork branch 2 times, most recently from 5bd8a9c to 93993c6 Compare July 19, 2024 12:10
@Swoorup Swoorup force-pushed the sj-merge-duckdb-fork branch 4 times, most recently from 1359aa7 to 0ed2fcf Compare July 20, 2024 13:47
@Swoorup Swoorup force-pushed the sj-merge-duckdb-fork branch from 0ed2fcf to 1668fc7 Compare July 20, 2024 14:06
@Swoorup Swoorup force-pushed the sj-merge-duckdb-fork branch from 1668fc7 to 4885e66 Compare August 13, 2024 03:11
@Swoorup
Copy link
Contributor Author

Swoorup commented Sep 25, 2024

NOTICE: I have abandoned this work. Anyone is welcome to take over and integrate bits and pieces they may find useful.

@Swoorup Swoorup closed this Sep 25, 2024
@Swoorup Swoorup deleted the sj-merge-duckdb-fork branch October 27, 2024 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants