Skip to content

Commit

Permalink
feat: add metadata cache provider with APIs for write and query
Browse files Browse the repository at this point in the history
This adds the MetaDataCacheProvider for managing metadata caches in the
influxdb3 instance. This includes APIs to create caches through the WAL
as well as from a catalog on initialization, to write data into the
managed caches, and to query data out of them.

The query side is fairly involved, relying on Datafusion's TableFunctionImpl
and TableProvider traits to make querying the cache using a user-defined
table function (UDTF) possible.

The predicate code was modified to only support two kinds of predicates:
IN and NOT IN, which simplifies the code, and maps nicely with the DataFusion
LiteralGuarantee which we leverage to derive the predicates from the
incoming queries.

A custom ExecutionPlan implementation was added specifically for the
metadata cache that can report the predicates that are pushed down to
the cache during query planning/execution.

A big set of tests was added to to check that queries are working, and
that predicates are being pushed down properly.
  • Loading branch information
hiltontj committed Nov 21, 2024
1 parent 53f54a6 commit 5d159c9
Show file tree
Hide file tree
Showing 11 changed files with 1,956 additions and 596 deletions.
8 changes: 7 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 8 additions & 2 deletions influxdb3_cache/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,18 @@ influxdb3_wal = { path = "../influxdb3_wal" }
# crates.io dependencies
anyhow.workspace = true
arrow.workspace = true
async-trait.workspace = true
datafusion.workspace = true
indexmap.workspace = true
parking_lot.workspace = true
thiserror.workspace = true
tokio.workspace = true

[dev-dependencies]
# Core Crates
arrow_util.workspace = true
# Local deps
influxdb3_write = { path = "../influxdb3_write" }
# crates.io
insta.workspace = true

[lints]
workspace = true
Loading

0 comments on commit 5d159c9

Please sign in to comment.