Is Tiered Storage currently planned?

[shafqatevo]

Hi,

Couldn't find Tiered Storage in the roadmap (https://nats.io/about/#roadmap). Can anyone pls share what's the current plan for this much-needed feature? What timeframe can it be expected?

[derekcollison]

We have it planned but no schedule yet on when we will do this work as of yet.

[shafqatevo]

Thanks Derek!

This is a major feature many will be eager to get.

[Material-Scientist]

Hi, is there an update on this?

Kafka has recently added tired storage support with the upgrade to KRAFT. I'm not sure how others feel about it, but for me it's the only thing that keeps me from dropping Kafka in favor of NATS Jetstream.

[derekcollison]

Can you tell us more about what you would value the most from tiered storage?

[Material-Scientist]

So, the (experimental) Kafka tiered storage class (org.apache.kafka.server.log.remote.storage.LocalTieredStorage) just uses a local directory to move segments once they are rolled.

In my case, that directory is a mounted SeaweedFS drive (append-only store), which first stores the data on a fast drive (e.g. SSD), and eventually moves it to slower tiers (e.g. HDD, S3).

Both Kafka & SeaweedFS cache the data back to the local drive as soon as a consumer requires old segments.

Since storage is often the most expensive part about operating Kafka, it's important to have support for tiered storage.

It would be great if NATS supported tiered storage in a similar fashion (i.e. sealed segments are moved to a different directory, and cached when needed).