Performance limits for partitions

[jamiedemaria]

Assets with over 25,000 partitions can cause performance issues in Dagster.

What performance is impacted?

The main impact is to the Dagster UI. Specifically, the UI will be slower when loading the asset graph and asset detail pages.

Is this a hard limit?

No. You won't see errors if your asset has over 25,000 partitions. But you will likely see slower UI performance.

Does this limit apply across all assets, or per-asset?

This applies per asset. For example, 25 assets with 1,000 partitions each will have normal UI performance, but one asset with 25,000 partitions will result in a slower UI

Are specific kinds of partitions less performant than others?

No. Performance is generally the same across partition types

[ShootingStarD]

Hi, I just stumbled on this post and I was wondering then how to mitigate this problem?
For example at my work we have 5 minutes partitions, so partition every 5 minutes. For one month it makes already 8 640 partitions.
For time partition, we can set a start time but this start time is not dynamic, we have to hardcode it. How can I handle my time partitions so that the slow down is reduced?

The assets is partitioned in 5 minutes because we want to trigger a job every 5 minutes

Thanks for all advices

[davidonlaptop]

Hello I'm also having thoughts about this. I'm new to dagster but I think this could be resolve with a job that consolidates your old partitions into a single one. Here's an idea:

Partition Compaction Job
Since your orchestration partitions (those used by Dagster) can be different than your storage partitions (e.g. Snowflake, parquet etc), you may be able to only compact the orchestration partitions without reprocessing your data.

For example, say you create a new orchestration partition every minute, you proceed as usual and ensure it is stored properly in your final storage. Then you have a job that runs at midnight every day (or every hour), and compacts the previous partitions into one.

I believe this would need to use something like a Sensor. That sensor can publish AddPartitionRequest events and DeletePartitionRequests, that could be processed immediately, or on demand via the Ui.

There should be a way to tag those requests with a "compaction" label.

Downstream jobs
When the downstream jobs receives the AddPartitionRequest events and DeletePartitionRequests, they can simply ignore those events marked as "compaction", and mark the partitions as done. No need to touch the data in your storage.

Would that work?