Dynamic Partitioning in the middle on pipeline

[mudravrik]

Imagine situation where I have assets A and B.
The A is an exposure asset from dbt for example, so it is:

• dependent on something else eager-ly
• does not have actual dagster code controlled by me
• compute something like a list of object to be computeted
• there is no obvious time or static partition key there
  Asset B is:
• dependent on A
• perform some relatively heavy computation (i.e. calls some NER model and hold its massive result in RAM before flushing to IOManager), so it does not fit into one runner

With all this in mind, I have to somehow split asset B computation in chuncks. The most straight-forward way to do it for me is to add some generic partition key to asset A and Dynamic Partition with sensor to asset B. In theory, I want it work like this: A updates -> Sensor triggers and generates new partitions -> B starts and calculates all new partitions.

However, I have no clue, how to make everything work in that order without A triggering B directly instead of waiting for sensor to update the list of partitions. Any suggestions, please?

[alangenfeld]

the sensor can return a SensorResult that includes dynamic_partitions_requests for the new dynamic partitions and RunRequest s for those newly added partitions
https://docs.dagster.io/_apidocs/schedules-sensors#dagster.SensorResult