Complex pipelines, hierarchical pipelines, and "PipelineRunner" as a component

[mikebellerU]

Hi -- I've tried a lot of LLM orchestration tools out there and haystack seems really promising. Thanks
for making it available!

I am currently building complex RAG pipelines -- often involving multiple routes and parallel
execution. I have built one out with haystack, and I find that the pipelines are getting too
complex. I'd like to break them down into testable 'sub-pipelines' and run the smaller
ones from a higher-level pipeline. This would keep the pieces testable, and help make
the whole pipeline more understandable.

I noticed there doesn't seem to be
any PipelineRunner component, which might enable this. Is there a reason for that?
I'm suggesting something
conceptually like this. (Non working pseudo code:):

from haystack import component
from typing import Any, Dict

@component
class PipelineRunner:
    """
    Component which runs a pipeline on a given input data.
    """

    def __init__(self, pipeline: Pipeline):
        self.pipeline = pipeline

    @component.output_types(output=Dict[str, Any])
    def run(self, data: Dict[str, Any]):
        return self.pipeline.run(data=data)

Does anyone know -- Is this sort of hierarchical pipeline concept something being done already,
or perhaps on the roadmap?

Note: I can imagine also that a "ParallelPipelineRunner" would be an awesome next step after that, which
could allow reusing a sub-pipeline but running in parallel (using ThreadPoolExecutor for example) or on a cluster.

[vasanth-asokan]

Happened to stumble upon this thread.

We basically wrote something like this ourselves for composing pipelines out of sub-pipelines and have them orchestrated from the top-level. Our sub-pipelines are proper Haystack component types themselves. Once you take that approach, in your super pipeline run method, the sub-pipeline just becomes another component whose run method is orchestrated by the generic Haystack pipeline execution system.

hope that helps. Yes, would be nice if a higher level abstraction existing in Haystack itself, but it's also not very complex to invent and in general, we like how Haystack remains simple in its abstractions.