Outsourcing Reader (and other highly resource consuming component)

[guillim]

Question
Hello, I am wondering if there is a way to outsource the Reader, or any other highly resource consuming component.

Additional context
The point is you can use a 2Go RAM server for 99% of any pipelines. It will fit most use-cases. However, the Reader part will require very specific hardware/RAM and can't scale. So instead of having to scale the Haystack server, we could simply have an outside service for running our Reader only. This external dedicated service could be connected as in the pipelines.yaml

Curious to read your thoughts

[lalitpagaria]

In my view it is nice idea. It will help in building scalable pipeline (for example scaling individual component and putting it behind load balancer with caching support)

There are multiple ways to achieve this but one quick hack I see to add operation mode remote and local with few components. For remove mode then user have to provide uri and pipeline will create http client which will make api call to that uri. Supplied rest endpoint should support same request and response, which are supported by it's run function. For example Reader component in remote mode should follow this syntax.

I am exploring this space for Obsei, like ray, temporal, airflow etc. But recently come across Apache Beam's Pipeline IO implementation https://beam.apache.org/documentation/io/built-in/. From design point of view I liked it. SO main idea to add communication (rest, binary protocol, streaming etc) as a part of edge and node have to follow what edge is require. But yes it will make make tool quite bloated.

I am curious to get more views around this topic :)

[guillim]

I agree with the URI first implementation. That will be the fastest way to make it happen !

[tholor]

Yep, definitely a good idea and direction. We are exploring Ray for the purpose of parallelizing our pipelines and allowing distributed execution (#688). For our deployments, I could see a mix of Ray and Kubernetes to distribute our pipeline nodes to the right machines, but we are still in the phase of sorting out a good design here. open for any ideas and experience sharing regarding the above frameworks. From my perspective airflow is more suitable for batch jobs than real-time search queries. Not sure how Apache Beam fits in there, but I also mostly heard from people using it for bigger batch jobs with Spark & Co.

[guillim]

@MichaelBitard adding you here since we started talking about it last week

[lalitpagaria]

Even there is demand in Ray to integrate Apache Beam, if you like then bump up +1 count there -
ray-project/ray#16622

[tholor]

Moved it here to discussions as we want in future a clearer separation between these "early ideation & discussion" and actual issues/bugs/features.

[lalitpagaria]

On DataTalks.Club community, I asked question about developing scalable model serving ML Pipeline to @hanneshapke (Author of "Building Machine Learning Pipelines). He suggested interesting approach of using "combination between Apache Beam and Kubernetes". Quoting his response -

Beam lets you scale all data heavy tasks. You can start off with a direct runner (runs within a k8s container) and you can later export the data heavy tasks to its own cluster (with Beam using Apache Flink or GCP Dataflow)

[guillim]

I saw recently on HuggingFace that there is an API endpoint for hosted Readers. It could be an option for companies who don't need on-premise and can publish to HF (our case lol)

Reference:
see How to serve this model with the Accelerated Inference API on this page

[lalitpagaria]

Ha ha
But their costing is very steep and based on number of chars. At least out of bound for hobbyist or small startup.

[guillim]

True. But fast to test. and also, it could be a starting point for creating the Haystack connection

[lalitpagaria]

@guillim @tholor @oryx1729 I am currently trying to revamp Obsei. Hence working in similar direction. What do you think about it. I feel some portion can be also relevant for Haystack as well. Coping ticket content here, also I would love to hear feedback about my proposed design.

• Introduce generic node structure
• Introduce different types of node structure (Observer, Informer, Analyzer, Preprocessor, Router etc)
• Constraint Each Node should be stateless, hence each incoming request will bring Node config with it (be it either via incoming payload, file, DB, Cache etc)
• Constraint Each Node should have reproducible properties ie avoid procedure call. Hence anyone can create setup locally or remotely.
• Unify input/output payload (first part if done in this ticket Unify input/output payload for each nodes obsei/obsei#144)
• Optional Payload should able to accept text, image, vectors, audio, file etc.
• Introduce payload convertor
• Introduce channel structure
• Introduce different types of channel structure (Local, Rest, Messaging, Binary Protocol, etc)
• Each channel will receive/emit payload and can have a payload convertor. Payload convertor stage will be based on where channel is used ie input or output. For example if listening to Kafka events then payload convertor stage will sit after consumer and if sending payload via Rest then convertor will sit before Rest call.
• Each Node can have multiple incoming and outgoing channels. For example analyzer node can receive payload from http request, events or locally via procedure call and similarly it can sent payload to multiple http endpoints.