feature-request: dynamically add new upstreams

[3deep5me]

Hi @chenhunghan,

i played around with this batcher in a k8s-enviroment. It works fine so far. 😊
I added a headless service and HPA before ialocol so if the utilization increases a additional ialocol pod is scheduled.
My problem now is: How to dynamic add a new upstream to the upstream list?

As far as i see it`s not possible to do this on runtime, right?

In my dreams the text-inference-batcher would get the pod IPs directly and dynamically from the k8s-api with an selector on the pods which should be added to the upstream-list. (Similar to selector on Services)

I would be interested in your thoughts on this!

[chenhunghan]

Hello, I do have plan to add the ability to resolve "real" ips from k8s, exactly as you said. However it's a bit of work.

Alternative we can put multiple containers in a pod (e.g. tib + multiple OpenAI compatible workers), and make tib as the entry for the traffic from k8s service. We scale the pod as computing needs increase. I am more incline into later at this moment, but don't have concrete example YAMLs now, and I might have missed something. The reason for inclination is I wish tib can not only be used in k8s so not strictly tights to k8s api or its implementation, if there is a way to make it run independently in a container, that's better IMO.

To dynamically add upstream, no it's not possible at the moment unfortunately, and it's a bit of chicken-egg problem, we need to think of better way to do this architecturally so the load balancer (tib) won't into some sort of loop with upstream, I don't have a concrete idea, welcome to make a plan if you have something in mind.

[3deep5me]

The multi containers in pod approach sounds interesting but also a little bit hacky :D.

If you want a agnostic way maybe this is an idea?

Tib could do a additional ns-lookup one the host/record of an upstream and adds the IPs behind the record as upstreams.
The DNS record of a headless service has always all IPs listet which are selected by the service. This mechanism could also be used on every other platform I think.

Could may go a bit more into the chicken egg problem?
(What's the architecture problem right now?)

[chenhunghan]

Nsloop sounds good, I guess we can go with that approach. With that I guess we don't necessary need to be able to dynamically add upstreams?

[3deep5me]

If we want to Autoscale we need dynamically add upstreams. However the nslookup approach makes sense anyway I guess.

[chenhunghan]

I don't know the implementation details of HPA/k8s services, if pods scale up, does ns-lookup return more IPs?

[3deep5me]

Yes it does.
I didn't test it. But I am really sure otherwise it's kinda useless.