Dynamic node registration for IceGrid to enable grid autoscaling on Kubernetes

[masariello]

Would anyone see any major issues with adding support for dynamic node registration in IceGrid?

I'm new to IceGrid and investigating the opportunity of using it for fully auto-scalable deployments to Kubernetes clusters

The objective would be to simply deploy the registry(ies), and then configure the node deployments with a replica count > 1 and ephemeral storage for the services, and just see them connect to the registry and provide compute power to the grid.

IceGrid nodes can be configured with IceLocatorDiscovery which makes it easy for the nodes to advertise themselves to the registry.

But to support dynamic discovery and autoscaling on a Kubernetes deployment one would also need to:

1. remove the need for the IceGrid.Node.Name
2. remove the need for the IceGrid Registry to maintain a fixed database of node names

Does this sound like something that one could achieve with the current IceGrid code-base?

Thank you

[bernardnormier]

Nodes - and node names - are pretty fundamental in IceGrid, and I don't see how this dynamic node registration would work.

When you describe an application in IceGrid, you specify on which node(s) your servers will execute. Making the nodes dynamic / extensible would definitely be a big change.

More generally, we see Kubernetes and IceGrid as alternative orchestrators. Kubernetes is an orchestrator for containers, where a container can provide an Ice server (or several Ice servers). And IceGrid is an orchestrator for Ice servers (without containers).

If you're writing a new application with Ice and Kubernetes, I would recommend using Kubernetes directly--no IceGrid.

The more interesting situation is: you already have an application using IceGrid, and you want to migrate this application to Kubernetes without changing too much. If you are in this situation, please contact us by email.

[masariello]

If you're writing a new application with Ice and Kubernetes, I would recommend using Kubernetes directly--no IceGrid.

I'm in this case.

I guess you are referring to Kubernetes' replica count and the various auto-scaling techniques one can implement with it, right?

Thank you so much for getting back on this!

[bernardnormier]

Yes, use Kubernetes's auto-scaling tools to deploy more pods/containers. You would also typically use Kubernetes DNS for discovery/replication.

We're certainly interested in adding features to Ice that make Kubernetes + Ice work well together. Please let us know if you have more ideas!

For example, we may add a "health check adapter" that exposes your Ice server as a gRPC service with respect to liveness:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe

[masariello]

Thank you Bernard!

Let me explain my type of app. I've come across a few incarnations of this problem, so I reckon you might find there's a market for a generic OSS solution.

I've reviewed the Ice docs to see if any of the services do this, but I didn't find a good fit. Feel free to correct my aim here.

This kind of requireement used to be called "grid computing". It's similar to what IceStorm does, but IceStom is more like a pub-sub eventing framework, if I understood the docs correctly.

The requirement is that client has to invoke SomeInterface::someMethod with a very large number of slightly different inputs, and receive the responses

We are talking from a few 100K to a few million invocations. Think Monte Carlo simulation of some model, for ex. But I can think of various other apps that need essentially the same thing.

The arguments for each invocation of SomeInterface::someMethod can go from <1K to maybe 1~2MB, so the entire set of invocations can end up moving around quite a lot data on the wire. But this is really not something that the middleware can help much with. It's really a design matter for the app itself.

If you layer this "grid computing" model, then you end up with something that looks like a Map-Reduce architecture, where sub-batches of results can be reduced by an intermediate server.

A common solution is introducing an intermediary "ThreadPool" server that places every single invocation request on a distributed queue and have the worker nodes consume the items. The queue would need to provide at-least-once guarantees. I think rabbitmq is a popular choice.

The queue is used purely for load balancing. Auto-scaling would be achieved by monitoring the length of it.

The ThreadPool server would add a "logical thread id" to the invocations and queue them up for processing.

The replies could be placed on a response queue and routed back to client.

Here the generic bits are the ThreadPool broker server, and the logic to consumer the work items from queue and return the responses on the response queue.

[bernardnormier]

Ice does not support directly your use-case. In particular, IceStorm is not a good fit: IceStorm is about forwarding one-way requests - with IceStorm, clients (publishers) and servers (subscribers) are decoupled and you don't get responses back.

You could use IceGrid and IceGrid replication for such an application. With IceGrid, the request don't transit through IceGrid: they get queued on the actual servers ("workers"), not the IceGrid registries/nodes. So, once a request is queued on a worker, another worker can't take it. And IceGrid doesn't provide auto-scaling.

A few years back, we would have looked at your use-case and thought "what if we add feature X and Y to IceGrid to support use-case?". Like find a way to accept a request in one busy worker and auto-forward this request to an idle worker.

The better approach - the IceRPC approach - is to give you APIs to help you to implement this "grid computing system" yourself, with exactly the logic you want. For this particular system, you could use IceRPC's excellent support for queueing requests and routing requests and responses in intermediary servers like this "ThreadPool server".