Scalability

[grossmj]

There are multiple options to do this.

Current situation

Currently, each node has a compute_id field to tell the controller on which compute to run.

Some nodes, like cloud nodes can have their compute_id set to null/none, in this case the user has to manually select what compute to use (this happens when you drag & drop the node in the user interface)

Once created, a node cannot be moved to another compute.

Option 1 - Controller decides

Given a pool of computes, the controller automatically decides which compute to use in case no compute_id is set. However, not all computes perform the same so the controller must decide which compute is the "best".

The decision could be based on multiple metrics:

• Compute allowed for the user.
• Number of vCPU/cores
• Amount of RAM
• CPU vendor (e.g. Qemu -cpu option : if 'host', then AMD vs Intel could generate issues).
• CPU performance, a coefficient could be calculated to compare.
• Average load
• Number of nodes already running

Also, a way to migrate a node to another compute should be provided but transferring images and project files could take sometime.

Option 2 - Central storage

A central storage on controller with the image pushed as needed could be an option. The only problem would be large images transfer time. An alternative would be for the controller to store images and projects in a network drive shared with computes.

One big advantage is a node could easily be moved to another compute.

Downside is this could be complicated to set up for an user. They would have to created a NFS and SMB network share and then update images_path and projects_path in gns3_server.conf to point to the network share.

Note

Both these options should be optional, users should retain the possibility to select or force a compute.

[llevier]

I propose to set a perf parameter, simple value, taken from (example) CPU benchmark.net. So for example, master = 16217, slave1=10243, slave2=31216. Other info (#cores/RAM) are accessible from gns3 process. Data storage is irrelevant since all GNS3 server should share the same storage (to be able to run any node). Physical network design is also irrelevant since it should be ready so node can work and data exchange. That is a farm :-)
When gns3 will estimate where to launch node, things are easy : get free resources, division of perf params to have a 'realistic' performance table, then depending on what the node requires (RAM/#vCPU), master asks whatever slave compute to run node.

Other option : ignore CPU model (meaning Intel vs AMD or Intel vX vs Intel vY) since NO NODE will have -cpu host.

IF any slave can run any node of the projects controller by the master, then master could propose (option) to decide where whatever node will be launched or if compute_id is enforced. Since console infos etc are provided by master, no issue for gns3 client.

Potential issues:

• slaves must be accessible via the network so console are accessibles. Solution : master acts as a ws proxy (Vigrid design).
• -cpu host + different CPU (raised above)
• Future : GPU access from Qemu (Vigrid also offers that) and so GNS3 might require to have similar option for GPU. Technically that is not GNS3's concern, at a moment admin should be professionnal and create realistic architectures. Also compute_id enforces might solve this, forcing whatever node to run only on the compute_id having the GPU.

[llevier]

Realizing something MAJOR: once a node could be ran on any server, you opened the door of...High Availability.
It could be a new controller action, monitoring nodes and upon 'all nodes + the compute they were running on' are dead, controller could start the same nodes on another GNS3 slave ;-)

[srieger1]

In our scenario Option 1 would be more important than Option 2. I totally get the benefit of having central image storage, e.g., regarding unified management of images, no need for replication etc., but when a lot of students start projects in parallel, i.e., in the beginning of a course, a lot of IO pressure would then be put on the central storage, leading to large delays to start the projects.

Maybe the typical solution in cloud infrastructures to replicate and cache images as soon as they are needed on a compute node could be an option. First access will be slow then, but subsequent ones will be fast. At least in our use case we would accept the duplicated storage and the delay for initial replication, to then have a scale-out solution for IO across the compute nodes.

Hence, regarding the scheduling of projects, we also would like to have a nearly equalized distribution of started nodes/projects across compute nodes. So the controller should, e.g., select the compute node still offering the most available resources/least number of running projects/nodes etc.

But you're right. Both options should be possible to select based on individual requirements. Supposedly, in some setups Option 2 is more important than Option 1, e.g., to deduplicate image storage and allow easy management (deprovisioning of projects, deletion of images etc.)

[llevier]

@srieger1 Agree with central images. I do the same already. However, some GNS3 behaviors are still a problem. If you upload appliance disk with same name, you could overwrite image source et potentially because of Qemu overlay, lost your VM(s).

So with Jeremy, when we studied RBAC, I proposed the concept of 'public' vs 'private' appliance. Public means it is shared, when private it is yours :-). Still to consider checking appliance image dependancies before overwriting it (at least warn user about the risk).

About your load, at this moment my starting script works around starting node considering NAS load (easy). But tomorrow Vigrid will change its display to list project THEN servers they can be ran on, and clicking on start, Vigrid will decide where and at what 'speed' to launch it. Also NFS options are critical. Since I changed the packet size + split the load on 2 NAS, selected faster filesystems and finally changed NFS options, I never faced issues but delay. Vigrid NAS is oftenly at 10Gb/s on 2 fibers and able to deliver data at 700MB/s without problem...