NERC Maintenance (Taint nerc-ocp-prod GPUs and add acceleratorProfiles) - Jan 7

[dystewart]

Motivation

To prevent general non-GPU workloads from scheduling on nodes with GPUs we will be tainting the nerc-ocp-prod GPU nodes. This is important because users are billed on a per host basis, so currently user workloads can be unintentionally scheduled on GPU nodes even when no GPU resources are allocated, resulting in being billed for GPU usage. Additionally, this keeps the GPUs clear of all workloads not explicitly requesting GPU resources.
As a result of adding taints to the GPU nodes, we will also be adding accelerators to allow RHOAI users who have GPUs allocated, to select which tainted GPU they would like to land on (eg. A100 or V100).

We will also fix the "None" acceleratorProfile behavior in: issue

Completion Criteria

During the Jan 7 maintenance window, nerc-ocp-prod GPU nodes are tainted and acceleratorProfiles are added to the nerc-ocp-prod cluster RHOAI installation.

Description

• RHOAI user docs updated to reflect accelerator selection (required before Jan 7) Update user facing docs to reflect GPU acceleratorProfiles - Jan 7 Maintenance #852
• Taint nerc-ocp-prod GPU nodes tainted by GPU type (Jan 7) Taint nerc-ocp-prod GPU nodes (Jan 7 Maintenence) #846
• Add acceleratorProfiles to nerc-ocp-prod RHOAI (Jan 7) Add acceleratorProfiles for RHOAI in nerc-ocp-prod cluster (Jan 7 maintenance) #847
• Fix None acceleratorProfile behavior (Jan 7) PR

Completion dates

Required - 2025-01-07

[Milstein]

@here: Maintenance schedule is setup: https://nerc.instatus.com/cm4retg180022119ladrkl2t9
https://nerc.mghpcc.org/event/network-equipment-switch-maintenance-20250107/

Email announcement to all [NERC Users] is sent!