Enable labels for ClusterUUID and CliqueId #965

ArangoGutierrez · 2024-09-24T16:27:54Z

No description provided.

internal/resource/types.go

internal/resource/nvml-mig-device.go

internal/lm/nvml.go

internal/resource/nvml-device.go

internal/lm/nvml.go

elezar · 2024-09-25T08:52:57Z

internal/lm/nvml.go

@@ -218,6 +224,45 @@ func newGPUModeLabeler(devices []resource.Device) (Labeler, error) {
 return labels, nil
 }

+func newImexDomainLabeler(devices []resource.Device) (Labeler, error) {
+ var commonClusterUUID, commonCliqueID string


nit: could we separate these lines? It makes future diffs easier.

PR now on new direction

cmd/gpu-feature-discovery/main.go

klueska · 2024-09-25T12:15:47Z

deployments/helm/nvidia-device-plugin/templates/daemonset-gfd.yml

+ {{- if .Values.imex.enabled }}
+ - name: imex-nodes-config
+ mountPath: {{ .Values.imex.configFile | quote }}
+ {{- end }}


I'm not sure if we need an enabled flag around any of these mounting directives. Instead we should wrap it with a {{- if typeIs "string" .Values.imexNodesConfigFile }} such that when this is nil (the default), we use the default value for that is set for this file in the CLI.

Noted, will edit

Actually thinking about this again -- we alway need to mount (at least) the folder for this, don't we? Otherwise the file won't be made available in the container (even though there is a default set path for it in the binarx).

internal/lm/nvml.go

deployments/container/native-only.mk

ArangoGutierrez · 2024-09-30T13:43:12Z

go.sum

@@ -1,7 +1,7 @@
 github.com/NVIDIA/go-gpuallocator v0.5.0 h1:166ICvPv2dU9oZ2J3kJ4y3XdbGCi6LhXgFZJtrqeu3A=
 github.com/NVIDIA/go-gpuallocator v0.5.0/go.mod h1:zos5bTIN01hpQioOyu9oRKglrznImMQvm0bZllMmckw=
-github.com/NVIDIA/go-nvlib v0.6.1 h1:0/5FvaKvDJoJeJ+LFlh+NDQMxMlVw9wOXrOVrGXttfE=
-github.com/NVIDIA/go-nvlib v0.6.1/go.mod h1:9UrsLGx/q1OrENygXjOuM5Ey5KCtiZhbvBlbUIxtGWY=
+github.com/NVIDIA/go-nvlib v0.6.2-0.20240928162840-41955a08425b h1:k5ptZB9RGUaR5RcK0R8Cfa4mtTHrSZZ73BFyD3c6KvM=


Change introduced at NVIDIA/go-nvlib#44

elezar · 2024-09-30T14:53:58Z

deployments/helm/nvidia-device-plugin/templates/daemonset-gfd.yml

@@ -182,6 +186,10 @@ spec:
 mountPath: "/etc/kubernetes/node-feature-discovery/features.d"
 - name: host-sys
 mountPath: "/sys"
+ {{- if and (eq (typeOf .Values.imexNodesConfigFile) "string") (ne .Values.imexNodesConfigFile "") }}


Question: Why not use typeIs?

changed back to typeIs

elezar · 2024-10-01T10:11:50Z

internal/resource/nvml-device.go

+
+ // ClusterUUID
+ byteSlice := gfInfo.ClusterUuid[:]
+ clusterUUID := string(byteSlice)


Why not just:

Suggested change

clusterUUID := string(byteSlice)

clusterUUID := string(gfInfo.ClusterUuid)

The resulting label is nvidia.com/gpu.clusteruuid:�Hg�/B��U��6~�

elezar · 2024-10-01T10:15:00Z

internal/lm/nvml.go

@@ -254,3 +353,20 @@ func getDeviceClasses(devices []resource.Device) ([]uint32, error) {
 }
 return classes, nil
 }
+
+// generateUUID generates a UUID-like string based on an MD5 hash of the seed and random bytes.
+func generateUUID(seed string) string {


Question: Should this really be random? Given the same input, should we not expect the UUID to be the same?

Does using https://pkg.go.dev/hash/[email protected] not serve our purposes here? (I'm fine with using md5 -- in which case we should just add the relevant //nolint annotations.

I tried hash and md5, all return values too long not accepted as a label value. I have added the //nolint to use @klueska sugested UUID generator

It's not random -- it just uses the rand package sourced with a specific seed to guarantee it always gives us back the same value.

There's likely no reason we couldn't use the crypto package for the same purpose though -- I just used rand when I first wrote this because it was the first one I found that did what I needed.

elezar · 2024-10-01T10:17:01Z

internal/lm/nvml.go

+ sort.Strings(ips)
+
+ // Join the sorted IPs into a single string
+ sortedIPs := strings.Join(ips, "\n")


It may be more readable to generate the config ID here instead of where we consume sortedIPs. We're not really interested in the IPs, but the ID associated with the unique set.

you mean not creating the variable sortedIPs but directly injecting the strings.Join(ips, "\n") when we call the generateUUID?

Agreed. I in fact, i would reorganize this whole function to get the clusterUUID and cliqueIDs first, and then generate the IMEX domain ID and then inserting it directly in the labels without this intermediate veriable.

elezar · 2024-10-01T10:21:11Z

internal/lm/nvml.go

+ labels := Labels{
+ "nvidia.com/gpu.clusteruuid": commonClusterUUID,
+ "nvidia.com/gpu.cliqueid": commonCliqueID,
+ "nvidia.com/gpu.imex-domain": generateUUID(sortedIPs) + "-" + commonCliqueID,


Does this make sense if generateUUID doesn't return the same value across restarts?

given that is the same seed, the resulting UUID should be the same

from all my testing, I have always got the same label 0f884867-ba2f-4294-9155-b495ff367eea , so the UUID is tied to the Ip's , even after a GFD restart

I wrote the generateUUID function this was borrowed from specifically to ensure that it always generates the same number given the same seed value.

klueska · 2024-10-01T11:33:36Z

deployments/helm/nvidia-device-plugin/templates/daemonset-gfd.yml

@@ -199,6 +205,9 @@ spec:
 - name: host-sys
 hostPath:
 path: "/sys"
+ - name: imex-nodes-config
+ hostPath:
+ path: "/etc/nvidia-imex"


This needs to be relative to hostDriverRoot

This mount is going to fail if this doesn't exist on the host.

Isn't there a way to make it just mount an empty directory if it doesn't exist?

I believe the option is something like DirectoryOrCreate

https://kubernetes.io/docs/concepts/storage/volumes/#hostpath-volume-types

klueska · 2024-10-01T11:34:40Z

deployments/helm/nvidia-device-plugin/values.yaml

+imex:
+ nodesConfigFile: null


I don't think this should be configurable from helm. Making it configurable from the config file should be enough.

klueska · 2024-10-01T11:38:39Z

internal/lm/nvml.go

+ if err != nil {
+ return nil, fmt.Errorf("error getting clique ID: %v", err)
+ }


What is this random err check here?

removed, is a leftover from when we were calling getGpuFabric twice

elezar · 2024-10-01T11:40:28Z

internal/lm/nvml.go

+ if err != nil && os.IsNotExist(err) {
+ return nil, nil
+ }
+ if err != nil {
+ return nil, fmt.Errorf("failed to open imex config file: %v", err)
+ }


Suggested change

if err != nil && os.IsNotExist(err) {

return nil, nil

}

if err != nil {

return nil, fmt.Errorf("failed to open imex config file: %v", err)

}

if os.IsNotExist(err) {

return nil, nil

} else if err != nil {

return nil, fmt.Errorf("failed to open imex config file: %v", err)

}

I prefer the first one myself .... I have a hatred for else statements if they can be avoided.

elezar · 2024-10-01T11:44:07Z

internal/lm/nvml.go

+ ok, err := d.IsFabricAttached()
+ if !ok {
+ continue
+ }
+ if err != nil {
+ return nil, fmt.Errorf("error checking imex capability: %v", err)
+ }


rename the ok variable. Should we also not check err first?

Suggested change

ok, err := d.IsFabricAttached()

if !ok {

continue

}

if err != nil {

return nil, fmt.Errorf("error checking imex capability: %v", err)

}

isFabricAttached, err := d.IsFabricAttached()

if err != nil {

return nil, fmt.Errorf("error checking imex capability: %v", err)

}

if !isFabricAttached {

continue

}

elezar · 2024-10-01T11:51:24Z

internal/lm/nvml.go

@@ -218,6 +231,86 @@ func newGPUModeLabeler(devices []resource.Device) (Labeler, error) {
 return labels, nil
 }

+func newImexDomainLabeler(configFile string, device []resource.Device) (Labeler, error) {


Let's move the IMEX specifics to a separate file.

with a new NewImexLabeler ?

elezar · 2024-10-01T11:51:41Z

internal/lm/nvml.go

@@ -218,6 +231,86 @@ func newGPUModeLabeler(devices []resource.Device) (Labeler, error) {
 return labels, nil
 }

+func newImexDomainLabeler(configFile string, device []resource.Device) (Labeler, error) {


Suggested change

func newImexDomainLabeler(configFile string, device []resource.Device) (Labeler, error) {

func newImexDomainLabeler(configFile string, devices []resource.Device) (Labeler, error) {

elezar · 2024-10-01T11:51:57Z

internal/lm/nvml.go

+
+ var commonClusterUUID string
+ var commonCliqueID string
+ for _, d := range device {


Suggested change

for _, d := range device {

for _, device := range devices {

elezar · 2024-10-01T11:57:57Z

internal/resource/nvml-device.go

+ byteSlice[10:16])
+
+ // CliqueID
+ cliqueId := fmt.Sprintf("%d", uint64(gfInfo.CliqueId))


Since CliqueId is already a uint32 should we not just be able to Sprintf it directly?

Suggested change

cliqueId := fmt.Sprintf("%d", uint64(gfInfo.CliqueId))

cliqueId := fmt.Sprintf("%d", gfInfo.CliqueId)

Signed-off-by: Carlos Eduardo Arango Gutierrez <[email protected]>

ArangoGutierrez · 2024-10-02T14:28:43Z

Unit test has been fixed, and also golangci-lint is happy now, PR is ready for review

ArangoGutierrez requested review from klueska and elezar September 24, 2024 16:27

ArangoGutierrez self-assigned this Sep 24, 2024