Add GPU/TPU addonplacementscore in node scope and cluster scope for e…

…ach managed clusters, develop the add-on in addontemplate mode.

Signed-off-by: z1ens <[email protected]>

.gitignore

@@ -13,3 +13,7 @@
 
 # Dependency directories (remove the comment below to include it)
 # vendor/
+
+# Ignore macOS and IDE specific files
+.DS_Store
+.idea/

resource-usage-collect-addon/.gitignore

@@ -14,3 +14,7 @@ bin/
 
 # Dependency directories (remove the comment below to include it)
 vendor/
+
+# Ignore macOS and IDE specific files
+.DS_Store
+.idea/

resource-usage-collect-addon/Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.21 AS builder
+FROM golang:1.22 AS builder
 WORKDIR /go/src/open-cluster-management.io/addon-contrib/resource-usage-collect
 COPY . .
 ENV GO_PACKAGE open-cluster-management.io/addon-contrib/resource-usage-collect

resource-usage-collect-addon/Makefile

@@ -20,7 +20,7 @@ PWD=$(shell pwd)
 # Image URL to use all building/pushing image targets;
 GO_BUILD_PACKAGES :=./pkg/...
 IMAGE ?= resource-usage-collect-addon
-IMAGE_REGISTRY ?= quay.io/haoqing
+IMAGE_REGISTRY ?= zheshen
 IMAGE_TAG ?= latest
 IMAGE_NAME ?= $(IMAGE_REGISTRY)/$(IMAGE):$(IMAGE_TAG)
 
@@ -43,7 +43,7 @@ vet: ## Run go vet against code.
 ##@ Build
 .PHONY: build
 build: fmt vet ## Build manager binary.
- GOFLAGS="" go build -o addon ./pkg/addon/main.go ./pkg/addon/controller.go 
+ GOFLAGS="" go build -o addon ./pkg/main.go
 
 .PHONY: images
 images: ## Build addon binary.

resource-usage-collect-addon/README.md

@@ -1,82 +1,125 @@
 
-# Prototype of extensible scheduling using resources usage.
-We already support [extensible placement scheduling](https://github.com/open-cluster-management-io/enhancements/blob/main/enhancements/sig-architecture/32-extensiblescheduling/32-extensiblescheduling.md), which allows use of [addonplacementscore](https://github.com/open-cluster-management-io/enhancements/blob/main/enhancements/sig-architecture/32-extensiblescheduling/32-extensiblescheduling.md#addonplacementscore-api) to select clusters, but we lack an addonplacementscore that contains cluster resource usage information.
+# Resource usage collect addon
 
-In this repo, I developed an addon through addon-freamwork, this addon is mainly used to collect resource usage information on the cluster, and generate an addonplacementscore under the cluster namespace of the hub.
+## Background
 
-More details refer to [Extend the multicluster scheduling capabilities with placement](https://open-cluster-management.io/scenarios/extend-multicluster-scheduling-capabilities/)
+With the rapid advancement of artificial intelligence, an increasing number of developers are required to schedule and plan AI/ML workloads based on available resources to achieve optimal performance and resource efficiency.
+
+
+Open-Cluster-Management (OCM) has already implemented `Placement` and supports [extensible placement scheduling](https://github.com/open-cluster-management-io/enhancements/blob/main/enhancements/sig-architecture/32-extensiblescheduling/32-extensiblescheduling.md), which allows for advanced, customizable workload scheduling across clusters. The key components are:
+
+- `Placement`: This feature enables the dynamic selection of a set of `ManagedClusters` within one or more `ManagedClusterSets` to facilitate Multi-Cluster scheduling.
+- `AddOnPlacementScore`: An API introduced by `Placement` to support scheduling based on customized scores.
+
+The `resource-usage-addon` is developed with `AddonTemplate`, and operates within this framework.
+- Once installed on the hub cluster, the addon deploys an agent on each managed cluster.
+- Agent pods on the managed clusters collect resource usage data and calculate a corresponding score.
+- These scores are then used by `Placement` to inform cluster selection, ensuring workloads are deployed on clusters with the most appropriate available resources.
+
+This repository, developed as part of [Google Summer of Code 2024](https://github.com/open-cluster-management-io/ocm/issues/369), introduces enhancements to the `resource-usage-addon`, including new support for scheduling based on GPU and TPU resource availability.
+This update is particularly valuable for developers seeking to optimize AI/ML workloads across multiple clusters.
+
+
+REF:
+- [GSoC 2024: Scheduling AI workload among multiple clusters #369](https://github.com/open-cluster-management-io/ocm/issues/369)
+- [Extend the multicluster scheduling capabilities with placement](https://open-cluster-management.io/scenarios/extend-multicluster-scheduling-capabilities/)
+- [What-is-an-addon](https://open-cluster-management.io/concepts/addon/#what-is-an-add-on)
+- [What-is-a-placement](https://open-cluster-management.io/concepts/placement/#select-clusters-in-managedclusterset)
+- [Enhancement:addontemplate](https://github.com/open-cluster-management-io/enhancements/tree/main/enhancements/sig-architecture/82-addon-template)
 
 # Quickstart
-## Prepare
-You have at least two running kubernetes cluster. One is the hub cluster, the other is managedcluster.
+## Prerequisite
+1. Follow the instructions on [OCM official website](https://open-cluster-management.io/getting-started/quick-start/), install `clusteradm` command-line tool and set up a hub (manager) cluster with two managed clusters.
+ If prefer using a different kubernetes distribution, follow the instructions in [Set-hub-and-managed-cluster](https://open-cluster-management.io/getting-started/quick-start/#setup-hub-and-managed-cluster).
 
-You can create an ocm environment by running below command, which will create a hub and two managedclusters for you.
+2. Command line tool `kubectl` installed.
 
-```bash
-curl -sSL https://raw.githubusercontent.com/open-cluster-management-io/OCM/main/solutions/setup-dev-environment/local-up.sh | bash
-```
+3. [Docker](https://www.docker.com/) installed.
 
 ## Deploy
 
-Set environment variables.
+**Export `kubeconfig` file of your hub cluster.**
 
 ```bash
 export KUBECONFIG=</path/to/hub_cluster/kubeconfig> # export KUBECONFIG=~/.kube/config
 ```
 
-Build the docker image to run the sample AddOn.
+**Build the docker image to run the resource-usage-addon.**
 
 ```bash
 # build image
-export IMAGE_NAME=quay.io/haoqing/resource-usage-collect-addon:latest
+export IMAGE_NAME=zheshen/resource-usage-collect-addon-template:latest
 make images
 ```
 
-If your are using kind, load image into kind cluster.
+**If you are using kind, load image to your hub cluster.**
 
 ```bash
 kind load docker-image $IMAGE_NAME --name cluster_name # kind load docker-image $IMAGE_NAME --name hub
 ```
 
-And then deploy the example AddOns controller on hub cluster.
+**On the hub cluster, deploy the addon.**
 
 ```bash
 make deploy
 ```
 
-On the hub cluster, verify the resource-usage-collect-controller pod is running.
-```bash
-$ kubectl get pods -n open-cluster-management | grep resource-usage-collect-controller
-resource-usage-collect-controller-55c58bbc5-t45dh 1/1 Running 0 71s
-```
+## What's Next
 
-## What is next
+If deployed successfully:
 
-After the deployment is complete, addon will create an addonplacementscore in its own namespace for each managedcluster in the hub.
+On the hub cluster, you can see the `AddonTemplate`, and check the `ManagedClusterAddon` status.
+```bash
+$ kubectl get addontemplate
+NAME ADDON NAME
+resource-usage-collect resource-usage-collect
+
+$ kubectl get mca -A
+NAMESPACE NAME AVAILABLE DEGRADED PROGRESSING
+cluster1 resource-usage-collect True False
+cluster2 resource-usage-collect True False
+```
 
+After a short while, on the hub cluster, `AddonPlacementScore` for each managed cluster will be generated.
 ```bash
 $ kubectl config use kind-hub
 $ kubectl get addonplacementscore -A
 NAMESPACE NAME AGE
 cluster1 resource-usage-score 3m23s
 cluster2 resource-usage-score 3m24s
 ```
+### Resource Scoring Strategies
+
+#### Node Scope Score
+- Node Scope Score: Indicates the available resources on the node with the most capacity in the cluster, aiding in selecting the best node for resource-intensive workloads.
+- Code Representation: Represented as `cpuNodeAvailable`, `gpuNodeAvailable`, etc., indicating available CPU and GPU resources on specific nodes.
+
+#### Example Use Scenarios:
+- Scenario: Suppose you have a cluster with three nodes: Node A with 2 available GPUs, Node B with 4 available GPUs, and Node C with 6 available GPUs. You need to deploy a job that requires 1 GPU.
+- Scheduling Strategies: Using the Node Scope Score, specifically `gpuNodeAvailable`, the scheduler could identify Node A as the optimal node by choosing a lower `gpuNodeAvailable` for this job under a bin-packing strategy. The scheduler would prefer to place the job on Node A to keep Nodes B and C more available for future jobs that may require more resources. This approach minimizes fragmentation and ensures that larger jobs can be accommodated later.
 
-### For example
+#### Cluster Scope Score
+- Cluster Scope Score reflects the total available resources across the entire cluster, helping to determine if the cluster can support additional workloads.
+- Code Representation: Represented as `cpuClusterAvailable`, `gpuClusterAvailable`, etc., aggregating available resources across all nodes in the cluster.
 
-Select a cluster with more available CPU.
+#### Example Use Scenarios:
+- Scenario: Consider a multi-cluster environment where Cluster X has 10 available GPUs across all nodes, Cluster Y has 6 available GPUs, and Cluster Z has 8 available GPUs. You need to deploy two jobs that first one requires 3 GPUs, and the other requires 4 GPUs.
+- Scheduling Strategies: Using the Cluster Scope Score, specifically `gpuClusterAvailable`, the scheduler would prefer the first job Cluster X for the job because it has the most available GPU resource. Then the Cluster X's score becoming lower, the scheduler will then deploy the second job on Cluster Z. This ensures that workloads are spread out, maximizing resource utilization across clusters and avoiding overloading a single cluster.
+
+### Use Placement to select clusters
+Consider this example use case: As a developer, I want to select a cluster with the most available GPU resources and deploy a job on it.
 
 Bind the default ManagedClusterSet to default Namespace.
 ```bash
 clusteradm clusterset bind default --namespace default
 ```
-
+User could create a placement to select one cluster who has the most GPU resources.
 ```bash
 cat <<EOF | kubectl apply -f -
 apiVersion: cluster.open-cluster-management.io/v1beta1
 kind: Placement
 metadata:
- name: placement
+ name: placement1
  namespace: default
 spec:
  numberOfClusters: 1
@@ -87,18 +130,23 @@ spec:
  type: AddOn
  addOn:
  resourceName: resource-usage-score
- scoreName: cpuAvailable
+ scoreName: gpuAvailable
  weight: 1
 EOF
 ```
-
+After the `placement` is created, the user wants to deploy a job to the selected cluster, could use `clusteradm` command to combine these two steps.
 ```bash
-kubectl get placementdecisions -A
-```
+clusteradm create work my-first-work -f work1.yaml --placement default/placement1
+````
+Then the work will be deployed to the cluster who has been selected. User could see the changes in `addonPlacementScore` if the GPU resources has been consumed by the job.
 
-# Clean up
+# Uninstall in the addon.
 
 ```bash
 # clean up this addon
 make undeploy
 ```
+
+### Troubleshoot
+1. If `make deploy` could not work, it might be that there has an auto-generated `kustomization_tmp.yaml.tmp` file, delete it and rerun the command.
+ Also make sure you are under hub cluster context, check the `kustomization.yaml` file, delete the part under `configMapGenerator`(if there is one exists).

resource-usage-collect-addon/deploy/kustomization.yaml

@@ -1,13 +1,13 @@
-namespace: open-cluster-management
-
 resources:
-- resources/clusterrole.yaml
-- resources/controller.yaml
-- resources/clustermanagementaddon.yaml
+- resources/addon-template.yaml
+- resources/cluster-management-addon.yaml
+- resources/cluster-role.yaml
+- resources/managed-cluster-set-binding.yaml
+- resources/placement.yaml
 
 images:
 - name: example-addon-image
- newName: quay.io/open-cluster-management/resource-usage-collect-addon
+ newName: zheshen/resource-usage-collect-addon
  newTag: latest
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization

resource-usage-collect-addon/deploy/resources/addon-template.yaml

@@ -0,0 +1,72 @@
+apiVersion: addon.open-cluster-management.io/v1alpha1
+kind: AddOnTemplate
+metadata:
+ name: resource-usage-collect
+spec:
+ addonName: resource-usage-collect
+ agentSpec:
+ workload:
+ manifests:
+ - kind: ClusterRole
+ apiVersion: rbac.authorization.k8s.io/v1
+ metadata:
+ name: resource-usage-collect-agent
+ rules:
+ - apiGroups: [ "" ]
+ resources: [ "nodes","configmaps", "pods", "events" ]
+ verbs: [ "get", "list", "watch", "create", "update", "delete", "deletecollection", "patch" ]
+ - apiGroups: [ "coordination.k8s.io" ]
+ resources: [ "leases" ]
+ verbs: [ "create", "get", "list", "update", "watch", "patch" ]
+ - kind: ClusterRoleBinding
+ apiVersion: rbac.authorization.k8s.io/v1
+ metadata:
+ name: resource-usage-collect-agent
+ roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: resource-usage-collect-agent
+ subjects:
+ - kind: ServiceAccount
+ name: resource-usage-collect-agent-sa
+ namespace: open-cluster-management-agent-addon
+ - kind: Deployment
+ apiVersion: apps/v1
+ metadata:
+ name: resource-usage-collect-agent
+ namespace: open-cluster-management-agent-addon
+ labels:
+ app: resource-usage-collect-agent
+ spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: resource-usage-collect-agent
+ template:
+ metadata:
+ labels:
+ app: resource-usage-collect-agent
+ spec:
+ serviceAccount: resource-usage-collect-agent-sa
+ containers:
+ - name: resource-usage-collect-agent
+ image: zheshen/resource-usage-collect-addon:latest
+ imagePullPolicy: Always
+ args:
+ - "/addon"
+ - "agent"
+ - "--hub-kubeconfig={{HUB_KUBECONFIG}}"
+ - "--cluster-name={{CLUSTER_NAME}}"
+ - "--addon-namespace=open-cluster-management-agent-addon"
+ - kind: ServiceAccount
+ apiVersion: v1
+ metadata:
+ name: resource-usage-collect-agent-sa
+ namespace: open-cluster-management-agent-addon
+ registration:
+ - type: KubeClient
+ kubeClient:
+ hubPermissions:
+ - type: CurrentCluster
+ currentCluster:
+ clusterRoleName: open-cluster-management:resource-usage-collect:agent

resource-usage-collect-addon/deploy/resources/cluster-management-addon.yaml

@@ -0,0 +1,17 @@
+apiVersion: addon.open-cluster-management.io/v1alpha1
+kind: ClusterManagementAddOn
+metadata:
+ name: resource-usage-collect
+spec:
+ addOnMeta:
+ displayName: resource-usage-collect
+ supportedConfigs:
+ - group: addon.open-cluster-management.io
+ resource: addontemplates
+ defaultConfig:
+ name: resource-usage-collect
+ installStrategy:
+ type: Placements
+ placements:
+ - name: placement-all
+ namespace: default

resource-usage-collect-addon/deploy/resources/cluster-role.yaml

@@ -0,0 +1,31 @@
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+ name: open-cluster-management:resource-usage-collect:agent
+rules:
+ - apiGroups: [ "" ]
+ resources: [ "nodes","configmaps", "pods", "events" ]
+ verbs: [ "get", "list", "watch", "create", "update", "delete","deletecollection", "patch" ]
+ - apiGroups: [ "coordination.k8s.io" ]
+ resources: [ "leases" ]
+ verbs: [ "create", "get", "list", "update", "watch", "patch" ]
+ - apiGroups: [ "cluster.open-cluster-management.io" ]
+ resources: [ "addonplacementscores" ]
+ verbs: [ "get", "list", "watch", "create", "update", "delete","deletecollection", "patch" ]
+ - apiGroups: [ "cluster.open-cluster-management.io" ]
+ resources: [ "addonplacementscores/status" ]
+ verbs: [ "update", "patch" ]
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+ name: open-cluster-management-addon-manager-resource-usage-collect
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: open-cluster-management:resource-usage-collect:agent
+subjects:
+ - kind: ServiceAccount
+ name: addon-manager-controller-sa
+ namespace: open-cluster-management-hub