diff --git a/_docs/development-environment.md b/_docs/development-environment.md new file mode 100644 index 00000000..c3eb38a7 --- /dev/null +++ b/_docs/development-environment.md @@ -0,0 +1,252 @@ +# Setting-up development environment + +**Warning** +All links to the `provider` repo are referencing to the `gpu` branch. As soon as `gpu` is merged into `main` all links need update. + +This page covers setting up development environment for both [node](https://github.com/akash-network/node) and [provider](https://github.com/akash-network/provider) repositories. +The provider repo elected as placeholder for all the scripts as it depends on the `node` repo (Better explanation?) +Should you already know what this guide is all about - feel free to explore [examples](#how-to-use-runbook) + +## Code + +Checkout code if not done so into place of your convenience. +For this example, repositories will be located in `~/go/src/github.com/akash-network` + +Checkout below assumes `git` is set to use SSH connection to GitHub + +```shell +cd ~/go/src/github.com/akash-network # all commands below assume this as current directory +git clone git@github.com:akash-netowrk/node +git clone git@github.com:akash-netowrk/provider +``` + +## Requirements + +- `Go` must be installed. Both projects are keeping up-to-date with major version on development branches. + Both repositories are using the latest version of the Go, however only minor that has to always match. + +### Install tools + +Run following script to install all system-wide tools. +Currently supported host platforms: + +- MacOS +- Debian based OS + PRs with another hosts are welcome (except Windows) + +```shell +./provider/script/install_dev_dependencies.sh +``` + +## How it works + +### General behaviour + +All examples are located within [_run](https://github.com/akash-network/provider/blob/gpu/_run) directory. +[Commands](#commands) are implemented as `make` targets. + +There are three ways we use to set up the k8s cluster. + +- kind +- minukube +- ssh + +Both `kind` and `minikube` are e2e, i.e. the configuration is capable of spinning up cluster and the local host, whereas `ssh` expects cluster to be configured before use. + +### Runbook + +There are four configuration variants, each presented as directory within [_run](https://github.com/akash-network/provider/blob/gpu/_run). + +- `kube` - uses `kind` to set up local cluster. It is widely used by e2e testing of the provider. Provider and the node run as host services. All operators run as kubernetes deployments. +- `single` - uses `kind` to set up local cluster. Main difference is both node and provider (and all operators) are running within k8s cluster as deployments. (at some point we will merge `single` + with `kube` and call it `kind`) +- `minikube` - not in use for now +- `ssh` - expects cluster to be up and running. mainly used to test sophisticated features like `GPU` or `IP leases` + +The only difference between environments above is how they set up. Once running, all commands are the same. + +Running through the entire runbook requires multiples terminals. +Each command is marked __t1__-__t3__ to indicate a suggested terminal number. + +If at any point something goes wrong and cluster needs to be run from the beginning: + +```shell +cd _run/ +make kube-cluster-delete +make clean +``` + +### Kustomize + +TBD + +#### Parameters + +| Name | Default value | Effective on target(s) | Notes | +|:---------------------|:------------------------------------------------------------------------------:|------------------------------------------------------------------------------------------------------------------------------------------|-------| +| `SKIP_BUILD` | `false` | | +| `DSEQ` | `1` | `deployment-*`
`lease-*`
`bid-*`
`send-manifest` | +| `OSEQ` | `1` | `deployment-*`
`lease-*`
`bid-*`
`send-manifest` | +| `GSEQ` | `1` | `deployment-*`
`lease-*`
`bid-*`
`send-manifest` | +| `KUSTOMIZE_INSTALLS` | Depends on runbook
Refer to each runbook's `Makefile` to see default value | `kustomize-init`
`kustomize-templates`
`kustomize-set-images`
`kustomize-configure-services`
`kustomize-deploy-services` | | + +##### Keys + +Each configuration creates four [keys](https://github.com/akash-network/provider/blob/gpu/_run/common.mk#L40..L43): +They keys are assigned to the targets and under normal circumstances there is no need to alter it. However, it can be done with setting `KEY_NAME`: + +```shell +# create provider from **provider** key +make provider-create + +# create provider from custom key +KEY_NAME=other make provider-create +``` + +#### How to use runbook + +##### Kube + +This runbook requires three terminals + +1. Open runbook + + __all three terminals__ + ```shell + cd _run/kube + ``` + +2. Create and provision local kind cluster. + + __t1 run__ + ```shell + make kube-cluster-setup + ``` +3. Start akash node + + __t2 run__ + ```shell + make node-run + ``` +4. Create provider + + __t1 run__ + ```shell + make provider-create + ``` + +5. Start the provider + + __t3 run__ + ```shell + make provider-create + ``` + +6. Start the provider + + __t1 run__ + ```shell + make provider-create + ``` + +7. __t1__ Create a deployment. Check that the deployment was created. Take note of the `dseq` - deployment sequence: + + ```shell + make deployment-create + ``` + + ```shell + make query-deployments + ``` + + After a short time, you should see an order created for this deployment with the following command: + + ```shell + make query-orders + ``` + + The Provider Services Daemon should see this order and bid on it. + + ```shell + make query-bids + ``` + +8. __t1 When a bid has been created, you may create a lease__ + + To create a lease, run + + ```shell + make lease-create + ``` + + You can see the lease with: + + ```shell + make query-leases + ``` + + You should now see "pending" inventory in the provider status: + + ```shell + make provider-status + ``` + +9. __t1 Distribute Manifest__ + + Now that you have a lease with a provider, you need to send your + workload configuration to that provider by sending it the manifest: + + ```shell + make send-manifest + ``` + + You can check the status of your deployment with: + + ```shell + make provider-lease-status + ``` + + You can reach your app with the following (Note: `Host:` header tomfoolery abound) + + ```shell + make provider-lease-ping + ``` + +10. __t1 Get service status__ + + ```sh + make provider-lease-status + ``` + + Fetch logs from deployed service (all pods) + + ```sh + make provider-lease-logs + ``` + +##### Kube for e2e tests + +This runbook requires two terminal + +1. Open runbook + + __t1__ + ```shell + cd _run/kube + ``` + +2. Create and provision local kind cluster for e2e testing. + + __t1 run__ + ```shell + make kube-cluster-setup-e2e + +3. Run e2e tests + + ```shell + make test-e2e-intergration + ``` + +##### Single + +##### SSH diff --git a/_docs/kustomize/akash-operator-inventory/cluster_role.yaml b/_docs/kustomize/akash-operator-inventory/cluster-roles.yaml similarity index 69% rename from _docs/kustomize/akash-operator-inventory/cluster_role.yaml rename to _docs/kustomize/akash-operator-inventory/cluster-roles.yaml index fe972d19..4611249b 100644 --- a/_docs/kustomize/akash-operator-inventory/cluster_role.yaml +++ b/_docs/kustomize/akash-operator-inventory/cluster-roles.yaml @@ -62,3 +62,30 @@ rules: - get - list - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: feature-discovery-node + labels: + akash.network: "true" + app.kubernetes.io/name: feature-discovery-node + app.kubernetes.io/component: inventory + app.kubernetes.io/part-of: operator +rules: + - apiGroups: + - '' + resources: + - nodes + verbs: + - get + - list + - watch + - apiGroups: + - '' + resources: + - pods + verbs: + - get + - list + - watch diff --git a/_docs/kustomize/akash-operator-inventory/daemonset.yaml b/_docs/kustomize/akash-operator-inventory/daemonset.yaml new file mode 100644 index 00000000..491e200e --- /dev/null +++ b/_docs/kustomize/akash-operator-inventory/daemonset.yaml @@ -0,0 +1,54 @@ +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: feature-discovery-node + namespace: akash-services + labels: + akash.network: "true" + app.kubernetes.io/name: feature-discovery-node + app.kubernetes.io/component: inventory + app.kubernetes.io/part-of: operator +spec: + selector: + matchLabels: + app.kubernetes.io/name: feature-discovery-node + app.kubernetes.io/component: inventory + app.kubernetes.io/part-of: operator + template: + metadata: + labels: + akash.network: "true" + app.kubernetes.io/name: feature-discovery-node + app.kubernetes.io/component: inventory + app.kubernetes.io/part-of: operator + spec: + serviceAccountName: feature-discovery-node + containers: + - name: inventory-node + image: ghcr.io/akash-network/provider + args: + - "provider-services" + - "operator" + - "inventory" + - "node" + imagePullPolicy: IfNotPresent + ports: + - containerPort: 8081 + name: api + resources: + requests: + memory: "64Mi" + cpu: "250m" + limits: + memory: "128Mi" + cpu: "500m" + env: + - name: AP_POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: AP_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName diff --git a/_docs/kustomize/akash-operator-inventory/kustomization.yaml b/_docs/kustomize/akash-operator-inventory/kustomization.yaml index fd79eef6..56ed777f 100644 --- a/_docs/kustomize/akash-operator-inventory/kustomization.yaml +++ b/_docs/kustomize/akash-operator-inventory/kustomization.yaml @@ -2,9 +2,9 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: akash-services resources: - - deployment.yaml + - service-accounts.yaml + - cluster-roles.yaml + - role-bindings.yaml - service.yaml - - rbac.yaml - - service_account.yaml - - cluster_role.yaml - - role-binding.yaml + - daemonset.yaml + - deployment.yaml diff --git a/_docs/kustomize/akash-operator-inventory/role-binding.yaml b/_docs/kustomize/akash-operator-inventory/role-binding.yaml deleted file mode 100644 index 49766396..00000000 --- a/_docs/kustomize/akash-operator-inventory/role-binding.yaml +++ /dev/null @@ -1,17 +0,0 @@ -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: inventory-operator - labels: - akash.network: "true" - app.kubernetes.io/name: akash - app.kubernetes.io/instance: inventory - app.kubernetes.io/component: operator -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: inventory-operator -subjects: - - kind: ServiceAccount - name: inventory-operator - namespace: akash-services diff --git a/_docs/kustomize/akash-operator-inventory/rbac.yaml b/_docs/kustomize/akash-operator-inventory/role-bindings.yaml similarity index 50% rename from _docs/kustomize/akash-operator-inventory/rbac.yaml rename to _docs/kustomize/akash-operator-inventory/role-bindings.yaml index 0ff59351..f6b48d45 100644 --- a/_docs/kustomize/akash-operator-inventory/rbac.yaml +++ b/_docs/kustomize/akash-operator-inventory/role-bindings.yaml @@ -1,3 +1,40 @@ +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: inventory-operator + labels: + akash.network: "true" + app.kubernetes.io/name: akash + app.kubernetes.io/instance: inventory + app.kubernetes.io/component: operator +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: inventory-operator +subjects: + - kind: ServiceAccount + name: inventory-operator + namespace: akash-services +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: feature-discovery-node + labels: + akash.network: "true" + app.kubernetes.io/name: feature-discovery-node + app.kubernetes.io/component: inventory + app.kubernetes.io/part-of: operator +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: feature-discovery-node +subjects: + - kind: ServiceAccount + name: feature-discovery-node + namespace: akash-services +--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: diff --git a/_docs/kustomize/akash-operator-inventory/service_account.yaml b/_docs/kustomize/akash-operator-inventory/service-accounts.yaml similarity index 50% rename from _docs/kustomize/akash-operator-inventory/service_account.yaml rename to _docs/kustomize/akash-operator-inventory/service-accounts.yaml index 9be2d98b..86d3d9c2 100644 --- a/_docs/kustomize/akash-operator-inventory/service_account.yaml +++ b/_docs/kustomize/akash-operator-inventory/service-accounts.yaml @@ -1,3 +1,4 @@ +--- apiVersion: v1 kind: ServiceAccount metadata: @@ -9,3 +10,14 @@ metadata: app.kubernetes.io/instance: inventory app.kubernetes.io/component: operator automountServiceAccountToken: true +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: feature-discovery-node + namespace: akash-services + labels: + akash.network: "true" + app.kubernetes.io/name: feature-discovery-node + app.kubernetes.io/component: inventory + app.kubernetes.io/part-of: operator diff --git a/_docs/kustomize/templates/akash-operator-inventory/kustomization.yaml b/_docs/kustomize/templates/akash-operator-inventory/kustomization.yaml index 18c55e4a..8e7cfd45 100644 --- a/_docs/kustomize/templates/akash-operator-inventory/kustomization.yaml +++ b/_docs/kustomize/templates/akash-operator-inventory/kustomization.yaml @@ -11,3 +11,9 @@ patches: group: apps name: inventory-operator version: v1 + - path: docker-image.yaml + target: + kind: DaemonSet + group: apps + name: feature-discovery-node + version: v1 diff --git a/cmd/provider-services/cmd/flags/kube_config.go b/cmd/provider-services/cmd/flags/kube_config.go index e3304855..cf3e48bf 100644 --- a/cmd/provider-services/cmd/flags/kube_config.go +++ b/cmd/provider-services/cmd/flags/kube_config.go @@ -10,6 +10,6 @@ const ( ) func AddKubeConfigPathFlag(cmd *cobra.Command) error { - cmd.Flags().String(FlagKubeConfig, "$HOME/.kube/config", "kubernetes configuration file path") - return viper.BindPFlag(FlagKubeConfig, cmd.Flags().Lookup(FlagKubeConfig)) + cmd.PersistentFlags().String(FlagKubeConfig, "$HOME/.kube/config", "kubernetes configuration file path") + return viper.BindPFlag(FlagKubeConfig, cmd.PersistentFlags().Lookup(FlagKubeConfig)) } diff --git a/cmd/provider-services/cmd/root.go b/cmd/provider-services/cmd/root.go index ecd3729b..23df1192 100644 --- a/cmd/provider-services/cmd/root.go +++ b/cmd/provider-services/cmd/root.go @@ -25,7 +25,6 @@ import ( ) func NewRootCmd() *cobra.Command { - encodingConfig := app.MakeEncodingConfig() cmd := &cobra.Command{ diff --git a/cmd/provider-services/cmd/run.go b/cmd/provider-services/cmd/run.go index 04942522..621feb50 100644 --- a/cmd/provider-services/cmd/run.go +++ b/cmd/provider-services/cmd/run.go @@ -131,7 +131,7 @@ func RunCmd() *cobra.Command { cmd.Flags().String(flags.FlagChainID, "", "The network chain ID") if err := viper.BindPFlag(flags.FlagChainID, cmd.Flags().Lookup(flags.FlagChainID)); err != nil { - return nil + panic(err) } flags.AddTxFlagsToCmd(cmd) @@ -140,211 +140,211 @@ func RunCmd() *cobra.Command { cmd.Flags().Bool(FlagClusterK8s, false, "Use Kubernetes cluster") if err := viper.BindPFlag(FlagClusterK8s, cmd.Flags().Lookup(FlagClusterK8s)); err != nil { - return nil + panic(err) } cmd.Flags().String(providerflags.FlagK8sManifestNS, "lease", "Cluster manifest namespace") if err := viper.BindPFlag(providerflags.FlagK8sManifestNS, cmd.Flags().Lookup(providerflags.FlagK8sManifestNS)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagGatewayListenAddress, "0.0.0.0:8443", "Gateway listen address") if err := viper.BindPFlag(FlagGatewayListenAddress, cmd.Flags().Lookup(FlagGatewayListenAddress)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagBidPricingStrategy, "scale", "Pricing strategy to use") if err := viper.BindPFlag(FlagBidPricingStrategy, cmd.Flags().Lookup(FlagBidPricingStrategy)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagBidPriceCPUScale, "0", "cpu pricing scale in uakt per millicpu") if err := viper.BindPFlag(FlagBidPriceCPUScale, cmd.Flags().Lookup(FlagBidPriceCPUScale)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagBidPriceMemoryScale, "0", "memory pricing scale in uakt per megabyte") if err := viper.BindPFlag(FlagBidPriceMemoryScale, cmd.Flags().Lookup(FlagBidPriceMemoryScale)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagBidPriceStorageScale, "0", "storage pricing scale in uakt per megabyte") if err := viper.BindPFlag(FlagBidPriceStorageScale, cmd.Flags().Lookup(FlagBidPriceStorageScale)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagBidPriceEndpointScale, "0", "endpoint pricing scale in uakt") if err := viper.BindPFlag(FlagBidPriceEndpointScale, cmd.Flags().Lookup(FlagBidPriceEndpointScale)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagBidPriceIPScale, "0", "leased ip pricing scale in uakt") if err := viper.BindPFlag(FlagBidPriceIPScale, cmd.Flags().Lookup(FlagBidPriceIPScale)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagBidPriceScriptPath, "", "path to script to run for computing bid price") if err := viper.BindPFlag(FlagBidPriceScriptPath, cmd.Flags().Lookup(FlagBidPriceScriptPath)); err != nil { - return nil + panic(err) } cmd.Flags().Uint(FlagBidPriceScriptProcessLimit, 32, "limit to the number of scripts run concurrently for bid pricing") if err := viper.BindPFlag(FlagBidPriceScriptProcessLimit, cmd.Flags().Lookup(FlagBidPriceScriptProcessLimit)); err != nil { - return nil + panic(err) } cmd.Flags().Duration(FlagBidPriceScriptTimeout, time.Second*10, "execution timelimit for bid pricing as a duration") if err := viper.BindPFlag(FlagBidPriceScriptTimeout, cmd.Flags().Lookup(FlagBidPriceScriptTimeout)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagBidDeposit, cfg.BidDeposit.String(), "Bid deposit amount") if err := viper.BindPFlag(FlagBidDeposit, cmd.Flags().Lookup(FlagBidDeposit)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagClusterPublicHostname, "", "The public IP of the Kubernetes cluster") if err := viper.BindPFlag(FlagClusterPublicHostname, cmd.Flags().Lookup(FlagClusterPublicHostname)); err != nil { - return nil + panic(err) } cmd.Flags().Uint(FlagClusterNodePortQuantity, 1, "The number of node ports available on the Kubernetes cluster") if err := viper.BindPFlag(FlagClusterNodePortQuantity, cmd.Flags().Lookup(FlagClusterNodePortQuantity)); err != nil { - return nil + panic(err) } cmd.Flags().Duration(FlagClusterWaitReadyDuration, time.Second*5, "The time to wait for the cluster to be available") if err := viper.BindPFlag(FlagClusterWaitReadyDuration, cmd.Flags().Lookup(FlagClusterWaitReadyDuration)); err != nil { - return nil + panic(err) } cmd.Flags().Duration(FlagInventoryResourcePollPeriod, time.Second*5, "The period to poll the cluster inventory") if err := viper.BindPFlag(FlagInventoryResourcePollPeriod, cmd.Flags().Lookup(FlagInventoryResourcePollPeriod)); err != nil { - return nil + panic(err) } cmd.Flags().Uint(FlagInventoryResourceDebugFrequency, 10, "The rate at which to log all inventory resources") if err := viper.BindPFlag(FlagInventoryResourceDebugFrequency, cmd.Flags().Lookup(FlagInventoryResourceDebugFrequency)); err != nil { - return nil + panic(err) } cmd.Flags().Bool(FlagDeploymentIngressStaticHosts, false, "") if err := viper.BindPFlag(FlagDeploymentIngressStaticHosts, cmd.Flags().Lookup(FlagDeploymentIngressStaticHosts)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagDeploymentIngressDomain, "", "") if err := viper.BindPFlag(FlagDeploymentIngressDomain, cmd.Flags().Lookup(FlagDeploymentIngressDomain)); err != nil { - return nil + panic(err) } cmd.Flags().Bool(FlagDeploymentIngressExposeLBHosts, false, "") if err := viper.BindPFlag(FlagDeploymentIngressExposeLBHosts, cmd.Flags().Lookup(FlagDeploymentIngressExposeLBHosts)); err != nil { - return nil + panic(err) } cmd.Flags().Bool(FlagDeploymentNetworkPoliciesEnabled, true, "Enable network policies") if err := viper.BindPFlag(FlagDeploymentNetworkPoliciesEnabled, cmd.Flags().Lookup(FlagDeploymentNetworkPoliciesEnabled)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagDockerImagePullSecretsName, "", "Name of the local image pull secret configured with kubectl") if err := viper.BindPFlag(FlagDockerImagePullSecretsName, cmd.Flags().Lookup(FlagDockerImagePullSecretsName)); err != nil { - return nil + panic(err) } cmd.Flags().Uint64(FlagOvercommitPercentMemory, 0, "Percentage of memory overcommit") if err := viper.BindPFlag(FlagOvercommitPercentMemory, cmd.Flags().Lookup(FlagOvercommitPercentMemory)); err != nil { - return nil + panic(err) } cmd.Flags().Uint64(FlagOvercommitPercentCPU, 0, "Percentage of CPU overcommit") if err := viper.BindPFlag(FlagOvercommitPercentCPU, cmd.Flags().Lookup(FlagOvercommitPercentCPU)); err != nil { - return nil + panic(err) } cmd.Flags().Uint64(FlagOvercommitPercentStorage, 0, "Percentage of storage overcommit") if err := viper.BindPFlag(FlagOvercommitPercentStorage, cmd.Flags().Lookup(FlagOvercommitPercentStorage)); err != nil { - return nil + panic(err) } cmd.Flags().StringSlice(FlagDeploymentBlockedHostnames, nil, "hostnames blocked for deployments") if err := viper.BindPFlag(FlagDeploymentBlockedHostnames, cmd.Flags().Lookup(FlagDeploymentBlockedHostnames)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagAuthPem, "", "") if err := providerflags.AddKubeConfigPathFlag(cmd); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagDeploymentRuntimeClass, "gvisor", "kubernetes runtime class for deployments, use none for no specification") if err := viper.BindPFlag(FlagDeploymentRuntimeClass, cmd.Flags().Lookup(FlagDeploymentRuntimeClass)); err != nil { - return nil + panic(err) } cmd.Flags().Duration(FlagBidTimeout, 5*time.Minute, "time after which bids are cancelled if no lease is created") if err := viper.BindPFlag(FlagBidTimeout, cmd.Flags().Lookup(FlagBidTimeout)); err != nil { - return nil + panic(err) } cmd.Flags().Duration(FlagManifestTimeout, 5*time.Minute, "time after which bids are cancelled if no manifest is received") if err := viper.BindPFlag(FlagManifestTimeout, cmd.Flags().Lookup(FlagManifestTimeout)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagMetricsListener, "", "ip and port to start the metrics listener on") if err := viper.BindPFlag(FlagMetricsListener, cmd.Flags().Lookup(FlagMetricsListener)); err != nil { - return nil + panic(err) } cmd.Flags().Duration(FlagWithdrawalPeriod, time.Hour*24, "period at which withdrawals are made from the escrow accounts") if err := viper.BindPFlag(FlagWithdrawalPeriod, cmd.Flags().Lookup(FlagWithdrawalPeriod)); err != nil { - return nil + panic(err) } cmd.Flags().Duration(FlagLeaseFundsMonitorInterval, time.Minute*10, "interval at which lease is checked for funds available on the escrow accounts. >= 1m") if err := viper.BindPFlag(FlagLeaseFundsMonitorInterval, cmd.Flags().Lookup(FlagLeaseFundsMonitorInterval)); err != nil { - return nil + panic(err) } cmd.Flags().Uint64(FlagMinimumBalance, mparams.DefaultBidMinDeposit.Amount.Mul(sdk.NewIntFromUint64(2)).Uint64(), "minimum account balance at which withdrawal is started") if err := viper.BindPFlag(FlagMinimumBalance, cmd.Flags().Lookup(FlagMinimumBalance)); err != nil { - return nil + panic(err) } cmd.Flags().String(FlagProviderConfig, "", "provider configuration file path") if err := viper.BindPFlag(FlagProviderConfig, cmd.Flags().Lookup(FlagProviderConfig)); err != nil { - return nil + panic(err) } cmd.Flags().Duration(FlagRPCQueryTimeout, time.Minute, "timeout for requests made to the RPC node") if err := viper.BindPFlag(FlagRPCQueryTimeout, cmd.Flags().Lookup(FlagRPCQueryTimeout)); err != nil { - return nil + panic(err) } cmd.Flags().Duration(FlagCachedResultMaxAge, 5*time.Second, "max. cache age for results from the RPC node") if err := viper.BindPFlag(FlagCachedResultMaxAge, cmd.Flags().Lookup(FlagCachedResultMaxAge)); err != nil { - return nil + panic(err) } cmd.Flags().Bool(FlagEnableIPOperator, false, "enable usage of the IP operator to lease IP addresses") if err := viper.BindPFlag(FlagEnableIPOperator, cmd.Flags().Lookup(FlagEnableIPOperator)); err != nil { - return nil + panic(err) } cmd.Flags().Duration(FlagTxBroadcastTimeout, 30*time.Second, "tx broadcast timeout. defaults to 30s") if err := viper.BindPFlag(FlagTxBroadcastTimeout, cmd.Flags().Lookup(FlagTxBroadcastTimeout)); err != nil { - return nil + panic(err) } if err := providerflags.AddServiceEndpointFlag(cmd, serviceHostnameOperator); err != nil { - return nil + panic(err) } if err := providerflags.AddServiceEndpointFlag(cmd, serviceIPOperator); err != nil { - return nil + panic(err) } return cmd diff --git a/cmd/provider-services/main.go b/cmd/provider-services/main.go index 66254bf1..2e5d81f5 100644 --- a/cmd/provider-services/main.go +++ b/cmd/provider-services/main.go @@ -1,7 +1,10 @@ package main import ( + "context" + "errors" "os" + "os/signal" "github.com/cosmos/cosmos-sdk/server" @@ -10,16 +13,19 @@ import ( pcmd "github.com/akash-network/provider/cmd/provider-services/cmd" ) -// In main we call the rootCmd func main() { + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt) + defer stop() + rootCmd := pcmd.NewRootCmd() - if err := acmd.Execute(rootCmd, "AP"); err != nil { - switch e := err.(type) { - case server.ErrorCode: - os.Exit(e.Code) - default: - os.Exit(1) + err := acmd.ExecuteWithCtx(ctx, rootCmd, "AP") + + if err != nil { + if errors.As(err, &server.ErrorCode{}) { + os.Exit(err.(server.ErrorCode).Code) } + + os.Exit(1) } } diff --git a/gateway/rest/router.go b/gateway/rest/router.go index a5f8116e..5667881d 100644 --- a/gateway/rest/router.go +++ b/gateway/rest/router.go @@ -7,7 +7,6 @@ import ( "encoding/json" "fmt" "io" - "io/ioutil" "net/http" "net/http/httputil" "net/url" @@ -32,6 +31,7 @@ import ( manifestValidation "github.com/akash-network/akash-api/go/manifest/v2beta2" dtypes "github.com/akash-network/akash-api/go/node/deployment/v1beta3" mtypes "github.com/akash-network/akash-api/go/node/market/v1beta4" + "github.com/akash-network/node/util/wsutil" "github.com/akash-network/provider" @@ -44,8 +44,6 @@ import ( "github.com/akash-network/provider/gateway/utils" pmanifest "github.com/akash-network/provider/manifest" ipoptypes "github.com/akash-network/provider/operator/ipoperator/types" - - v1 "github.com/akash-network/akash-api/go/inventory/v1" ) type CtxAuthKey string @@ -533,49 +531,53 @@ func createStatusHandler(log log.Logger, sclient provider.StatusClient, provider } } -func createFeaturesHandler(log log.Logger, sclient provider.StatusClient, providerAddr sdk.Address) http.HandlerFunc { +func createFeaturesHandler(log log.Logger, _ provider.StatusClient, _ sdk.Address) http.HandlerFunc { return func(w http.ResponseWriter, req *http.Request) { - - // URLs slice and use in range allows execution in both dev and prod - urls := []string{ - "http://inventory-operator.akash-services.svc.cluster.local:8081/getClusterState", - "http://localhost:8081/getClusterState", - } - - var resp *http.Response - var err error - for _, url := range urls { - resp, err = http.Get(url) - if err != nil { - fmt.Printf("Failed to get '%s': %v\n", url, err) - continue - } - defer resp.Body.Close() - break - } - - if err != nil { - fmt.Printf("All attempts failed: %v\n", err) - return - } - - defer resp.Body.Close() - - bodyBytes, err := ioutil.ReadAll(resp.Body) - if err != nil { - fmt.Println(err) - } - - var clusterState v1.Cluster - err = json.Unmarshal(bodyBytes, &clusterState) - if err != nil { - fmt.Println(err) - } - - fmt.Println("clusterState: ", clusterState) - - writeJSON(log, w, clusterState) - + // // URLs slice and use in range allows execution in both dev and prod + // urls := []string{ + // "http://inventory-operator.akash-services.svc.cluster.local:8081/getClusterState", + // "http://localhost:8081/getClusterState", + // } + // + // var resp *http.Response + // var err error + // for _, u := range urls { + // resp, err = http.Get(u) + // if err != nil { + // fmt.Printf("Failed to get '%s': %v\n", u, err) + // continue + // } + // + // break + // } + // + // defer func() { + // _ = resp.Body.Close() + // }() + // + // if err != nil { + // fmt.Printf("All attempts failed: %v\n", err) + // return + // } + // + // defer func() { + // _ = resp.Body.Close() + // }() + // + // bodyBytes, err := io.ReadAll(resp.Body) + // if err != nil { + // fmt.Println(err) + // } + // + // var clusterState inventory.Cluster + // err = json.Unmarshal(bodyBytes, &clusterState) + // if err != nil { + // fmt.Println(err) + // } + // + // fmt.Println("clusterState: ", clusterState) + // + // writeJSON(log, w, clusterState) } } diff --git a/go.mod b/go.mod index 5cca9698..88f03be8 100644 --- a/go.mod +++ b/go.mod @@ -13,25 +13,26 @@ require ( github.com/go-logr/logr v1.2.4 github.com/go-logr/zapr v1.2.4 github.com/golang-jwt/jwt/v4 v4.5.0 - github.com/google/uuid v1.3.0 + github.com/google/uuid v1.4.0 github.com/gorilla/context v1.1.1 github.com/gorilla/mux v1.8.0 github.com/gorilla/websocket v1.5.0 + github.com/jaypipes/ghw v0.12.0 github.com/moby/term v0.5.0 github.com/pkg/errors v0.9.1 github.com/prometheus/client_golang v1.16.0 github.com/prometheus/common v0.44.0 github.com/rook/rook v1.11.1 github.com/shopspring/decimal v1.3.1 - github.com/spf13/cobra v1.7.0 + github.com/spf13/cobra v1.8.0 github.com/spf13/pflag v1.0.5 - github.com/spf13/viper v1.16.0 + github.com/spf13/viper v1.18.2 github.com/stretchr/testify v1.8.4 github.com/tendermint/tendermint v0.34.27 go.uber.org/zap v1.24.0 - golang.org/x/net v0.14.0 - golang.org/x/sync v0.3.0 - google.golang.org/grpc v1.57.0 + golang.org/x/net v0.19.0 + golang.org/x/sync v0.5.0 + google.golang.org/grpc v1.59.0 gopkg.in/yaml.v3 v3.0.1 k8s.io/api v0.26.1 k8s.io/apimachinery v0.26.1 @@ -42,24 +43,6 @@ require ( sigs.k8s.io/kind v0.20.0 ) -replace ( - // use cosmos fork of keyring - github.com/99designs/keyring => github.com/cosmos/keyring v1.2.0 - - github.com/cosmos/ledger-cosmos-go => github.com/akash-network/ledger-go/cosmos v0.14.3 - - // Fix upstream GHSA-h395-qcrw-5vmq vulnerability. - // TODO Remove it: https://github.com/cosmos/cosmos-sdk/issues/10409 - github.com/gin-gonic/gin => github.com/gin-gonic/gin v1.8.1 - - github.com/gogo/protobuf => github.com/regen-network/protobuf v1.3.3-alpha.regen.1 - github.com/tendermint/tendermint => github.com/akash-network/cometbft v0.34.27-akash - - github.com/zondax/hid => github.com/troian/hid v0.13.2 - - github.com/zondax/ledger-go => github.com/akash-network/ledger-go v0.14.3 -) - require ( cosmossdk.io/api v0.2.6 // indirect cosmossdk.io/core v0.5.1 // indirect @@ -72,6 +55,7 @@ require ( github.com/ChainSafe/go-schnorrkel v0.0.0-20200405005733-88cbf1b4c40d // indirect github.com/DataDog/zstd v1.5.0 // indirect github.com/HdrHistogram/hdrhistogram-go v1.1.2 // indirect + github.com/StackExchange/wmi v1.2.1 // indirect github.com/Workiva/go-datastructures v1.0.53 // indirect github.com/alessio/shellescape v1.4.1 // indirect github.com/armon/go-metrics v0.4.1 // indirect @@ -114,12 +98,14 @@ require ( github.com/emicklei/go-restful/v3 v3.10.0 // indirect github.com/evanphx/json-patch v5.6.0+incompatible // indirect github.com/evanphx/json-patch/v5 v5.6.0 // indirect - github.com/fatih/color v1.13.0 // indirect + github.com/fatih/color v1.14.1 // indirect github.com/felixge/httpsnoop v1.0.2 // indirect - github.com/fsnotify/fsnotify v1.6.0 // indirect + github.com/fsnotify/fsnotify v1.7.0 // indirect github.com/getsentry/sentry-go v0.17.0 // indirect + github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32 // indirect github.com/go-kit/log v0.2.1 // indirect github.com/go-logfmt/logfmt v0.5.1 // indirect + github.com/go-ole/go-ole v1.2.6 // indirect github.com/go-openapi/jsonpointer v0.19.5 // indirect github.com/go-openapi/jsonreference v0.20.0 // indirect github.com/go-openapi/swag v0.22.3 // indirect @@ -145,7 +131,7 @@ require ( github.com/gtank/ristretto255 v0.1.2 // indirect github.com/hashicorp/errwrap v1.1.0 // indirect github.com/hashicorp/go-cleanhttp v0.5.2 // indirect - github.com/hashicorp/go-hclog v1.3.1 // indirect + github.com/hashicorp/go-hclog v1.5.0 // indirect github.com/hashicorp/go-immutable-radix v1.3.1 // indirect github.com/hashicorp/go-multierror v1.1.1 // indirect github.com/hashicorp/go-plugin v1.4.5 // indirect @@ -168,10 +154,11 @@ require ( github.com/imdario/mergo v0.3.13 // indirect github.com/improbable-eng/grpc-web v0.14.1 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect + github.com/jaypipes/pcidb v1.0.0 // indirect github.com/jmhodges/levigo v1.0.1-0.20191019112844-b572e7f4cdac // indirect github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/klauspost/compress v1.15.11 // indirect + github.com/klauspost/compress v1.17.0 // indirect github.com/kr/pretty v0.3.1 // indirect github.com/kr/text v0.2.0 // indirect github.com/kube-object-storage/lib-bucket-provisioner v0.0.0-20221122204822-d1a8c34382f1 // indirect @@ -182,7 +169,7 @@ require ( github.com/magiconair/properties v1.8.7 // indirect github.com/mailru/easyjson v0.7.7 // indirect github.com/mattn/go-colorable v0.1.13 // indirect - github.com/mattn/go-isatty v0.0.16 // indirect + github.com/mattn/go-isatty v0.0.17 // indirect github.com/mattn/go-runewidth v0.0.13 // indirect github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect github.com/mimoo/StrobeGo v0.0.0-20210601165009-122bf33a46e0 // indirect @@ -201,7 +188,7 @@ require ( github.com/oklog/run v1.1.0 // indirect github.com/openshift/api v0.0.0-20210105115604-44119421ec6b // indirect github.com/pelletier/go-toml v1.9.5 // indirect - github.com/pelletier/go-toml/v2 v2.0.8 // indirect + github.com/pelletier/go-toml/v2 v2.1.0 // indirect github.com/petermattis/goid v0.0.0-20180202154549-b0b1615b78e5 // indirect github.com/pierrec/lz4 v2.6.1+incompatible // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect @@ -215,13 +202,15 @@ require ( github.com/rs/cors v1.8.2 // indirect github.com/rs/zerolog v1.30.0 // indirect github.com/ryanuber/go-glob v1.0.0 // indirect + github.com/sagikazarmark/locafero v0.4.0 // indirect + github.com/sagikazarmark/slog-shim v0.1.0 // indirect github.com/sasha-s/go-deadlock v0.3.1 // indirect github.com/sirupsen/logrus v1.9.0 // indirect - github.com/spf13/afero v1.9.5 // indirect - github.com/spf13/cast v1.5.1 // indirect - github.com/spf13/jwalterweatherman v1.1.0 // indirect + github.com/sourcegraph/conc v0.3.0 // indirect + github.com/spf13/afero v1.11.0 // indirect + github.com/spf13/cast v1.6.0 // indirect github.com/stretchr/objx v0.5.0 // indirect - github.com/subosito/gotenv v1.4.2 // indirect + github.com/subosito/gotenv v1.6.0 // indirect github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect github.com/tecbot/gorocksdb v0.0.0-20191217155057-f0fad39f321c // indirect github.com/tendermint/go-amino v0.16.0 // indirect @@ -233,21 +222,21 @@ require ( go.etcd.io/bbolt v1.3.6 // indirect go.step.sm/crypto v0.34.0 // indirect go.uber.org/atomic v1.10.0 // indirect - go.uber.org/multierr v1.8.0 // indirect - golang.org/x/crypto v0.12.0 // indirect - golang.org/x/exp v0.0.0-20221019170559-20944726eadf // indirect + go.uber.org/multierr v1.9.0 // indirect + golang.org/x/crypto v0.16.0 // indirect + golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect golang.org/x/mod v0.12.0 // indirect - golang.org/x/oauth2 v0.11.0 // indirect - golang.org/x/sys v0.11.0 // indirect - golang.org/x/term v0.11.0 // indirect - golang.org/x/text v0.12.0 // indirect - golang.org/x/time v0.3.0 // indirect - golang.org/x/tools v0.6.0 // indirect + golang.org/x/oauth2 v0.15.0 // indirect + golang.org/x/sys v0.15.0 // indirect + golang.org/x/term v0.15.0 // indirect + golang.org/x/text v0.14.0 // indirect + golang.org/x/time v0.5.0 // indirect + golang.org/x/tools v0.13.0 // indirect gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect google.golang.org/appengine v1.6.7 // indirect - google.golang.org/genproto v0.0.0-20230706204954-ccb25ca9f130 // indirect - google.golang.org/genproto/googleapis/api v0.0.0-20230706204954-ccb25ca9f130 // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20230720185612-659f7aaaa771 // indirect + google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17 // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20231120223509-83a465c0220f // indirect google.golang.org/protobuf v1.31.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/ini.v1 v1.67.0 // indirect @@ -264,6 +253,24 @@ require ( sigs.k8s.io/yaml v1.3.0 // indirect ) +replace ( + // use cosmos fork of keyring + github.com/99designs/keyring => github.com/cosmos/keyring v1.2.0 + + github.com/cosmos/ledger-cosmos-go => github.com/akash-network/ledger-go/cosmos v0.14.3 + + // Fix upstream GHSA-h395-qcrw-5vmq vulnerability. + // TODO Remove it: https://github.com/cosmos/cosmos-sdk/issues/10409 + github.com/gin-gonic/gin => github.com/gin-gonic/gin v1.8.1 + + github.com/gogo/protobuf => github.com/regen-network/protobuf v1.3.3-alpha.regen.1 + github.com/tendermint/tendermint => github.com/akash-network/cometbft v0.34.27-akash + + github.com/zondax/hid => github.com/troian/hid v0.13.2 + + github.com/zondax/ledger-go => github.com/akash-network/ledger-go v0.14.3 +) + // these replaces required for rook to work replace ( github.com/googleapis/gnostic => github.com/googleapis/gnostic v0.4.1 diff --git a/go.sum b/go.sum index b3cf62aa..fb345d46 100644 --- a/go.sum +++ b/go.sum @@ -6,7 +6,6 @@ cloud.google.com/go v0.39.0/go.mod h1:rVLT6fkc8chs9sfPtFc1SBH6em7n+ZoXaG+87tDISt cloud.google.com/go v0.43.0/go.mod h1:BOSR3VbTLkk6FDC/TcffxP4NF/FFBGA5ku+jvKOP7pg= cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU= cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY= -cloud.google.com/go v0.44.3/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY= cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc= cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0= cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To= @@ -20,11 +19,10 @@ cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOY cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY= cloud.google.com/go v0.72.0/go.mod h1:M+5Vjvlc2wnp6tjzE102Dw08nGShTscUx2nZMufOKPI= cloud.google.com/go v0.74.0/go.mod h1:VV1xSbzvo+9QJOxLDaJfTjx5e+MePCpCWwvftOeQmWk= -cloud.google.com/go v0.75.0/go.mod h1:VGuuCn7PG0dwsd5XPVm2Mm3wlh3EL55/79EKB6hlPTY= cloud.google.com/go v0.78.0/go.mod h1:QjdrLG0uq+YwhjoVOLsS1t7TW8fs36kLs4XO5R5ECHg= cloud.google.com/go v0.79.0/go.mod h1:3bzgcEeQlzbuEAYu4mrWhKqWjmpprinYgKJLgKHnbb8= cloud.google.com/go v0.81.0/go.mod h1:mk/AM35KwGk/Nm2YSeZbxXdrNK3KZOYHmLkOqC2V6E0= -cloud.google.com/go v0.110.4 h1:1JYyxKMN9hd5dR2MYTPWkGUgcoxVVhg0LKNKEo0qvmk= +cloud.google.com/go v0.110.10 h1:LXy9GEO+timppncPIAZoOj3l58LIU9k+kn48AN7IO3Y= cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE= cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc= @@ -32,19 +30,19 @@ cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUM cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc= cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ= cloud.google.com/go/bigtable v1.2.0/go.mod h1:JcVAOl45lrTmQfLj7T6TxyMzIN/3FGGcFm+2xVAli2o= -cloud.google.com/go/compute v1.20.1 h1:6aKEtlUiwEpJzM001l0yFkpXmUVXaN8W+fbkb2AZNbg= -cloud.google.com/go/compute v1.20.1/go.mod h1:4tCnrn48xsqlwSAiLf1HXMQk8CONslYbdiEZc9FEIbM= +cloud.google.com/go/compute v1.23.3 h1:6sVlXXBmbd7jNX0Ipq0trII3e4n1/MsADLK6a+aiVlk= +cloud.google.com/go/compute v1.23.3/go.mod h1:VCgBUoMnIVIR0CscqQiPJLAG25E3ZRZMzcFZeQ+h8CI= cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY= cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA= cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk= cloud.google.com/go/firestore v1.1.0/go.mod h1:ulACoGHTpvq5r8rxGJ4ddJZBZqakUQqClKRT5SZwBmk= -cloud.google.com/go/iam v1.1.0 h1:67gSqaPukx7O8WLLHMa0PNs3EBGd2eE4d+psbO/CO94= -cloud.google.com/go/iam v1.1.0/go.mod h1:nxdHjaKfCr7fNYx/HJMM8LgiMugmveWlkatear5gVyk= -cloud.google.com/go/kms v1.15.0 h1:xYl5WEaSekKYN5gGRyhjvZKM22GVBBCzegGNVPy+aIs= -cloud.google.com/go/kms v1.15.0/go.mod h1:c9J991h5DTl+kg7gi3MYomh12YEENGrf48ee/N/2CDM= -cloud.google.com/go/monitoring v1.15.1 h1:65JhLMd+JiYnXr6j5Z63dUYCuOg770p8a/VC+gil/58= -cloud.google.com/go/monitoring v1.15.1/go.mod h1:lADlSAlFdbqQuwwpaImhsJXu1QSdd3ojypXrFSMr2rM= +cloud.google.com/go/iam v1.1.5 h1:1jTsCu4bcsNsE4iiqNT5SHwrDRCfRmIaaaVFhRveTJI= +cloud.google.com/go/iam v1.1.5/go.mod h1:rB6P/Ic3mykPbFio+vo7403drjlgvoWfYpJhMXEbzv8= +cloud.google.com/go/kms v1.15.5 h1:pj1sRfut2eRbD9pFRjNnPNg/CzJPuQAzUujMIM1vVeM= +cloud.google.com/go/kms v1.15.5/go.mod h1:cU2H5jnp6G2TDpUGZyqTCoy1n16fbubHZjmVXSMtwDI= +cloud.google.com/go/monitoring v1.16.3 h1:mf2SN9qSoBtIgiMA4R/y4VADPWZA7VCNJA079qLaZQ8= +cloud.google.com/go/monitoring v1.16.3/go.mod h1:KwSsX5+8PnXv5NJnICZzW2R8pWTis8ypC4zmdRD63Tw= cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw= cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA= @@ -54,7 +52,6 @@ cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0Zeo cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk= cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs= cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0= -cloud.google.com/go/storage v1.14.0/go.mod h1:GrKmX003DSIwi9o29oFT7YDnHYwZoctc3fOKtUw0Xmo= code.cloudfoundry.org/gofileutils v0.0.0-20170111115228-4d0c80011a0f/go.mod h1:sk5LnIjB/nIEU7yP5sDQExVm62wu0pBh3yrElngUisI= collectd.org v0.3.0/go.mod h1:A/8DzQBkF6abtvrT2j/AU/4tiBgJWYyh0y/oB/4MlWE= cosmossdk.io/api v0.2.6 h1:AoNwaLLapcLsphhMK6+o0kZl+D6MMUaHVqSdwinASGU= @@ -184,6 +181,8 @@ github.com/Shopify/goreferrer v0.0.0-20181106222321-ec9c9a553398/go.mod h1:a1uqR github.com/Shopify/sarama v1.19.0/go.mod h1:FVkBWblsNy7DGZRfXLU0O9RCGt5g3g3yEuWXgklEdEo= github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI= github.com/StackExchange/wmi v0.0.0-20180116203802-5d049714c4a6/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg= +github.com/StackExchange/wmi v1.2.1 h1:VIkavFPXSjcnS+O8yTq7NI32k0R5Aj+v39y29VYDOSA= +github.com/StackExchange/wmi v1.2.1/go.mod h1:rcmrprowKIVzvc+NUiLncP2uuArMWLCbu9SBzvHz7e8= github.com/VictoriaMetrics/fastcache v1.6.0/go.mod h1:0qHz5QP0GMX4pfmMA/zt5RgfNuXJrTP0zS7DqpHGGTw= github.com/VividCortex/gohistogram v1.0.0 h1:6+hBz+qvs0JOrrNhhmR7lFxo5sINxBCGXrdtl/UvroE= github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g= @@ -440,7 +439,7 @@ github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwc github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/cpuguy83/go-md2man/v2 v2.0.1/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= -github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= +github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= github.com/creachadair/taskgroup v0.3.2 h1:zlfutDS+5XG40AOxcHDSThxKzns8Tnr9jnr6VqkYlkM= github.com/creachadair/taskgroup v0.3.2/go.mod h1:wieWwecHVzsidg2CsUnFinW1faVN4+kq+TDlRJQ0Wbk= github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY= @@ -576,8 +575,9 @@ github.com/fasthttp-contrib/websocket v0.0.0-20160511215533-1f3b11f56072/go.mod github.com/fatih/camelcase v1.0.0/go.mod h1:yN2Sb0lFhZJUdVvtELVWefmrXpuZESvPmqwoZc+/fpc= github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU= -github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w= github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk= +github.com/fatih/color v1.14.1 h1:qfhVLaG5s+nCROl1zJsZRxFeYrHLqWroPOQ8BWiNb4w= +github.com/fatih/color v1.14.1/go.mod h1:2oHN61fhTpgcxD3TSWCgKDiH1+x4OiDVVGH8WlgGZGg= github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo= github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M= github.com/felixge/httpsnoop v1.0.1/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= @@ -597,12 +597,12 @@ github.com/frankban/quicktest v1.4.0/go.mod h1:36zfPVQyHxymz4cH7wlDmVwDrJuljRB60 github.com/frankban/quicktest v1.4.1/go.mod h1:36zfPVQyHxymz4cH7wlDmVwDrJuljRB60qkgn7rorfQ= github.com/frankban/quicktest v1.10.0/go.mod h1:ui7WezCLWMWxVWr1GETZY3smRy0G4KWq9vcPtJmFl7Y= github.com/frankban/quicktest v1.13.0/go.mod h1:qLE0fzW0VuyUAJgPU19zByoIr0HtCHN/r/VLSOOIySU= -github.com/frankban/quicktest v1.14.4 h1:g2rn0vABPOOXmZUj+vbmUp0lPoXEMuhTpIluN0XL9UY= -github.com/frankban/quicktest v1.14.4/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0= +github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= +github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= -github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY= -github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw= +github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= +github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= github.com/fullsailor/pkcs7 v0.0.0-20190404230743-d7302db945fa/go.mod h1:KnogPXtdwXqoenmZCw6S+25EAm2MkxbG0deNDu4cbSA= github.com/fvbommel/sortorder v1.0.1/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui720w+kxuqRs0= github.com/gammazero/deque v0.0.0-20190130191400-2afb3858e9c7/go.mod h1:GeIq9qoE43YdGnDXURnmKTnGg15pQz4mYkXSTChbneI= @@ -618,6 +618,7 @@ github.com/getsentry/sentry-go v0.17.0/go.mod h1:B82dxtBvxG0KaPD8/hfSV+VcHD+Lg/x github.com/ghemawat/stream v0.0.0-20171120220530-696b145b53b9/go.mod h1:106OIgooyS7OzLDOpUGgm9fA3bQENb/cFSyyBmMoJDs= github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= +github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32 h1:Mn26/9ZMNWSw9C9ERFA1PUxfmGpolnw2v0bKOREu5ew= github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32/go.mod h1:GIjDIg/heH5DOkXY3YJ/wNhfHsQHoXGjl8G8amsYQ1I= github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE= github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI= @@ -664,6 +665,7 @@ github.com/go-logr/zapr v1.2.4 h1:QHVo+6stLbfJmYGkQ7uGHUCu5hnAFAj6mDe6Ea0SeOo= github.com/go-logr/zapr v1.2.4/go.mod h1:FyHWQIzQORZ0QVE1BtVHv3cKtNLuXsbNLtpuhNapBOA= github.com/go-martini/martini v0.0.0-20170121215854-22fa46961aab/go.mod h1:/P9AEU963A2AYjv4d1V5eVL1CQbEJq6aCNHDDjibzu8= github.com/go-ole/go-ole v1.2.1/go.mod h1:7FAglXiTm7HKlQRDeOQ6ZNUHidzCWXuZWq/1dTyBNF8= +github.com/go-ole/go-ole v1.2.5/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY= github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= github.com/go-openapi/analysis v0.0.0-20180825180245-b006789cd277/go.mod h1:k70tL6pCuVxPJOHXQ+wIac1FUrvNkHolPie/cLEU6hI= @@ -883,12 +885,11 @@ github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hf github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= -github.com/google/pprof v0.0.0-20201218002935-b9804c9f04c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= -github.com/google/s2a-go v0.1.4 h1:1kZ/sQM3srePvKs3tXAvQzo66XfcReoqFpIpIccE7Oc= -github.com/google/s2a-go v0.1.4/go.mod h1:Ej+mSEMGRnqRzjc7VtF+jdBwYG5fuJfiZ8ELkjEwM0A= +github.com/google/s2a-go v0.1.7 h1:60BLSyTrOV4/haCDW4zb1guZItoSq8foHCXrAnjBo/o= +github.com/google/s2a-go v0.1.7/go.mod h1:50CgR4k1jNlWBu4UfS4AcfhVe1r6pdZPygJ3R8F0Qdw= github.com/google/safetext v0.0.0-20220905092116-b49f7bc46da2 h1:SJ+NtwL6QaZ21U+IrK7d0gGgpjGGvd2kz+FzTHVzdqI= github.com/google/safetext v0.0.0-20220905092116-b49f7bc46da2/go.mod h1:Tv1PlzqC9t8wNnpPdctvtSUOPUUg4SHeE6vR1Ir2hmg= github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ= @@ -896,16 +897,15 @@ github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+ github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= -github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/googleapis/enterprise-certificate-proxy v0.2.5 h1:UR4rDjcgpgEnqpIEvkiqTYKBCKLNmlge2eVjoZfySzM= -github.com/googleapis/enterprise-certificate-proxy v0.2.5/go.mod h1:RxW0N9901Cko1VOCW3SXCpWP+mlIEkk2tP7jnHy9a3w= +github.com/google/uuid v1.4.0 h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4= +github.com/google/uuid v1.4.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/enterprise-certificate-proxy v0.3.2 h1:Vie5ybvEvT75RniqhfFxPRy3Bf7vr3h0cechB90XaQs= +github.com/googleapis/enterprise-certificate-proxy v0.3.2/go.mod h1:VLSiSSBs/ksPL8kq3OBOQ6WRI2QnaFynd1DCjZ62+V0= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/googleapis/gax-go/v2 v2.12.0 h1:A+gCJKdRfqXkr+BIRGtZLibNXf0m1f9E4HG56etFpas= github.com/googleapis/gax-go/v2 v2.12.0/go.mod h1:y+aIqrI5eb1YGMVJfuV3185Ts/D7qKpsEkdD5+I6QGU= github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg= -github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8/go.mod h1:dvDLG8qkwmyD9a/MJJN3XJcT3xFxOKAvTZGvuZmac9g= github.com/gophercloud/gophercloud v0.1.0 h1:P/nh25+rzXouhytV2pUHBb65fnds26Ghl8/391+sT5o= github.com/gophercloud/gophercloud v0.1.0/go.mod h1:vxM41WHh5uqHVBMZHzuwNOHh8XEoIEcSTewFxm1c5g8= github.com/gopherjs/gopherjs v0.0.0-20180628210949-0892b62f0d9f/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= @@ -982,8 +982,8 @@ github.com/hashicorp/go-hclog v0.10.1/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39 github.com/hashicorp/go-hclog v0.12.0/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ= github.com/hashicorp/go-hclog v0.14.1/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ= github.com/hashicorp/go-hclog v0.16.2/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ= -github.com/hashicorp/go-hclog v1.3.1 h1:vDwF1DFNZhntP4DAjuTpOw3uEgMUpXh1pB5fW9DqHpo= -github.com/hashicorp/go-hclog v1.3.1/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M= +github.com/hashicorp/go-hclog v1.5.0 h1:bI2ocEMgcVlz55Oj1xZNBsVi900c7II+fWDyV9o+13c= +github.com/hashicorp/go-hclog v1.5.0/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M= github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= github.com/hashicorp/go-immutable-radix v1.1.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= github.com/hashicorp/go-immutable-radix v1.3.1 h1:DKHmCUm2hRBK510BaiZlwvpD40f8bJFeZnpfm2KLowc= @@ -1210,6 +1210,10 @@ github.com/iris-contrib/schema v0.0.1/go.mod h1:urYA3uvUNG1TIIjOSCzHr9/LmbQo8LrO github.com/jackc/fake v0.0.0-20150926172116-812a484cc733/go.mod h1:WrMFNQdiFJ80sQsxDoMokWK1W5TQtxBFNpzWTD84ibQ= github.com/jackc/pgx v3.3.0+incompatible/go.mod h1:0ZGrqGqkRlliWnWB4zKnWtjbSWbGkVEFm4TeybAXq+I= github.com/jackpal/go-nat-pmp v1.0.2/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= +github.com/jaypipes/ghw v0.12.0 h1:xU2/MDJfWmBhJnujHY9qwXQLs3DBsf0/Xa9vECY0Tho= +github.com/jaypipes/ghw v0.12.0/go.mod h1:jeJGbkRB2lL3/gxYzNYzEDETV1ZJ56OKr+CSeSEym+g= +github.com/jaypipes/pcidb v1.0.0 h1:vtZIfkiCUE42oYbJS0TAq9XSfSmcsgo9IdxSm9qzYU8= +github.com/jaypipes/pcidb v1.0.0/go.mod h1:TnYUvqhPBzCKnH34KrIX22kAeEbDCSRJ9cqLRCuNDfk= github.com/jcmturner/aescts v1.0.1/go.mod h1:k9gJoDUf1GH5r2IBtBjwjDCoLELYxOcEhitdP8RL7qQ= github.com/jcmturner/dnsutils v1.0.1/go.mod h1:tqMo38L01jO8AKxT0S9OQVlGZu3dkEt+z5CA+LOhwB0= github.com/jcmturner/gofork v1.0.0/go.mod h1:MK8+TM0La+2rjBD4jE12Kj1pCCxK7d2LK/UM3ncEo0o= @@ -1291,8 +1295,8 @@ github.com/klauspost/compress v1.9.7/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0 github.com/klauspost/compress v1.10.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.11.7/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.12.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg= -github.com/klauspost/compress v1.15.11 h1:Lcadnb3RKGin4FYM/orgq0qde+nc15E5Cbqg4B9Sx9c= -github.com/klauspost/compress v1.15.11/go.mod h1:QPwzmACJjUTFsnSHH934V6woptycfrDDJnH7hvFVbGM= +github.com/klauspost/compress v1.17.0 h1:Rnbp4K9EjcDuVuHtd0dgA4qNuv9yKDYKK1ulpJwgrqM= +github.com/klauspost/compress v1.17.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/cpuid v0.0.0-20170728055534-ae7887de9fa5/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek= github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek= github.com/klauspost/cpuid v1.2.1/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek= @@ -1384,8 +1388,9 @@ github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcME github.com/mattn/go-isatty v0.0.11/go.mod h1:PhnuNfih5lzO57/f3n+odYbM4JtupLOxQOAqxQCu2WE= github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= -github.com/mattn/go-isatty v0.0.16 h1:bq3VjFmv/sOjHtdEhmkEV4x1AJtvUvOJ2PFAZ5+peKQ= github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= +github.com/mattn/go-isatty v0.0.17 h1:BTarxUcIeDqL27Mc+vyvdWYSL28zpIhv3RoTdsLMPng= +github.com/mattn/go-isatty v0.0.17/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= github.com/mattn/go-runewidth v0.0.3/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= github.com/mattn/go-runewidth v0.0.7/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI= @@ -1587,8 +1592,8 @@ github.com/pelletier/go-toml v1.9.4/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCko github.com/pelletier/go-toml v1.9.5 h1:4yBQzkHv+7BHq2PQUZF3Mx0IYxG7LsP222s7Agd3ve8= github.com/pelletier/go-toml v1.9.5/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c= github.com/pelletier/go-toml/v2 v2.0.1/go.mod h1:r9LEWfGN8R5k0VXJ+0BkIe7MYkRdwZOjgMj2KwnJFUo= -github.com/pelletier/go-toml/v2 v2.0.8 h1:0ctb6s9mE31h0/lhu+J6OPmVeDxJn+kYnJc2jZR9tGQ= -github.com/pelletier/go-toml/v2 v2.0.8/go.mod h1:vuYfssBdrU2XDZ9bYydBu6t+6a6PYNcZljzZR9VXg+4= +github.com/pelletier/go-toml/v2 v2.1.0 h1:FnwAJ4oYMvbT/34k9zzHuZNrhlz48GB3/s6at6/MHO4= +github.com/pelletier/go-toml/v2 v2.1.0/go.mod h1:tJU2Z3ZkXwnxa4DPO899bsyIoywizdUvyaeZurnPPDc= github.com/performancecopilot/speed v3.0.0+incompatible/go.mod h1:/CLtqpZ5gBg1M9iaPbIdPPGyKcA8hKdoy6hAWba7Yac= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= github.com/peterh/liner v1.0.1-0.20180619022028-8c1271fcf47f/go.mod h1:xIteQHvHuaLYG9IFj6mSxM0fCKrs34IrEQUhOYuGPHc= @@ -1615,7 +1620,6 @@ github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/profile v1.2.1/go.mod h1:hJw3o1OdXxsrSjjVksARp5W95eeEaEfptyVZyv6JUPA= github.com/pkg/sftp v1.10.1/go.mod h1:lYOWFsE0bwd1+KfKJaKeuokY15vzFx25BLbzYYoAxZI= -github.com/pkg/sftp v1.13.1/go.mod h1:3HaPG6Dq1ILlpPZRO0HVMrsydcdLt6HRDccSgb87qRg= github.com/pkg/term v0.0.0-20180730021639-bffc007b7fd5/go.mod h1:eCbImbZ95eXtAUIbLAuAVnBnwf83mjf6QIVH8SHYwqQ= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= @@ -1732,6 +1736,10 @@ github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb github.com/ryanuber/columnize v2.1.0+incompatible/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts= github.com/ryanuber/go-glob v1.0.0 h1:iQh3xXAumdQ+4Ufa5b25cRpC5TYKlno6hsv6Cb3pkBk= github.com/ryanuber/go-glob v1.0.0/go.mod h1:807d1WSdnB0XRJzKNil9Om6lcp/3a0v4qIHxIXzX/Yc= +github.com/sagikazarmark/locafero v0.4.0 h1:HApY1R9zGo4DBgr7dqsTH/JJxLTTsOt7u6keLGt6kNQ= +github.com/sagikazarmark/locafero v0.4.0/go.mod h1:Pe1W6UlPYUk/+wc/6KFhbORCfqzgYEpgQ3O5fPuL3H4= +github.com/sagikazarmark/slog-shim v0.1.0 h1:diDBnUNK9N/354PgrxMywXnAwEr1QZcOr6gto+ugjYE= +github.com/sagikazarmark/slog-shim v0.1.0/go.mod h1:SrcSrq8aKtyuqEI1uvTDTK1arOWRIczQRv+GVI1AkeQ= github.com/samuel/go-zookeeper v0.0.0-20180130194729-c4fab1ac1bec/go.mod h1:gi+0XIa01GRL2eRQVjQkKGqKF3SF9vZR/HnPullcV2E= github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da/go.mod h1:gi+0XIa01GRL2eRQVjQkKGqKF3SF9vZR/HnPullcV2E= github.com/sasha-s/go-deadlock v0.3.1 h1:sqv7fDNShgjcaxkO0JNcOAlr8B9+cV5Ey/OB71efZx0= @@ -1777,27 +1785,28 @@ github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4k github.com/sony/gobreaker v0.4.1/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY= github.com/sony/gobreaker v0.4.2-0.20210216022020-dd874f9dd33b h1:br+bPNZsJWKicw/5rALEo67QHs5weyD5tf8WST+4sJ0= github.com/sony/gobreaker v0.4.2-0.20210216022020-dd874f9dd33b/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY= +github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo= +github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI= github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= github.com/spf13/afero v1.6.0/go.mod h1:Ai8FlHk4v/PARR026UzYexafAt9roJ7LcLMAmO6Z93I= -github.com/spf13/afero v1.9.5 h1:stMpOSZFs//0Lv29HduCmli3GUfpFoF3Y1Q/aXj/wVM= -github.com/spf13/afero v1.9.5/go.mod h1:UBogFpq8E9Hx+xc5CNTTEpTnuHVmXDwZcZcE1eb/UhQ= +github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8= +github.com/spf13/afero v1.11.0/go.mod h1:GH9Y3pIexgf1MTIWtNGyogA5MwRIDXGUr+hbWNoBjkY= github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= -github.com/spf13/cast v1.5.1 h1:R+kOtfhWQE6TVQzY+4D7wJLBgkdVasCEFxSUBYBYIlA= -github.com/spf13/cast v1.5.1/go.mod h1:b9PdjNptOpzXr7Rq1q9gJML/2cdGQAo69NKzQ10KN48= +github.com/spf13/cast v1.6.0 h1:GEiTHELF+vaR5dhz3VqZfFSzZjYbgeKDpBxQVS4GYJ0= +github.com/spf13/cast v1.6.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo= github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ= github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU= github.com/spf13/cobra v1.1.1/go.mod h1:WnodtKOvamDL/PwE2M4iKs8aMDBZ5Q5klgD3qfVJQMI= github.com/spf13/cobra v1.2.1/go.mod h1:ExllRjgxM/piMAM+3tAZvg8fsklGAf3tPfi+i8t68Nk= github.com/spf13/cobra v1.4.0/go.mod h1:Wo4iy3BUC+X2Fybo0PDqwJIv3dNRiZLHQymsfxlB84g= -github.com/spf13/cobra v1.7.0 h1:hyqWnYt1ZQShIddO5kBpj3vu05/++x6tJ6dg8EC572I= -github.com/spf13/cobra v1.7.0/go.mod h1:uLxZILRyS/50WlhOIKD7W6V5bgeIt+4sICxh6uRMrb0= +github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0= +github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho= github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= -github.com/spf13/jwalterweatherman v1.1.0 h1:ue6voC5bR5F8YxI5S67j9i582FU4Qvo2bmqnqMYADFk= github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo= github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= github.com/spf13/pflag v1.0.1/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= @@ -1808,8 +1817,8 @@ github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s= github.com/spf13/viper v1.7.0/go.mod h1:8WkrPz2fc9jxqZNCJI/76HCieCp4Q8HaLFoCha5qpdg= github.com/spf13/viper v1.8.1/go.mod h1:o0Pch8wJ9BVSWGQMbra6iw0oQ5oktSIBaujf1rJH9Ns= -github.com/spf13/viper v1.16.0 h1:rGGH0XDZhdUOryiDWjmIvUSWpbNqisK8Wk0Vyefw8hc= -github.com/spf13/viper v1.16.0/go.mod h1:yg78JgCJcbrQOvV9YLXgkLaZqUidkY9K+Dd1FofRzQg= +github.com/spf13/viper v1.18.2 h1:LUXCnvUvSM6FXAsj6nnfc8Q2tp1dIgUfY9Kc8GsSOiQ= +github.com/spf13/viper v1.18.2/go.mod h1:EKmWIqdnk5lOcmR72yw6hS+8OPYcwD0jteitLMVB+yk= github.com/status-im/keycard-go v0.0.0-20190316090335-8537d3370df4/go.mod h1:RZLeN1LMWmRsyYjvAu+I6Dm9QmlDaIIt+Y+4Kd7Tp+Q= github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8= github.com/streadway/amqp v0.0.0-20190404075320-75d898a42a94/go.mod h1:AZpEONHx3DKn8O/DFsRAY58/XVQiIPMTMB1SddzLXVw= @@ -1831,12 +1840,11 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw= -github.com/subosito/gotenv v1.4.2 h1:X1TuBLAMDFbaTAChgCBLu3DU3UPyELpnF2jjJ2cz/S8= -github.com/subosito/gotenv v1.4.2/go.mod h1:ayKnFf/c6rvx/2iiLrJUk1e6plDbT3edrFNGqEflhK0= +github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8= +github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU= github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 h1:epCh84lMvA70Z7CTTCmYQn2CKbY8j86K7/FAIr141uY= github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7/go.mod h1:q4W45IWZaF22tdD+VEXcAWRA037jwmWEB5VWYORlTpc= github.com/tecbot/gorocksdb v0.0.0-20191217155057-f0fad39f321c h1:g+WoO5jjkqGAzHWCjJB1zZfXPIAaDpzXIEJ0eS6B5Ok= @@ -1979,8 +1987,8 @@ go.uber.org/goleak v1.2.0/go.mod h1:XJYK+MuIchqpmGmUSAzotztawfKvYLUIgg7guXrwVUo= go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4= go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU= -go.uber.org/multierr v1.8.0 h1:dg6GjLku4EH+249NNmoIciG9N/jURbDG+pFlTkhzIC8= -go.uber.org/multierr v1.8.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak= +go.uber.org/multierr v1.9.0 h1:7fIwc/ZtS0q++VgcfqFDxSBZVv/Xo49/SYnDFupUwlI= +go.uber.org/multierr v1.9.0/go.mod h1:X2jQV1h+kxSjClGpnseKVIxpmcjrj7MNnI0bnlfKTVQ= go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= go.uber.org/zap v1.9.1/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= @@ -2022,14 +2030,12 @@ golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPh golang.org/x/crypto v0.0.0-20201208171446-5f87f3452ae9/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I= golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I= golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= -golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220214200702-86341886e292/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= -golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= -golang.org/x/crypto v0.12.0 h1:tFM/ta59kqch6LlvYnPa0yx5a83cL2nHflFhYKvv9Yk= -golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= +golang.org/x/crypto v0.16.0 h1:mMMrFzRSCF0GvB7Ne27XVtVAaXLrPmgPC7/v0tkwHaY= +golang.org/x/crypto v0.16.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4= golang.org/x/exp v0.0.0-20180321215751-8460e604b9de/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20180807140117-3d87b88a115f/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= @@ -2045,8 +2051,8 @@ golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EH golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= golang.org/x/exp v0.0.0-20200331195152-e8c3332aa8e5/go.mod h1:4M0jN8W1tt0AVLNr8HDosyJCDCDuyL9N9+3m7wDWgKw= golang.org/x/exp v0.0.0-20200513190911-00229845015e/go.mod h1:4M0jN8W1tt0AVLNr8HDosyJCDCDuyL9N9+3m7wDWgKw= -golang.org/x/exp v0.0.0-20221019170559-20944726eadf h1:nFVjjKDgNY37+ZSYCJmtYf7tOlfQswHqplG2eosjOMg= -golang.org/x/exp v0.0.0-20221019170559-20944726eadf/go.mod h1:cyybsKvd6eL0RnXn6p/Grxp8F5bW7iYuBgsNCOHpMYE= +golang.org/x/exp v0.0.0-20230905200255-921286631fa9 h1:GoHiUyI/Tp2nVkLI2mCxVkOjsbSXD66ic0XW0js0R9g= +golang.org/x/exp v0.0.0-20230905200255-921286631fa9/go.mod h1:S2oDrQGGwySpoQPVqRShND87VCbxmc6bL1Yd2oYrm6k= golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= @@ -2145,8 +2151,8 @@ golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211209124913-491a49abca63/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= -golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14= -golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= +golang.org/x/net v0.19.0 h1:zTwKpTd2XuCqf8huc7Fo2iSy+4RHPd10s4KzeTnVr1c= +golang.org/x/net v0.19.0/go.mod h1:CfAk/cbD4CthTvqiEl8NpboMuiuOYsAr/7NOjZJtv1U= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190130055435-99b60b757ec1/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -2166,8 +2172,8 @@ golang.org/x/oauth2 v0.0.0-20210402161424-2e8d93401602/go.mod h1:KelEdhl1UZF7XfJ golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= -golang.org/x/oauth2 v0.11.0 h1:vPL4xzxBM4niKCW6g9whtaWVXTJf1U5e4aZxxFx/gbU= -golang.org/x/oauth2 v0.11.0/go.mod h1:LdF7O/8bLR/qWK9DrpXmbHLTouvRHK0SgJl0GmDBchk= +golang.org/x/oauth2 v0.15.0 h1:s8pnnxNVzjWyrvYdFUQq5llS1PX2zhPXmccZv99h7uQ= +golang.org/x/oauth2 v0.15.0/go.mod h1:q48ptWNTY5XWf+JNten23lcvHpLJ0ZSxF5ttTHKVCAM= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -2179,8 +2185,8 @@ golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.3.0 h1:ftCYgMx6zT/asHUrPw8BLLscYtGznsLAnjq5RH9P66E= -golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y= +golang.org/x/sync v0.5.0 h1:60k92dhOjHxJkrqnwsfl8KuaHbn/5dl0lUPUklKo3qE= +golang.org/x/sync v0.5.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= golang.org/x/sys v0.0.0-20170830134202-bb24a47a89ea/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -2214,6 +2220,7 @@ golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191002063906-3421d5a6bb1c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -2259,7 +2266,6 @@ golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210220050731-9a76102bfb43/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210305230114-8fe3ee5dd75b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210315160823-c6e025ad8005/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210316164454-77fc1eacc6aa/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -2269,7 +2275,6 @@ golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210403161142-5e06dd20ab57/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210420205809-ac73e9fd8988/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -2289,15 +2294,14 @@ golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220503163025-988cb79eb6c6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= -golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.15.0 h1:h48lPFYpsTvQJZF4EKyI4aLHaev3CxivZmv7yZig9pc= +golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.11.0 h1:F9tnn/DA/Im8nCwm+fX+1/eBwi4qFjRT++MhtVC4ZX0= -golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU= +golang.org/x/term v0.15.0 h1:y/Oo/a/q3IXu26lQgl04j/gjuBDOBlx7X6Om1j2CPW4= +golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0= golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -2309,8 +2313,8 @@ golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= -golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc= -golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= +golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -2321,8 +2325,8 @@ golang.org/x/time v0.0.0-20201208040808-7e3f01d25324/go.mod h1:tRJNPiyCQ0inRvYxb golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20220210224613-90d013bbcef8/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= -golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= +golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -2394,14 +2398,13 @@ golang.org/x/tools v0.0.0-20201208233053-a543418bbed2/go.mod h1:emZCQorbCU4vsT4f golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.0.0-20210105154028-b0ab187a4818/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.0.0-20210108195828-e2f9c7f1fc8e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= golang.org/x/tools v0.1.2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.3/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.10-0.20220218145154-897bd77cd717/go.mod h1:Uh6Zz+xoGYZom868N8YTex3t7RhtHDBrE8Gzo9bV56E= -golang.org/x/tools v0.6.0 h1:BOw41kyTf3PuCW1pVQf8+Cyg8pMlkYB1oo9iJ6D/lKM= -golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= +golang.org/x/tools v0.13.0 h1:Iey4qkscZuv0VvIt8E0neZjtPVQFSc870HQ448QgEmQ= +golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -2445,8 +2448,8 @@ google.golang.org/api v0.40.0/go.mod h1:fYKFpnQN0DsDSKRVRcQSDQNtqWPfM9i+zNPxepjR google.golang.org/api v0.41.0/go.mod h1:RkxM5lITDfTzmyKFPt+wGrCJbVfniCr2ool8kTBzRTU= google.golang.org/api v0.43.0/go.mod h1:nQsDGjRXMo4lvh5hP0TKqF244gqhGcr/YSIykhUk/94= google.golang.org/api v0.44.0/go.mod h1:EBOGZqzyhtvMDoxwS97ctnh0zUmYY6CxqXsc1AvkYD8= -google.golang.org/api v0.134.0 h1:ktL4Goua+UBgoP1eL1/60LwZJqa1sIzkLmvoR3hR6Gw= -google.golang.org/api v0.134.0/go.mod h1:sjRL3UnjTx5UqNQS9EWr9N8p7xbHpy1k0XGRLCf3Spk= +google.golang.org/api v0.153.0 h1:N1AwGhielyKFaUqH07/ZSIQR3uNPcV7NVw0vj+j4iR4= +google.golang.org/api v0.153.0/go.mod h1:3qNJX5eOmhiWYc67jRA/3GsDw97UFb5ivv7Y2PrriAY= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.2.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= @@ -2507,10 +2510,8 @@ google.golang.org/genproto v0.0.0-20201110150050-8816d57aaa9a/go.mod h1:FWY/as6D google.golang.org/genproto v0.0.0-20201201144952-b05cb90ed32e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20201210142538-e3217bee35cc/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20201214200347-8c77b98c765d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210108203827-ffc7fda8c3d7/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20210126160654-44e461bb6506/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20210222152913-aa3ee6e6a81c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= -google.golang.org/genproto v0.0.0-20210226172003-ab064af71705/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20210303154014-9728d6b83eeb/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20210310155132-4ce2db91004e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20210319143718-93e7006c17a6/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= @@ -2518,12 +2519,12 @@ google.golang.org/genproto v0.0.0-20210402141018-6c239bbf2bb1/go.mod h1:9lPAdzaE google.golang.org/genproto v0.0.0-20210602131652-f16073e35f0c/go.mod h1:UODoCrxHCcBojKKwX1terBiRUaqAsFqJiF615XL43r0= google.golang.org/genproto v0.0.0-20210624195500-8bfb893ecb84/go.mod h1:SzzZ/N+nwJDaO1kznhnlzqS8ocJICar6hYhVyhi++24= google.golang.org/genproto v0.0.0-20220107163113-42d7afdf6368/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20230706204954-ccb25ca9f130 h1:Au6te5hbKUV8pIYWHqOUZ1pva5qK/rwbIhoXEUB9Lu8= -google.golang.org/genproto v0.0.0-20230706204954-ccb25ca9f130/go.mod h1:O9kGHb51iE/nOGvQaDUuadVYqovW56s5emA88lQnj6Y= -google.golang.org/genproto/googleapis/api v0.0.0-20230706204954-ccb25ca9f130 h1:XVeBY8d/FaK4848myy41HBqnDwvxeV3zMZhwN1TvAMU= -google.golang.org/genproto/googleapis/api v0.0.0-20230706204954-ccb25ca9f130/go.mod h1:mPBs5jNgx2GuQGvFwUvVKqtn6HsUw9nP64BedgvqEsQ= -google.golang.org/genproto/googleapis/rpc v0.0.0-20230720185612-659f7aaaa771 h1:Z8qdAF9GFsmcUuWQ5KVYIpP3PCKydn/YKORnghIalu4= -google.golang.org/genproto/googleapis/rpc v0.0.0-20230720185612-659f7aaaa771/go.mod h1:TUfxEVdsvPg18p6AslUXFoLdpED4oBnGwyqk3dV1XzM= +google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17 h1:wpZ8pe2x1Q3f2KyT5f8oP/fa9rHAKgFPr/HZdNuS+PQ= +google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17/go.mod h1:J7XzRzVy1+IPwWHZUzoD0IccYZIrXILAQpc+Qy9CMhY= +google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17 h1:JpwMPBpFN3uKhdaekDpiNlImDdkUAyiJ6ez/uxGaUSo= +google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17/go.mod h1:0xJLfVdJqpAPl8tDg1ujOCGzx6LFLttXT5NhllGOXY4= +google.golang.org/genproto/googleapis/rpc v0.0.0-20231120223509-83a465c0220f h1:ultW7fxlIvee4HYrtnaRPon9HpEgFk5zYpmfMgtKB5I= +google.golang.org/genproto/googleapis/rpc v0.0.0-20231120223509-83a465c0220f/go.mod h1:L9KNLi232K1/xB6f7AlSX692koaRnKaWSR0stBki0Yc= google.golang.org/grpc v1.8.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw= google.golang.org/grpc v1.12.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw= google.golang.org/grpc v1.14.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw= @@ -2559,8 +2560,8 @@ google.golang.org/grpc v1.37.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQ google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM= google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34= google.golang.org/grpc v1.41.0/go.mod h1:U3l9uK9J0sini8mHphKoXyaqDA/8VyGnDee1zzIUK6k= -google.golang.org/grpc v1.57.0 h1:kfzNeI/klCGD2YPMUlaGNT3pxvYfga7smW3Vth8Zsiw= -google.golang.org/grpc v1.57.0/go.mod h1:Sd+9RMTACXwmub0zcNY2c4arhtrbBYD1AUHI/dt16Mo= +google.golang.org/grpc v1.59.0 h1:Z5Iec2pjwb+LEOqzpB2MR12/eKFhDPhuqW91O+4bwUk= +google.golang.org/grpc v1.59.0/go.mod h1:aUPDwccQo6OTjy7Hct4AfBPD1GptF4fyUjIkQ9YtF98= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= diff --git a/make/releasing.mk b/make/releasing.mk index c78cf1c0..9beb073d 100644 --- a/make/releasing.mk +++ b/make/releasing.mk @@ -3,13 +3,20 @@ GORELEASER_DEBUG ?= false GORELEASER_IMAGE := ghcr.io/goreleaser/goreleaser-cross:$(GOTOOLCHAIN_SEMVER) GORELEASER_MOUNT_CONFIG ?= false -ifeq ($(GORELEASER_RELEASE),true) - GORELEASER_SKIP_VALIDATE := false - GORELEASER_SKIP_PUBLISH := release --skip-publish=false -else - GORELEASER_SKIP_PUBLISH := --skip-publish=true - GORELEASER_SKIP_VALIDATE ?= false +GORELEASER_SKIP_FLAGS := $(GORELEASER_SKIP) +GORELEASER_SKIP := + +null := +space := $(null) # +comma := , + +ifneq ($(GORELEASER_RELEASE),true) GITHUB_TOKEN= + GORELEASER_SKIP_FLAGS += publish +endif + +ifneq ($(GORELEASER_SKIP_FLAGS),) + GORELEASER_SKIP := --skip=$(subst $(space),$(comma),$(strip $(GORELEASER_SKIP_FLAGS))) endif ifeq ($(GORELEASER_MOUNT_CONFIG),true) @@ -70,8 +77,7 @@ docker-image: -f .goreleaser-docker.yaml \ --debug=$(GORELEASER_DEBUG) \ --clean \ - --skip-validate \ - --skip-publish \ + --skip=publish,validate \ --snapshot .PHONY: gen-changelog @@ -100,8 +106,8 @@ release: gen-changelog -w /go/src/$(GO_MOD_NAME)\ $(GORELEASER_IMAGE) \ -f "$(GORELEASER_CONFIG)" \ - $(GORELEASER_SKIP_PUBLISH) \ - --skip-validate=$(GORELEASER_SKIP_VALIDATE) \ + release \ + $(GORELEASER_SKIP) \ --debug=$(GORELEASER_DEBUG) \ --clean \ --release-notes=/go/src/$(GO_MOD_NAME)/.cache/changelog.md diff --git a/operator/cmd.go b/operator/cmd.go index 31a9228d..ea35d61b 100644 --- a/operator/cmd.go +++ b/operator/cmd.go @@ -14,6 +14,7 @@ func Cmd() *cobra.Command { } cmd.AddCommand(inventory.Cmd()) + cmd.AddCommand(cmdPsutil()) return cmd } diff --git a/operator/inventory/ceph.go b/operator/inventory/ceph.go index 70269183..169f1f57 100644 --- a/operator/inventory/ceph.go +++ b/operator/inventory/ceph.go @@ -104,17 +104,17 @@ type ceph struct { exe RemotePodCommandExecutor ctx context.Context cancel context.CancelFunc - querier + querierStorage } -func NewCeph(ctx context.Context) (Storage, error) { +func NewCeph(ctx context.Context) (QuerierStorage, error) { ctx, cancel := context.WithCancel(ctx) c := &ceph{ - exe: NewRemotePodCommandExecutor(KubeConfigFromCtx(ctx), KubeClientFromCtx(ctx)), - ctx: ctx, - cancel: cancel, - querier: newQuerier(), + exe: NewRemotePodCommandExecutor(KubeConfigFromCtx(ctx), KubeClientFromCtx(ctx)), + ctx: ctx, + cancel: cancel, + querierStorage: newQuerierStorage(), } group := ErrGroupFromCtx(ctx) @@ -136,6 +136,8 @@ func (c *ceph) crdInstalled(log logr.Logger, rc *rookclientset.Clientset) bool { } } + // rc := rookclient.New() + // rc.CephV1().CephClusters("").List() return false } @@ -152,7 +154,7 @@ func (c *ceph) run() error { clusters := make(cephClusters) scs := make(cephStorageClasses) - scrapeData := resp{ + scrapeData := respStorage{ res: nil, err: errCephInventoryInProgress, } @@ -172,6 +174,14 @@ func (c *ceph) run() error { crdDiscoverTick := time.NewTicker(1 * time.Second) + // snapshot2v1 := func() inventory.ClusterStorage { + // res := make(inventory.ClusterStorage, 0, len(scs)) + // + // for name, sc := range scs { + // } + // return res + // } + for { select { case <-c.ctx.Done(): @@ -261,7 +271,7 @@ func (c *ceph) run() error { case req := <-c.reqch: req.respCh <- scrapeData case res := <-scrapech: - r := resp{} + r := respStorage{} if err := res.Error(); err != nil { r.err = errCephInventoryInProgress log.Error(err, "unable to pull ceph status") diff --git a/operator/inventory/cmd.go b/operator/inventory/cmd.go index d91134aa..f42a7cf5 100644 --- a/operator/inventory/cmd.go +++ b/operator/inventory/cmd.go @@ -5,13 +5,11 @@ import ( "encoding/json" "errors" "fmt" - "log" "net" "net/http" "sync" "time" - "github.com/akash-network/node/util/runner" "github.com/cskr/pubsub" "github.com/go-logr/logr" "github.com/go-logr/zapr" @@ -22,10 +20,18 @@ import ( "go.uber.org/zap" "go.uber.org/zap/zapcore" "golang.org/x/sync/errgroup" + "google.golang.org/grpc" + "google.golang.org/grpc/keepalive" + "google.golang.org/grpc/reflection" + "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes" + inventory "github.com/akash-network/akash-api/go/inventory/v1" + + "github.com/akash-network/node/util/runner" + "github.com/akash-network/provider/cluster/kube/clientcommon" providerflags "github.com/akash-network/provider/cmd/provider-services/cmd/flags" cmdutil "github.com/akash-network/provider/cmd/provider-services/cmd/util" @@ -33,6 +39,16 @@ import ( akashclientset "github.com/akash-network/provider/pkg/client/clientset/versioned" ) +type router struct { + *mux.Router + queryTimeout time.Duration +} + +type grpcMsgService struct { + inventory.ClusterRPCServer + ctx context.Context +} + func CmdSetContextValue(cmd *cobra.Command, key, val interface{}) { cmd.SetContext(context.WithValue(cmd.Context(), key, val)) } @@ -41,16 +57,17 @@ func Cmd() *cobra.Command { cmd := &cobra.Command{ Use: "inventory", Short: "kubernetes operator interfacing inventory", - Args: cobra.ExactArgs(0), SilenceUsage: true, - PreRunE: func(cmd *cobra.Command, args []string) error { + PersistentPreRunE: func(cmd *cobra.Command, args []string) error { zconf := zap.NewDevelopmentConfig() zconf.DisableCaller = true zconf.EncoderConfig.EncodeTime = func(time.Time, zapcore.PrimitiveArrayEncoder) {} zapLog, _ := zconf.Build() - cmd.SetContext(logr.NewContext(cmd.Context(), zapr.NewLogger(zapLog))) + group, ctx := errgroup.WithContext(cmd.Context()) + + cmd.SetContext(logr.NewContext(ctx, zapr.NewLogger(zapLog))) if err := loadKubeConfig(cmd); err != nil { return err @@ -58,11 +75,26 @@ func Cmd() *cobra.Command { kubecfg := KubeConfigFromCtx(cmd.Context()) - clientset, err := kubernetes.NewForConfig(kubecfg) + kc, err := kubernetes.NewForConfig(kubecfg) if err != nil { return err } + CmdSetContextValue(cmd, CtxKeyKubeConfig, kubecfg) + CmdSetContextValue(cmd, CtxKeyKubeClientSet, kc) + CmdSetContextValue(cmd, CtxKeyErrGroup, group) + + // lc := lifecycle.New() + // + // lc.WatchContext(cmd.Context()) + // + // CmdSetContextValue(cmd, CtxKeyLifecycle, lc) + + return nil + }, + PreRunE: func(cmd *cobra.Command, args []string) error { + kubecfg := KubeConfigFromCtx(cmd.Context()) + rc, err := rookclientset.NewForConfig(kubecfg) if err != nil { return err @@ -73,100 +105,103 @@ func Cmd() *cobra.Command { return err } - group, ctx := errgroup.WithContext(cmd.Context()) - cmd.SetContext(ctx) - - CmdSetContextValue(cmd, CtxKeyKubeClientSet, clientset) CmdSetContextValue(cmd, CtxKeyRookClientSet, rc) CmdSetContextValue(cmd, CtxKeyAkashClientSet, ac) CmdSetContextValue(cmd, CtxKeyPubSub, pubsub.New(1000)) - CmdSetContextValue(cmd, CtxKeyErrGroup, group) return nil }, RunE: func(cmd *cobra.Command, args []string) error { - bus := PubSubFromCtx(cmd.Context()) - group := ErrGroupFromCtx(cmd.Context()) + ctx := cmd.Context() + + bus := PubSubFromCtx(ctx) + group := ErrGroupFromCtx(ctx) - var storage []Storage - st, err := NewCeph(cmd.Context()) + var storage []QuerierStorage + st, err := NewCeph(ctx) if err != nil { return err } storage = append(storage, st) - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - // Channel to receive error messages from the goroutine - errChan := make(chan error, 1) - - // Start FeatureDiscovery in a separate goroutine - go func() { - errChan <- FeatureDiscovery(ctx) - }() - - // ... other code ... - - select { - case err := <-errChan: - // Handle error from FeatureDiscovery - // You might log the error, or take corrective action - log.Printf("FeatureDiscovery encountered an error: %v", err) - case <-cmd.Context().Done(): - // Handle the case where the main command is stopped - // Cancel the context used by FeatureDiscovery - cancel() - } - - if st, err = NewRancher(cmd.Context()); err != nil { + if st, err = NewRancher(ctx); err != nil { return err } + + fd := newFeatureDiscovery(ctx) + storage = append(storage, st) CmdSetContextValue(cmd, CtxKeyStorage, storage) + CmdSetContextValue(cmd, CtxKeyFeatureDiscovery, fd) - apiTimeout, _ := cmd.Flags().GetDuration(FlagAPITimeout) - queryTimeout, _ := cmd.Flags().GetDuration(FlagQueryTimeout) - port, _ := cmd.Flags().GetUint16(FlagAPIPort) + apiTimeout := viper.GetDuration(FlagAPITimeout) + queryTimeout := viper.GetDuration(FlagQueryTimeout) + restPort := viper.GetUint16(FlagRESTPort) + grpcPort := viper.GetUint16(FlagGRPCPort) - // fixme ovrclk/engineering#609 - // nolint: gosec srv := &http.Server{ - Addr: fmt.Sprintf(":%d", port), - Handler: newRouter(LogFromCtx(cmd.Context()).WithName("router"), apiTimeout, queryTimeout), + Addr: fmt.Sprintf(":%d", restPort), + Handler: newRouter(LogFromCtx(ctx).WithName("router"), apiTimeout, queryTimeout), BaseContext: func(_ net.Listener) context.Context { return cmd.Context() }, + ReadHeaderTimeout: 5 * time.Second, + ReadTimeout: 60 * time.Second, } + grpcSrv := grpc.NewServer(grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{ + MinTime: 30 * time.Second, + PermitWithoutStream: false, + })) + + inventory.RegisterClusterRPCServer(grpcSrv, &grpcMsgService{ + ctx: ctx, + }) + + reflection.Register(grpcSrv) + + group.Go(func() error { + return fd.Wait() + }) + group.Go(func() error { return srv.ListenAndServe() }) group.Go(func() error { - <-cmd.Context().Done() - return srv.Shutdown(cmd.Context()) + grpcLis, err := net.Listen("tcp", fmt.Sprintf(":%d", grpcPort)) + if err != nil { + return err + } + + return srv.Serve(grpcLis) + }) + + group.Go(func() error { + <-ctx.Done() + return srv.Shutdown(ctx) }) - factory := informers.NewSharedInformerFactory(KubeClientFromCtx(cmd.Context()), 0) + kc := KubeClientFromCtx(ctx) + factory := informers.NewSharedInformerFactory(kc, 0) - InformKubeObjects(cmd.Context(), + InformKubeObjects(ctx, bus, factory.Core().V1().Namespaces().Informer(), "ns") - InformKubeObjects(cmd.Context(), + InformKubeObjects(ctx, bus, factory.Storage().V1().StorageClasses().Informer(), "sc") - InformKubeObjects(cmd.Context(), + InformKubeObjects(ctx, bus, factory.Core().V1().PersistentVolumes().Informer(), "pv") - InformKubeObjects(cmd.Context(), + InformKubeObjects(ctx, bus, factory.Core().V1().Nodes().Informer(), "nodes") @@ -190,11 +225,18 @@ func Cmd() *cobra.Command { panic(err) } - cmd.Flags().Uint16(FlagAPIPort, 8080, "port to REST api") - if err = viper.BindPFlag(FlagAPIPort, cmd.Flags().Lookup(FlagAPIPort)); err != nil { + cmd.Flags().Uint16(FlagRESTPort, 8080, "port to REST api") + if err = viper.BindPFlag(FlagRESTPort, cmd.Flags().Lookup(FlagRESTPort)); err != nil { + panic(err) + } + + cmd.Flags().Uint16(FlagGRPCPort, 8081, "port to GRPC api") + if err = viper.BindPFlag(FlagGRPCPort, cmd.Flags().Lookup(FlagGRPCPort)); err != nil { panic(err) } + cmd.AddCommand(cmdFeatureDiscoveryNode()) + return cmd } @@ -211,10 +253,14 @@ func loadKubeConfig(c *cobra.Command) error { return nil } -func newRouter(_ logr.Logger, apiTimeout, queryTimeout time.Duration) *mux.Router { - router := mux.NewRouter() +func newRouter(_ logr.Logger, apiTimeout, queryTimeout time.Duration) *router { + mRouter := mux.NewRouter() + rt := &router{ + Router: mRouter, + queryTimeout: queryTimeout, + } - router.Use(func(h http.Handler) http.Handler { + mRouter.Use(func(h http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { rCtx, cancel := context.WithTimeout(r.Context(), apiTimeout) defer cancel() @@ -223,77 +269,221 @@ func newRouter(_ logr.Logger, apiTimeout, queryTimeout time.Duration) *mux.Route }) }) - router.HandleFunc("/inventory", func(w http.ResponseWriter, req *http.Request) { - storage := StorageFromCtx(req.Context()) - inv := akashv2beta2.Inventory{ - TypeMeta: metav1.TypeMeta{ - Kind: "Inventory", - APIVersion: "akash.network/v2beta2", - }, - ObjectMeta: metav1.ObjectMeta{ - CreationTimestamp: metav1.NewTime(time.Now().UTC()), - }, - Spec: akashv2beta2.InventorySpec{}, - Status: akashv2beta2.InventoryStatus{ - State: akashv2beta2.InventoryStatePulled, - }, + mRouter.HandleFunc("/inventory", rt.legacyInventoryHandler) + + inventoryRouter := mRouter.PathPrefix("/v1").Subrouter() + inventoryRouter.HandleFunc("/inventory", rt.InventoryHandler) + + return rt +} + +func (rt *router) InventoryHandler(w http.ResponseWriter, req *http.Request) { + storage := StorageFromCtx(req.Context()) + fd := FeatureDiscoveryFromCtx(req.Context()) + + var data []byte + + ctx, cancel := context.WithTimeout(req.Context(), rt.queryTimeout) + defer func() { + if errors.Is(ctx.Err(), context.DeadlineExceeded) { + w.WriteHeader(http.StatusRequestTimeout) + } + + if len(data) > 0 { + _, _ = w.Write(data) } + }() - var data []byte + datach := make(chan runner.Result, 1) + var wg sync.WaitGroup - ctx, cancel := context.WithTimeout(req.Context(), queryTimeout) - defer func() { - if errors.Is(ctx.Err(), context.DeadlineExceeded) { - w.WriteHeader(http.StatusRequestTimeout) - } + wg.Add(len(storage) + 1) + + for idx := range storage { + go func(idx int) { + defer wg.Done() - if len(data) > 0 { - _, _ = w.Write(data) + datach <- runner.NewResult(storage[idx].Query(ctx)) + }(idx) + } + + go func() { + defer wg.Done() + datach <- runner.NewResult(fd.Query(ctx)) + }() + + go func() { + defer cancel() + wg.Wait() + }() + + inv := inventory.Cluster{} + +done: + for { + select { + case <-ctx.Done(): + break done + case res := <-datach: + switch obj := res.Value().(type) { + case []akashv2beta2.InventoryClusterStorage: + for _, s := range obj { + nS := inventory.Storage{ + Quantity: inventory.ResourcePair{ + Allocatable: resource.NewQuantity(int64(s.Allocatable), resource.DecimalSI), + Allocated: resource.NewQuantity(int64(s.Allocated), resource.DecimalSI), + Attributes: nil, + }, + Info: inventory.StorageInfo{ + Class: s.Class, + }, + } + + inv.Storage = append(inv.Storage, nS) + } + case inventory.Nodes: + inv.Nodes = obj.Dup() } - }() + } + } - datach := make(chan runner.Result, 1) - var wg sync.WaitGroup + var err error + if data, err = json.Marshal(&inv); err != nil { + w.WriteHeader(http.StatusInternalServerError) + data = []byte(err.Error()) + } else { + w.Header().Set("Content-Type", "application/json") + } +} - wg.Add(len(storage)) +func (rt *router) legacyInventoryHandler(w http.ResponseWriter, req *http.Request) { + storage := StorageFromCtx(req.Context()) + inv := akashv2beta2.Inventory{ + TypeMeta: metav1.TypeMeta{ + Kind: "Inventory", + APIVersion: "akash.network/v2beta2", + }, + ObjectMeta: metav1.ObjectMeta{ + CreationTimestamp: metav1.NewTime(time.Now().UTC()), + }, + Spec: akashv2beta2.InventorySpec{}, + Status: akashv2beta2.InventoryStatus{ + State: akashv2beta2.InventoryStatePulled, + }, + } - for idx := range storage { - go func(idx int) { - defer wg.Done() + var data []byte - datach <- runner.NewResult(storage[idx].Query(ctx)) - }(idx) + ctx, cancel := context.WithTimeout(req.Context(), rt.queryTimeout) + defer func() { + if errors.Is(ctx.Err(), context.DeadlineExceeded) { + w.WriteHeader(http.StatusRequestTimeout) } - go func() { - defer cancel() - wg.Wait() - }() + if len(data) > 0 { + _, _ = w.Write(data) + } + }() - done: - for { - select { - case <-ctx.Done(): - break done - case res := <-datach: - if res.Error() != nil { - inv.Status.Messages = append(inv.Status.Messages, res.Error().Error()) - } + datach := make(chan runner.Result, 1) + var wg sync.WaitGroup - if inventory, valid := res.Value().([]akashv2beta2.InventoryClusterStorage); valid { - inv.Spec.Storage = append(inv.Spec.Storage, inventory...) - } + wg.Add(len(storage) + 1) + + for idx := range storage { + go func(idx int) { + defer wg.Done() + + datach <- runner.NewResult(storage[idx].Query(ctx)) + }(idx) + } + + go func() { + defer cancel() + wg.Wait() + }() + +done: + for { + select { + case <-ctx.Done(): + break done + case res := <-datach: + if res.Error() != nil { + inv.Status.Messages = append(inv.Status.Messages, res.Error().Error()) } - } - var err error - if data, err = json.Marshal(&inv); err != nil { - w.WriteHeader(http.StatusInternalServerError) - data = []byte(err.Error()) - } else { - w.Header().Set("Content-Type", "application/json") + if inventory, valid := res.Value().([]akashv2beta2.InventoryClusterStorage); valid { + inv.Spec.Storage = append(inv.Spec.Storage, inventory...) + } } - }) + } - return router + var err error + if data, err = json.Marshal(&inv); err != nil { + w.WriteHeader(http.StatusInternalServerError) + data = []byte(err.Error()) + } else { + w.Header().Set("Content-Type", "application/json") + } +} + +func (gm *grpcMsgService) QueryCluster(_ *inventory.VoidNoParam, stream inventory.ClusterRPC_QueryClusterServer) error { + bus := PubSubFromCtx(gm.ctx) + + sendch := make(chan interface{}, 1) + + // reqch := make(chan inventory.Nodes, 1) + + // select { + // case <-s.ctx.Done(): + // return s.ctx.Err() + // case <-stream.Context().Done(): + // return stream.Context().Err() + // case s.reqch <- reqch: + // } + // + // select { + // case <-s.ctx.Done(): + // return s.ctx.Err() + // case <-stream.Context().Done(): + // return stream.Context().Err() + // case sendch <- <-reqch: + // } + + subch := bus.Sub(topicNodes) + + defer func() { + bus.Unsub(subch, topicNodes) + }() + + var state inventory.Cluster + + for { + select { + case <-gm.ctx.Done(): + return gm.ctx.Err() + case <-stream.Context().Done(): + return stream.Context().Err() + case msg := <-sendch: + switch obj := msg.(type) { + case inventory.Nodes: + state.Nodes = obj + case inventory.ClusterStorage: + state.Storage = obj + } + + if err := stream.Send(&state); err != nil { + return err + } + case msg := <-subch: + select { + case <-gm.ctx.Done(): + return gm.ctx.Err() + case <-stream.Context().Done(): + return stream.Context().Err() + case sendch <- msg: + } + } + } } diff --git a/operator/inventory/feature-discovery-client.go b/operator/inventory/feature-discovery-client.go new file mode 100644 index 00000000..7d8ccdc6 --- /dev/null +++ b/operator/inventory/feature-discovery-client.go @@ -0,0 +1,339 @@ +package inventory + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/url" + "os" + "strings" + "time" + + inventory "github.com/akash-network/akash-api/go/inventory/v1" + "github.com/go-logr/logr" + "golang.org/x/sync/errgroup" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/watch" + "k8s.io/client-go/kubernetes" + "k8s.io/client-go/tools/portforward" + "k8s.io/client-go/transport/spdy" + + "github.com/akash-network/provider/cluster/kube/builder" +) + +type nodeStateEnum int + +const ( + daemonSetLabelSelector = "app=hostfeaturediscovery" + daemonSetNamespace = "akash-services" + reconnectTimeout = 5 * time.Second +) + +const ( + nodeStateUpdated nodeStateEnum = iota + nodeStateRemoved +) + +type podStream struct { + ctx context.Context + cancel context.CancelFunc + group *errgroup.Group + log logr.Logger + conn *grpc.ClientConn + stream inventory.NodeRPC_QueryNodeClient + broadcastStateCh chan<- nodeState + nodeName string + address string + port uint16 +} + +type nodeState struct { + state nodeStateEnum + name string + node inventory.Node +} + +type featureDiscovery struct { + querierNodes + ctx context.Context + group *errgroup.Group + log logr.Logger + kc *kubernetes.Clientset + nodeCh chan nodeState +} + +func newFeatureDiscovery(ctx context.Context) *featureDiscovery { + log := LogFromCtx(ctx).WithName("feature-discovery") + + group, ctx := errgroup.WithContext(ctx) + + fd := &featureDiscovery{ + log: log, + ctx: logr.NewContext(ctx, log), + group: group, + kc: KubeClientFromCtx(ctx), + querierNodes: newQuerierNodes(), + nodeCh: make(chan nodeState, 1), + } + + group.Go(func() error { + return fd.connectorRun() + }) + + group.Go(func() error { + return fd.run() + }) + + return fd +} + +func (fd *featureDiscovery) Wait() error { + return fd.group.Wait() +} + +func (fd *featureDiscovery) connectorRun() error { + watcher, err := fd.kc.CoreV1().Pods(daemonSetNamespace).Watch(fd.ctx, metav1.ListOptions{ + LabelSelector: builder.AkashManagedLabelName + "=true" + + ",app.kubernetes.io/name=feature-discovery-node" + + ",app.kubernetes.io/component=inventory" + + ",app.kubernetes.io/part-of=operator", + }) + + if err != nil { + return fmt.Errorf("error setting up Kubernetes watcher: %w", err) + } + + nodes := make(map[string]*podStream) + + for { + select { + case <-fd.ctx.Done(): + for _, nd := range nodes { + nd.cancel() + delete(nodes, nd.nodeName) + } + + return fd.ctx.Err() + case event := <-watcher.ResultChan(): + if obj, valid := event.Object.(*corev1.Pod); valid { + nodeName := obj.Spec.NodeName + + switch event.Type { + case watch.Added: + fallthrough + case watch.Modified: + if obj.Status.Phase == corev1.PodRunning && obj.Status.PodIP != "" { + if _, exists := nodes[nodeName]; exists { + continue + } + + var containerPort uint16 + + for _, container := range obj.Spec.Containers { + if container.Name == fdContainerName { + for _, port := range container.Ports { + if port.Name == fdContainerPortName { + containerPort = uint16(port.ContainerPort) + break + } + } + break + } + } + + nodes[nodeName] = newNodeWatcher(fd.ctx, nodeName, obj.Name, obj.Status.PodIP, containerPort, fd.nodeCh) + } + case watch.Deleted: + nd, exists := nodes[nodeName] + if !exists { + continue + } + + nd.cancel() + delete(nodes, nodeName) + } + } + } + } +} + +func (fd *featureDiscovery) run() error { + nodes := make(map[string]inventory.Node) + + snapshot := func() inventory.Nodes { + res := make(inventory.Nodes, 0, len(nodes)) + + for _, nd := range nodes { + res = append(res, nd.Dup()) + } + + return res + } + + bus := PubSubFromCtx(fd.ctx) + + for { + select { + case <-fd.ctx.Done(): + return fd.ctx.Err() + case evt := <-fd.nodeCh: + switch evt.state { + case nodeStateUpdated: + nodes[evt.name] = evt.node + case nodeStateRemoved: + delete(nodes, evt.name) + } + + bus.Pub(snapshot(), topicNodes) + case req := <-fd.reqch: + resp := respNodes{ + res: snapshot(), + } + + req.respCh <- resp + } + } +} + +func newNodeWatcher(ctx context.Context, nodeName string, podName string, address string, port uint16, evt chan<- nodeState) *podStream { + ctx, cancel := context.WithCancel(ctx) + group, ctx := errgroup.WithContext(ctx) + + ps := &podStream{ + ctx: ctx, + cancel: cancel, + group: group, + log: LogFromCtx(ctx).WithName("node-watcher"), + broadcastStateCh: evt, + nodeName: nodeName, + address: address, + port: port, + } + + kubecfg := KubeConfigFromCtx(ctx) + + if kubecfg.BearerTokenFile != "/var/run/secrets/kubernetes.io/serviceaccount/token" { + roundTripper, upgrader, err := spdy.RoundTripperFor(kubecfg) + if err != nil { + panic(err) + } + + path := fmt.Sprintf("/api/v1/namespaces/%s/pods/%s/portforward", daemonSetNamespace, podName) + hostIP := strings.TrimLeft(kubecfg.Host, "https:/") + serverURL := url.URL{Scheme: "https", Path: path, Host: hostIP} + + dialer := spdy.NewDialer(upgrader, &http.Client{Transport: roundTripper}, http.MethodPost, &serverURL) + + errch := make(chan error, 1) + pf, err := portforward.New(dialer, []string{fmt.Sprintf(":%d", port)}, ctx.Done(), make(chan struct{}), os.Stdout, os.Stderr) + + group.Go(func() error { + err := pf.ForwardPorts() + errch <- err + return err + }) + + select { + case <-pf.Ready: + case err := <-errch: + panic(err) + } + + ports, err := pf.GetPorts() + if err != nil { + + } + + ps.address = "localhost" + ps.port = ports[0].Local + } + + _ = ps.reconnect() + + return ps +} + +// nolint: unused +func (nd *podStream) reconnect() error { + select { + case <-nd.ctx.Done(): + return nd.ctx.Err() + default: + } + + var err error + // Establish the gRPC connection + nd.conn, err = grpc.Dial(fmt.Sprintf("%s:%d", nd.address, nd.port), grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock()) + if err != nil { + nd.log.Error(err, "couldn't dial endpoint") + return err + } + + client := inventory.NewNodeRPCClient(nd.conn) + + // Create a stream to receive updates from the node + nd.stream, err = client.QueryNode(nd.ctx, &inventory.VoidNoParam{}) + if err != nil { + nd.log.Error(err, "couldn't establish stream") + return err + } + + go nd.run() + + return nil +} + +func (nd *podStream) run() { + defer func() { + _ = nd.conn.Close() + }() + + for { + node, err := nd.stream.Recv() + if err != nil { + select { + case <-nd.ctx.Done(): + return + case nd.broadcastStateCh <- nodeState{ + state: nodeStateRemoved, + name: nd.nodeName, + }: + } + + // if context is still alive then try to reconnect + for { + if err = nd.reconnect(); err != nil { + if errors.Is(err, context.Canceled) { + return + } + + ctx, cancel := context.WithTimeout(nd.ctx, reconnectTimeout) + select { + case <-ctx.Done(): + err := ctx.Err() + cancel() + if !errors.Is(err, context.DeadlineExceeded) { + return + } + } + } + + return + } + } + + select { + case <-nd.ctx.Done(): + return + case nd.broadcastStateCh <- nodeState{ + state: nodeStateUpdated, + name: nd.nodeName, + node: node.Dup(), + }: + } + } +} diff --git a/operator/inventory/feature-discovery-node.go b/operator/inventory/feature-discovery-node.go new file mode 100644 index 00000000..fd484768 --- /dev/null +++ b/operator/inventory/feature-discovery-node.go @@ -0,0 +1,901 @@ +package inventory + +import ( + "embed" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "strconv" + "time" + + "github.com/cskr/pubsub" + "github.com/go-logr/logr" + "github.com/jaypipes/ghw/pkg/cpu" + "github.com/jaypipes/ghw/pkg/gpu" + "github.com/jaypipes/ghw/pkg/memory" + "github.com/spf13/cobra" + "github.com/spf13/viper" + "golang.org/x/net/context" + "google.golang.org/grpc" + "google.golang.org/grpc/keepalive" + "google.golang.org/grpc/reflection" + corev1 "k8s.io/api/core/v1" + kerrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/fields" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/watch" + + v1 "github.com/akash-network/akash-api/go/inventory/v1" + + "github.com/akash-network/provider/cluster/kube/builder" +) + +//go:embed gpu-info.json +var gpuDevs embed.FS + +const ( + fdContainerName = "inventory-node" + fdContainerPortName = "api" + topicNode = "node" + topicNodes = "nodes" +) + +type gpuDevice struct { + Interface string `json:"interface"` + MemorySize string `json:"memory_size"` +} + +type gpuDevices map[string]gpuDevice + +type gpuVendor struct { + Name string `json:"name"` + Devices gpuDevices `json:"devices"` +} + +type gpuVendors map[string]gpuVendor + +type dpReqType int + +const ( + dpReqCPU dpReqType = iota + dpReqGPU + dpReqMem +) + +type dpReadResp struct { + data interface{} + err error +} +type dpReadReq struct { + op dpReqType + resp chan<- dpReadResp +} + +type debuggerPod struct { + ctx context.Context + readch chan dpReadReq +} + +type Publisher interface { + Pub(msg interface{}, topics ...string) + TryPub(msg interface{}, topics ...string) +} + +type Subscriber interface { + Sub(topics ...string) chan interface{} + SubOnce(topics ...string) chan interface{} + SubOnceEach(topics ...string) chan interface{} + AddSub(ch chan interface{}, topics ...string) + AddSubOnceEach(ch chan interface{}, topics ...string) + Unsub(ch chan interface{}, topics ...string) +} + +type msgServiceServer struct { + v1.NodeRPCServer + ctx context.Context + log logr.Logger + sub Subscriber + reqch chan<- chan<- v1.Node +} + +type fdNodeServer struct { + ctx context.Context + log logr.Logger + reqch <-chan chan<- v1.Node + pub Publisher + nodeName string +} + +var ( + supportedGPUs = gpuVendors{} +) + +func init() { + f, err := gpuDevs.Open("gpu-info.json") + if err != nil { + panic(err) + } + // close pci.ids file when done + defer func() { + _ = f.Close() + }() + + data, err := io.ReadAll(f) + if err != nil { + panic(err) + } + + err = json.Unmarshal(data, &supportedGPUs) + if err != nil { + panic(err) + } +} + +func cmdFeatureDiscoveryNode() *cobra.Command { + cmd := &cobra.Command{ + Use: "node", + Short: "feature discovery daemon-set node", + Args: cobra.ExactArgs(0), + SilenceUsage: true, + PreRunE: func(cmd *cobra.Command, args []string) error { + kubecfg := KubeConfigFromCtx(cmd.Context()) + + var hw hwInfo + + log := LogFromCtx(cmd.Context()).WithName("feature-discovery-node") + + if kubecfg.BearerTokenFile != "/var/run/secrets/kubernetes.io/serviceaccount/token" { + log.Info("service is not running as kubernetes pod. starting debugger pod") + + dp := &debuggerPod{ + ctx: cmd.Context(), + readch: make(chan dpReadReq, 1), + } + + group := ErrGroupFromCtx(cmd.Context()) + + startch := make(chan struct{}) + + group.Go(func() error { + return dp.run(startch) + }) + + ctx, cancel := context.WithTimeout(cmd.Context(), 5*time.Second) + + select { + case <-ctx.Done(): + if !errors.Is(ctx.Err(), context.DeadlineExceeded) { + return ctx.Err() + } + case <-startch: + cancel() + } + + hw = dp + } else { + hw = &localHwReader{} + } + + CmdSetContextValue(cmd, CtxKeyHwInfo, hw) + + return nil + }, + RunE: func(cmd *cobra.Command, args []string) error { + ctx := cmd.Context() + log := LogFromCtx(ctx).WithName("feature-discovery-node") + + log.Info("starting k8s node features discovery") + + var err error + + podName := viper.GetString(FlagPodName) + nodeName := viper.GetString(FlagNodeName) + listenPort := viper.GetUint16(FlagGRPCPort) + + kc := KubeClientFromCtx(ctx) + + if listenPort == 0 { + // this is dirty hack to discover exposed api port if this service runs withing kubernetes + podInfo, err := kc.CoreV1().Pods(daemonSetNamespace).Get(ctx, podName, metav1.GetOptions{}) + if err != nil { + return err + } + + for _, container := range podInfo.Spec.Containers { + if container.Name == fdContainerName { + for _, port := range container.Ports { + if port.Name == fdContainerPortName { + listenPort = uint16(port.ContainerPort) + } + } + } + } + + if listenPort == 0 { + return fmt.Errorf("unable to detect pod's api port") + } + } + + bus := pubsub.New(1000) + + endpoint := fmt.Sprintf(":%d", listenPort) + + log.Info(fmt.Sprintf("grpc listening on \"%s\"", endpoint)) + + lis, err := net.Listen("tcp", endpoint) + if err != nil { + return err + } + + srv := grpc.NewServer(grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{ + MinTime: 30 * time.Second, + PermitWithoutStream: false, + })) + + reqch := make(chan chan<- v1.Node, 1) + + v1.RegisterNodeRPCServer(srv, &msgServiceServer{ + ctx: ctx, + log: log.WithName("msg-srv"), + sub: bus, + reqch: reqch, + }) + + reflection.Register(srv) + + group := ErrGroupFromCtx(ctx) + + fdns := &fdNodeServer{ + ctx: ctx, + log: log.WithName("watcher"), + reqch: reqch, + pub: bus, + nodeName: nodeName, + } + + startch := make(chan struct{}, 1) + group.Go(func() error { + return fdns.run(startch) + }) + + select { + case <-startch: + group.Go(func() error { + return srv.Serve(lis) + }) + case <-ctx.Done(): + return ctx.Err() + } + + select { + case <-ctx.Done(): + } + + err = group.Wait() + if !errors.Is(err, context.Canceled) { + return err + } + + return nil + }, + } + + cmd.Flags().String(FlagPodName, "", "instance name") + if err := viper.BindPFlag(FlagPodName, cmd.Flags().Lookup(FlagPodName)); err != nil { + panic(err) + } + + cmd.Flags().String(FlagNodeName, "", "node name") + if err := viper.BindPFlag(FlagNodeName, cmd.Flags().Lookup(FlagNodeName)); err != nil { + panic(err) + } + + return cmd +} + +func (nd *fdNodeServer) run(startch chan<- struct{}) error { + kc := KubeClientFromCtx(nd.ctx) + + nodeWatch, err := kc.CoreV1().Nodes().Watch(nd.ctx, metav1.ListOptions{ + LabelSelector: builder.AkashManagedLabelName + "=true", + FieldSelector: fields.OneTermEqualSelector(metav1.ObjectNameField, nd.nodeName).String(), + }) + if err != nil { + nd.log.Error(err, fmt.Sprintf("unable to start node watcher for \"%s\"", nd.nodeName)) + return err + } + + defer nodeWatch.Stop() + + podsWatch, err := kc.CoreV1().Pods(corev1.NamespaceAll).Watch(nd.ctx, metav1.ListOptions{ + FieldSelector: fields.OneTermEqualSelector("spec.nodeName", nd.nodeName).String(), + }) + if err != nil { + nd.log.Error(err, "unable to fetch pods") + return err + } + + defer podsWatch.Stop() + + node, initPods, err := initNodeInfo(nd.ctx, nd.nodeName) + if err != nil { + nd.log.Error(err, "unable to init node info") + return err + } + + select { + case <-nd.ctx.Done(): + return nd.ctx.Err() + case startch <- struct{}{}: + } + + signalch := make(chan struct{}, 1) + signalch <- struct{}{} + + trySignal := func() { + select { + case signalch <- struct{}{}: + default: + } + } + + for { + select { + case <-nd.ctx.Done(): + return nd.ctx.Err() + case <-signalch: + nd.pub.Pub(node.Dup(), topicNode) + case req := <-nd.reqch: + req <- node.Dup() + case res := <-podsWatch.ResultChan(): + if pod, valid := res.Object.(*corev1.Pod); valid { + switch res.Type { + case watch.Added: + if _, exists := initPods[pod.Name]; exists { + delete(initPods, pod.Name) + } else { + for _, container := range pod.Spec.Containers { + addAllocatedResources(&node, container.Resources.Requests) + } + } + case watch.Deleted: + if _, exists := initPods[pod.Name]; exists { + delete(initPods, pod.Name) + } + + for _, container := range pod.Spec.Containers { + subAllocatedResources(&node, container.Resources.Requests) + } + } + + trySignal() + } + } + } +} + +func addAllocatedResources(node *v1.Node, rl corev1.ResourceList) { + for name, quantity := range rl { + switch name { + case corev1.ResourceCPU: + node.CPU.Quantity.Allocated.Add(quantity) + case corev1.ResourceMemory: + node.Memory.Quantity.Allocated.Add(quantity) + case corev1.ResourceEphemeralStorage: + node.EphemeralStorage.Quantity.Allocated.Add(quantity) + case builder.ResourceGPUNvidia: + fallthrough + case builder.ResourceGPUAMD: + node.GPU.Quantity.Allocated.Add(quantity) + } + } +} + +func subAllocatedResources(node *v1.Node, rl corev1.ResourceList) { + for name, quantity := range rl { + switch name { + case corev1.ResourceCPU: + node.CPU.Quantity.Allocated.Sub(quantity) + case corev1.ResourceMemory: + node.Memory.Quantity.Allocated.Sub(quantity) + case corev1.ResourceEphemeralStorage: + node.EphemeralStorage.Quantity.Allocated.Sub(quantity) + case builder.ResourceGPUNvidia: + fallthrough + case builder.ResourceGPUAMD: + node.GPU.Quantity.Allocated.Sub(quantity) + } + } +} + +func initNodeInfo(ctx context.Context, name string) (v1.Node, map[string]corev1.Pod, error) { + kc := KubeClientFromCtx(ctx) + + cpuInfo, err := parseCPUInfo(ctx) + if err != nil { + return v1.Node{}, nil, err + } + + gpuInfo, err := parseGPUInfo(ctx) + if err != nil { + return v1.Node{}, nil, err + } + + res := v1.Node{ + CPU: v1.CPU{ + Quantity: v1.ResourcePair{ + Allocatable: resource.NewMilliQuantity(0, resource.DecimalSI), + Allocated: resource.NewMilliQuantity(0, resource.DecimalSI), + Attributes: nil, + }, + Info: cpuInfo, + }, + GPU: v1.GPU{ + Quantity: v1.ResourcePair{ + Allocatable: resource.NewQuantity(0, resource.DecimalSI), + Allocated: resource.NewQuantity(0, resource.DecimalSI), + }, + Info: gpuInfo, + }, + Memory: v1.Memory{ + Quantity: v1.ResourcePair{ + Allocatable: resource.NewQuantity(0, resource.DecimalSI), + Allocated: resource.NewQuantity(0, resource.DecimalSI), + }, + Info: nil, + }, + EphemeralStorage: v1.Storage{ + Quantity: v1.ResourcePair{ + Allocatable: resource.NewQuantity(0, resource.DecimalSI), + Allocated: resource.NewQuantity(0, resource.DecimalSI), + }, + Info: v1.StorageInfo{}, + }, + } + + knode, err := kc.CoreV1().Nodes().Get(ctx, name, metav1.GetOptions{}) + if err != nil { + return v1.Node{}, nil, fmt.Errorf("error fetching node %s: %w", name, err) + } + + for name, r := range knode.Status.Allocatable { + switch name { + case corev1.ResourceCPU: + val := r.DeepCopy() + res.CPU.Quantity.Allocatable = &val + res.CPU.Quantity.Allocated = resource.NewMilliQuantity(0, resource.DecimalSI) + case corev1.ResourceMemory: + val := r.DeepCopy() + res.Memory.Quantity.Allocatable = &val + res.Memory.Quantity.Allocated = resource.NewQuantity(0, resource.DecimalSI) + case corev1.ResourceEphemeralStorage: + val := r.DeepCopy() + res.EphemeralStorage.Quantity.Allocatable = &val + res.EphemeralStorage.Quantity.Allocated = resource.NewQuantity(0, resource.DecimalSI) + case builder.ResourceGPUNvidia: + case builder.ResourceGPUAMD: + val := r.DeepCopy() + res.GPU.Quantity.Allocatable = &val + res.GPU.Quantity.Allocated = resource.NewQuantity(0, resource.DecimalSI) + } + } + + initPods := make(map[string]corev1.Pod) + + podsList, err := kc.CoreV1().Pods(corev1.NamespaceAll).List(ctx, metav1.ListOptions{ + FieldSelector: fields.OneTermEqualSelector("spec.nodeName", name).String(), + }) + if err != nil { + } + + for _, pod := range podsList.Items { + for _, container := range pod.Spec.Containers { + addAllocatedResources(&res, container.Resources.Requests) + } + initPods[pod.Name] = pod + } + + return res, nil, nil +} + +func (s *msgServiceServer) QueryNode(_ *v1.VoidNoParam, stream v1.NodeRPC_QueryNodeServer) error { + sendch := make(chan interface{}, 1) + + reqch := make(chan v1.Node, 1) + + select { + case <-s.ctx.Done(): + return s.ctx.Err() + case <-stream.Context().Done(): + return stream.Context().Err() + case s.reqch <- reqch: + } + + select { + case <-s.ctx.Done(): + return s.ctx.Err() + case <-stream.Context().Done(): + return stream.Context().Err() + case sendch <- <-reqch: + } + + subch := s.sub.Sub(topicNode) + + defer func() { + s.sub.Unsub(subch, topicNode) + }() + + for { + select { + case <-s.ctx.Done(): + return s.ctx.Err() + case <-stream.Context().Done(): + return stream.Context().Err() + case nd := <-sendch: + msg := nd.(v1.Node) + if err := stream.Send(&msg); err != nil { + return err + } + case msg := <-subch: + select { + case <-s.ctx.Done(): + return s.ctx.Err() + case <-stream.Context().Done(): + return stream.Context().Err() + case sendch <- msg: + } + } + } +} + +type hwInfo interface { + CPU(context.Context) (*cpu.Info, error) + GPU(context.Context) (*gpu.Info, error) + Memory(context.Context) (*memory.Info, error) +} + +type localHwReader struct{} + +func (lfs *localHwReader) CPU(_ context.Context) (*cpu.Info, error) { + return cpu.New() +} + +func (lfs *localHwReader) GPU(_ context.Context) (*gpu.Info, error) { + return gpu.New() +} + +func (lfs *localHwReader) Memory(_ context.Context) (*memory.Info, error) { + return memory.New() +} + +func parseCPUInfo(ctx context.Context) (v1.CPUInfoS, error) { + if err := ctx.Err(); err != nil { + return nil, err + } + + hw := HWInfoFromCtx(ctx) + + cpus, err := hw.CPU(ctx) + if err != nil { + return nil, err + } + + res := make(v1.CPUInfoS, 0, len(cpus.Processors)) + + for _, c := range cpus.Processors { + res = append(res, v1.CPUInfo{ + ID: strconv.Itoa(c.ID), + Vendor: c.Vendor, + Model: c.Model, + Vcores: c.NumThreads, + }) + } + + return res, nil +} + +func parseGPUInfo(ctx context.Context) (v1.GPUInfoS, error) { + if err := ctx.Err(); err != nil { + return nil, err + } + + hw := HWInfoFromCtx(ctx) + + gpus, err := hw.GPU(ctx) + if err != nil { + return nil, err + } + + res := make(v1.GPUInfoS, 0) + + for _, dev := range gpus.GraphicsCards { + vendor, exists := supportedGPUs[dev.DeviceInfo.Vendor.ID] + if !exists { + continue + } + + model, exists := vendor.Devices[dev.DeviceInfo.Product.ID] + if !exists { + continue + } + + res = append(res, v1.GPUInfo{ + Vendor: dev.DeviceInfo.Vendor.Name, + Name: dev.DeviceInfo.Product.Name, + ModelID: dev.DeviceInfo.Product.ID, + Interface: model.Interface, + MemorySize: model.MemorySize, + }) + } + + return res, nil +} + +func (dp *debuggerPod) CPU(ctx context.Context) (*cpu.Info, error) { + respch := make(chan dpReadResp, 1) + + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-dp.ctx.Done(): + return nil, dp.ctx.Err() + case dp.readch <- dpReadReq{ + op: dpReqCPU, + resp: respch, + }: + } + + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-dp.ctx.Done(): + return nil, dp.ctx.Err() + case resp := <-respch: + return resp.data.(*cpu.Info), resp.err + } +} + +func (dp *debuggerPod) GPU(ctx context.Context) (*gpu.Info, error) { + respch := make(chan dpReadResp, 1) + + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-dp.ctx.Done(): + return nil, dp.ctx.Err() + case dp.readch <- dpReadReq{ + op: dpReqGPU, + resp: respch, + }: + } + + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-dp.ctx.Done(): + return nil, dp.ctx.Err() + case resp := <-respch: + return resp.data.(*gpu.Info), resp.err + } +} + +func (dp *debuggerPod) Memory(ctx context.Context) (*memory.Info, error) { + respch := make(chan dpReadResp, 1) + + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-dp.ctx.Done(): + return nil, dp.ctx.Err() + case dp.readch <- dpReadReq{ + op: dpReqMem, + resp: respch, + }: + } + + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-dp.ctx.Done(): + return nil, dp.ctx.Err() + case resp := <-respch: + return resp.data.(*memory.Info), resp.err + } +} + +func (dp *debuggerPod) run(startch chan<- struct{}) error { + log := LogFromCtx(dp.ctx) + + log.Info("staring debugger pod") + + req := &corev1.Pod{ + TypeMeta: metav1.TypeMeta{ + Kind: "Pod", + APIVersion: "v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: "fd-debugger-pod", + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: "psutil", + Image: "ghcr.io/akash-network/provider-test:latest-arm64", + Command: []string{ + "provider-services", + "operator", + "psutil", + "serve", + }, + Ports: []corev1.ContainerPort{ + { + Name: "api", + ContainerPort: 8081, + }, + }, + }, + }, + }, + } + + kc := KubeClientFromCtx(dp.ctx) + + pod, err := kc.CoreV1().Pods(daemonSetNamespace).Create(dp.ctx, req, metav1.CreateOptions{}) + if err != nil && !kerrors.IsAlreadyExists(err) { + return err + } + + defer func() { + // using default context here to delete pod as main might have been canceled + _ = kc.CoreV1().Pods(daemonSetNamespace).Delete(context.Background(), pod.Name, metav1.DeleteOptions{}) + }() + + watcher, err := kc.CoreV1().Pods(daemonSetNamespace).Watch(dp.ctx, metav1.ListOptions{ + Watch: true, + ResourceVersion: pod.ResourceVersion, + FieldSelector: fields.Set{"metadata.name": pod.Name}.AsSelector().String(), + LabelSelector: labels.Everything().String(), + }) + + if err != nil { + return err + } + + defer func() { + watcher.Stop() + }() + + var apiPort int32 + + for _, container := range pod.Spec.Containers { + if container.Name == "psutil" { + for _, port := range container.Ports { + if port.Name == "api" { + apiPort = port.ContainerPort + } + } + } + } + + if apiPort == 0 { + return fmt.Errorf("debugger pod does not have port named \"api\"") + } + +initloop: + for { + select { + case <-dp.ctx.Done(): + return dp.ctx.Err() + case evt := <-watcher.ResultChan(): + resp := evt.Object.(*corev1.Pod) + if resp.Status.Phase != corev1.PodPending { + watcher.Stop() + startch <- struct{}{} + break initloop + } + } + } + + for { + select { + case <-dp.ctx.Done(): + return dp.ctx.Err() + case readreq := <-dp.readch: + var res string + resp := dpReadResp{} + + switch readreq.op { + case dpReqCPU: + res = "cpu" + case dpReqGPU: + res = "gpu" + case dpReqMem: + res = "memory" + } + + result := kc.CoreV1().RESTClient().Get(). + Namespace(daemonSetNamespace). + Resource("pods"). + Name(fmt.Sprintf("%s:%d", pod.Name, apiPort)). + SubResource("proxy"). + Suffix(res). + Do(dp.ctx) + + resp.err = result.Error() + + if resp.err == nil { + var data []byte + data, resp.err = result.Raw() + if resp.err == nil { + switch readreq.op { + case dpReqCPU: + var res cpu.Info + resp.err = json.Unmarshal(data, &res) + resp.data = &res + case dpReqGPU: + var res gpu.Info + resp.err = json.Unmarshal(data, &res) + resp.data = &res + case dpReqMem: + var res memory.Info + resp.err = json.Unmarshal(data, &res) + resp.data = &res + } + } + } + + readreq.resp <- resp + } + } +} + +// // ExecCmd exec command on specific pod and wait the command's output. +// func ExecCmd(ctx context.Context, podName string, command string, stdin io.Reader, stdout io.Writer, stderr io.Writer) error { +// kc := KubeClientFromCtx(ctx) +// cfg := KubeConfigFromCtx(ctx) +// +// cmd := []string{ +// "sh", +// "-c", +// command, +// } +// +// option := &corev1.PodExecOptions{ +// Command: cmd, +// Stdin: true, +// Stdout: true, +// Stderr: true, +// TTY: true, +// } +// if stdin == nil { +// option.Stdin = false +// } +// +// req := kc.CoreV1(). +// RESTClient(). +// Post(). +// Resource("pods"). +// Name(podName). +// Namespace(daemonSetNamespace). +// SubResource("exec"). +// VersionedParams(option, scheme.ParameterCodec) +// +// exec, err := remotecommand.NewSPDYExecutor(cfg, "POST", req.URL()) +// if err != nil { +// return err +// } +// err = exec.StreamWithContext(ctx, remotecommand.StreamOptions{ +// Stdin: stdin, +// Stdout: stdout, +// Stderr: stderr, +// }) +// if err != nil { +// return err +// } +// +// return nil +// } diff --git a/operator/inventory/featureDiscoveryClient.go b/operator/inventory/featureDiscoveryClient.go deleted file mode 100644 index 93b5769c..00000000 --- a/operator/inventory/featureDiscoveryClient.go +++ /dev/null @@ -1,369 +0,0 @@ -package inventory - -import ( - "context" - "encoding/json" - "fmt" - "log" - "net/http" - "sync" - "time" - - corev1 "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/client-go/kubernetes" - "k8s.io/client-go/rest" - - "google.golang.org/grpc" - - "github.com/gorilla/mux" - - v1 "github.com/akash-network/akash-api/go/inventory/v1" -) - -const ( - daemonSetLabelSelector = "app=hostfeaturediscovery" - daemonSetNamespace = "akash-services" - grpcPort = ":50051" - nodeUpdateInterval = 5 * time.Second // Duration after which to print the cluster state - Added = "ADDED" - Deleted = "DELETED" -) - -var instance *ConcurrentClusterData -var once sync.Once - -// ConcurrentClusterData provides a concurrency-safe way to store and update cluster data. -type ConcurrentClusterData struct { - sync.RWMutex - cluster *v1.Cluster - podNodeMap map[string]int // Map of pod UID to node index in the cluster.Nodes slice -} - -// NewConcurrentClusterData initializes a new instance of ConcurrentClusterData with empty cluster data. -func NewConcurrentClusterData() *ConcurrentClusterData { - return &ConcurrentClusterData{ - cluster: &v1.Cluster{Nodes: []v1.Node{}}, - podNodeMap: make(map[string]int), - } -} - -// UpdateNode updates or adds the node to the cluster data. -func (ccd *ConcurrentClusterData) UpdateNode(podUID string, node *v1.Node) { - ccd.Lock() - defer ccd.Unlock() - - if nodeIndex, ok := ccd.podNodeMap[podUID]; ok { - // Node exists, update it - ccd.cluster.Nodes[nodeIndex] = *node - } else { - // Node does not exist, add it - ccd.cluster.Nodes = append(ccd.cluster.Nodes, *node) - ccd.podNodeMap[podUID] = len(ccd.cluster.Nodes) - 1 - } -} - -func (ccd *ConcurrentClusterData) RemoveNode(podUID string) { - ccd.Lock() - defer ccd.Unlock() - - if nodeIndex, ok := ccd.podNodeMap[podUID]; ok { - // Remove the node from the slice - ccd.cluster.Nodes = append(ccd.cluster.Nodes[:nodeIndex], ccd.cluster.Nodes[nodeIndex+1:]...) - delete(ccd.podNodeMap, podUID) // Remove the entry from the map - - // Update the indices in the map - for podUID, index := range ccd.podNodeMap { - if index > nodeIndex { - ccd.podNodeMap[podUID] = index - 1 - } - } - } -} - -// Helper function to perform a deep copy of the Cluster struct. -func deepCopy(cluster *v1.Cluster) *v1.Cluster { - if cluster == nil { - return nil - } - - if len(cluster.Nodes) == 0 { - // Log a warning instead of returning an error - log.Printf("Warning: Attempting to deep copy a cluster with an empty Nodes slice") - } - - // Create a new Cluster instance - copied := &v1.Cluster{} - - // Deep copy each field from the original Cluster to the new instance - // Deep copy the Nodes slice - copied.Nodes = make([]v1.Node, len(cluster.Nodes)) - for i, node := range cluster.Nodes { - // Assuming Node is a struct, create a copy - // If Node contains slices or maps, this process needs to be recursive - copiedNode := node // This is a shallow copy, adjust as needed - copied.Nodes[i] = copiedNode - } - - return copied -} - -func watchPods(clientset *kubernetes.Clientset, stopCh <-chan struct{}, clusterData *ConcurrentClusterData) error { - errCh := make(chan error, 1) // Buffered error channel - var wg sync.WaitGroup // WaitGroup to track goroutines - - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - - watcher, err := clientset.CoreV1().Pods(daemonSetNamespace).Watch(ctx, metav1.ListOptions{ - LabelSelector: daemonSetLabelSelector, - }) - - if err != nil { - return fmt.Errorf("error setting up Kubernetes watcher: %w", err) - } - - defer watcher.Stop() - - for { - select { - case <-stopCh: - log.Println("Stopping pod watcher") - wg.Wait() // Wait for all goroutines to finish - close(errCh) // Close the error channel - return nil - case err := <-errCh: - log.Printf("Error in goroutine: %v", err) - // Additional error handling logic can be placed here - case event, ok := <-watcher.ResultChan(): - if !ok { - wg.Wait() // Wait for all goroutines to finish - close(errCh) // Close the error channel - return fmt.Errorf("watcher channel closed unexpectedly") - } - - pod, ok := event.Object.(*corev1.Pod) - if !ok { - log.Println("Unexpected type in watcher event") - continue - } - - switch event.Type { - case Added: - if pod.Status.Phase == corev1.PodRunning && pod.Status.PodIP != "" { - wg.Add(1) - go func() { - defer wg.Done() - if err := connectToGrpcStream(pod, clusterData); err != nil { - errCh <- err - } - }() - } else { - wg.Add(1) - go func() { - defer wg.Done() - if err := waitForPodReadyAndConnect(clientset, pod, clusterData); err != nil { - errCh <- err - } - }() - } - log.Printf("Pod added: %s, UID: %s\n", pod.Name, pod.UID) - case Deleted: - clusterData.RemoveNode(string(pod.UID)) - log.Printf("Pod deleted: %s, UID: %s\n", pod.Name, pod.UID) - } - } - } -} - -// waitForPodReadyAndConnect waits for a pod to become ready before attempting to connect to its gRPC stream -func waitForPodReadyAndConnect(clientset *kubernetes.Clientset, pod *corev1.Pod, clusterData *ConcurrentClusterData) error { - ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) // 10-minute timeout - defer cancel() - - ticker := time.NewTicker(2 * time.Second) // Polling interval - defer ticker.Stop() - - for { - select { - case <-ctx.Done(): - return fmt.Errorf("timeout waiting for pod %s to become ready", pod.Name) - - case <-ticker.C: - currentPod, err := clientset.CoreV1().Pods(pod.Namespace).Get(ctx, pod.Name, metav1.GetOptions{}) - if err != nil { - return fmt.Errorf("error getting pod status: %w", err) - } - - if currentPod.Status.Phase == corev1.PodRunning && currentPod.Status.PodIP != "" { - // Handle the error returned by connectToGrpcStream - if err := connectToGrpcStream(currentPod, clusterData); err != nil { - return fmt.Errorf("error connecting to gRPC stream for pod %s: %w", pod.Name, err) - } - return nil - } - } - } -} - -func connectToGrpcStream(pod *corev1.Pod, clusterData *ConcurrentClusterData) error { - ipAddress := fmt.Sprintf("%s%s", pod.Status.PodIP, grpcPort) - fmt.Println("Connecting to:", ipAddress) - - // Establish the gRPC connection - conn, err := grpc.Dial(ipAddress, grpc.WithInsecure(), grpc.WithBlock()) - if err != nil { - return fmt.Errorf("failed to connect to pod IP %s: %v", pod.Status.PodIP, err) - } - defer conn.Close() - - client := v1.NewMsgClient(conn) - - // Create a stream to receive updates from the node - stream, err := client.QueryNode(context.Background(), &v1.VoidNoParam{}) - if err != nil { - return fmt.Errorf("could not query node for pod IP %s: %v", pod.Status.PodIP, err) - } - - for { - node, err := stream.Recv() - if err != nil { - // Handle stream error and remove the node - clusterData.RemoveNode(string(pod.UID)) - return fmt.Errorf("stream closed for pod UID %s: %v", pod.UID, err) - } - - // Update the node information in the cluster data - clusterData.UpdateNode(string(pod.UID), node) - } -} - -func printCluster() { - // Retrieve a deep copy of the current cluster state - cluster := GetCurrentClusterState() - - // If no nodes to print, just return - if len(cluster.Nodes) == 0 { - fmt.Println("No nodes in the cluster.") - return - } - - // Print the cluster state - jsonCluster, err := json.Marshal(cluster) - if err != nil { - log.Fatalf("error marshaling cluster struct into JSON: %v", err) - } - - fmt.Println(string(jsonCluster)) -} - -func FeatureDiscovery(ctx context.Context) error { - fmt.Println("Starting up gRPC client...") - - // Use in-cluster configuration - config, err := rest.InClusterConfig() - if err != nil { - return fmt.Errorf("error obtaining in-cluster config: %v", err) - } - - clientset, err := kubernetes.NewForConfig(config) - if err != nil { - return fmt.Errorf("error creating Kubernetes client: %v", err) - } - - clusterData := GetInstance() - - var wg sync.WaitGroup - - // Start the watcher in a goroutine with error handling - errCh := make(chan error, 1) - stopCh := make(chan struct{}) - wg.Add(1) - go func() { - defer wg.Done() - defer close(errCh) - if err := watchPods(clientset, stopCh, clusterData); err != nil { - errCh <- err - } - }() - - // Error handling goroutine - go func() { - for err := range errCh { - // Log errors but don't exit - log.Printf("Error from watchPods: %v", err) - } - }() - - // Start a ticker to periodically check/print the cluster state - ticker := time.NewTicker(nodeUpdateInterval) - go func() { - for { - select { - case <-ticker.C: - printCluster() - case <-ctx.Done(): - // Context canceled, cleanup and exit - ticker.Stop() - return - } - } - }() - - // API endpoint which serves feature discovery data to Akash Provider - router := mux.NewRouter() - router.HandleFunc("/getClusterState", getClusterStateHandler).Methods("GET") - - // Use a separate goroutine for HTTP server - httpErrCh := make(chan error, 1) - go func() { - httpErrCh <- http.ListenAndServe(":8081", router) - }() - - // Wait for all goroutines to finish or for context cancellation - select { - case err := <-httpErrCh: - return fmt.Errorf("HTTP server error: %v", err) - case err := <-errCh: - return err - case <-ctx.Done(): - close(stopCh) - wg.Wait() - return ctx.Err() - } -} - -// GetInstance returns the singleton instance of ConcurrentClusterData. -func GetInstance() *ConcurrentClusterData { - once.Do(func() { - log.Println("Initializing ConcurrentClusterData instance") - instance = &ConcurrentClusterData{ - cluster: &v1.Cluster{Nodes: []v1.Node{}}, - podNodeMap: make(map[string]int), - } - }) - return instance -} - -// GetCurrentClusterState returns a deep copy of the current state of the cluster and is used primarily for API GET data -func GetCurrentClusterState() *v1.Cluster { - // Use the singleton instance to get the cluster - clusterData := GetInstance() - - // Return a deep copy of the cluster - return deepCopy(clusterData.cluster) -} - -func getClusterStateHandler(w http.ResponseWriter, r *http.Request) { - clusterState := GetCurrentClusterState() - - w.Header().Set("Content-Type", "application/json") - if err := json.NewEncoder(w).Encode(clusterState); err != nil { - // Log the error - log.Printf("Error encoding response: %v", err) - - // Write an error response - http.Error(w, fmt.Sprintf("{\"error\": \"Internal Server Error: %v\"}", err), http.StatusInternalServerError) - return - } -} diff --git a/operator/inventory/gpu-info.json b/operator/inventory/gpu-info.json new file mode 100644 index 00000000..bdb9d5ec --- /dev/null +++ b/operator/inventory/gpu-info.json @@ -0,0 +1,31 @@ +{ + "10de":{ + "name": "nvidia", + "devices": { + "20b0": { + "interface": "SXM4", + "memory_size": "40GB" + }, + "20b1": { + "interface": "PCIe", + "memory_size": "40GB" + }, + "20b2": { + "interface": "SXM4", + "memory_size": "80GB" + }, + "20b3": { + "interface": "SXM", + "memory_size": "64GB" + }, + "20b5": { + "interface": "PCIe", + "memory_size": "80GB" + }, + "20f1": { + "interface": "PCIe", + "memory_size": "40GB" + } + } + } +} diff --git a/operator/inventory/rancher.go b/operator/inventory/rancher.go index c2c7d617..ca4572f5 100644 --- a/operator/inventory/rancher.go +++ b/operator/inventory/rancher.go @@ -21,7 +21,7 @@ type rancher struct { exe RemotePodCommandExecutor ctx context.Context cancel context.CancelFunc - querier + querierStorage } type rancherStorage struct { @@ -32,14 +32,14 @@ type rancherStorage struct { type rancherStorageClasses map[string]*rancherStorage -func NewRancher(ctx context.Context) (Storage, error) { +func NewRancher(ctx context.Context) (QuerierStorage, error) { ctx, cancel := context.WithCancel(ctx) r := &rancher{ - exe: NewRemotePodCommandExecutor(KubeConfigFromCtx(ctx), KubeClientFromCtx(ctx)), - ctx: ctx, - cancel: cancel, - querier: newQuerier(), + exe: NewRemotePodCommandExecutor(KubeConfigFromCtx(ctx), KubeClientFromCtx(ctx)), + ctx: ctx, + cancel: cancel, + querierStorage: newQuerierStorage(), } group := ErrGroupFromCtx(ctx) @@ -190,7 +190,7 @@ func (c *rancher) run() error { } } case req := <-c.reqch: - var resp resp + var resp respStorage if pvSynced { var res []akashv2beta2.InventoryClusterStorage diff --git a/operator/inventory/types.go b/operator/inventory/types.go index 33ef48b9..3bfc5293 100644 --- a/operator/inventory/types.go +++ b/operator/inventory/types.go @@ -3,6 +3,7 @@ package inventory import ( "context" + inventory "github.com/akash-network/akash-api/go/inventory/v1" "github.com/cskr/pubsub" rookexec "github.com/rook/rook/pkg/util/exec" @@ -14,13 +15,16 @@ import ( "k8s.io/client-go/tools/cache" providerflags "github.com/akash-network/provider/cmd/provider-services/cmd/flags" - akashv2beta1 "github.com/akash-network/provider/pkg/apis/akash.network/v2beta2" + akashv2beta2 "github.com/akash-network/provider/pkg/apis/akash.network/v2beta2" ) const ( FlagAPITimeout = "api-timeout" FlagQueryTimeout = "query-timeout" - FlagAPIPort = "api-port" + FlagRESTPort = "rest-port" + FlagGRPCPort = "grpc-port" + FlagPodName = "pod-name" + FlagNodeName = "node-name" ) type ContextKey string @@ -34,31 +38,52 @@ const ( CtxKeyLifecycle = ContextKey("lifecycle") CtxKeyErrGroup = ContextKey("errgroup") CtxKeyStorage = ContextKey("storage") + CtxKeyFeatureDiscovery = ContextKey("feature-discovery") CtxKeyInformersFactory = ContextKey("informers-factory") + CtxKeyHwInfo = ContextKey("hardware-info") ) -type resp struct { - res []akashv2beta1.InventoryClusterStorage +type respStorage struct { + res []akashv2beta2.InventoryClusterStorage err error } -type req struct { - respCh chan resp +type respNodes struct { + res inventory.Nodes + err error +} + +type reqStorage struct { + respCh chan respStorage +} + +type reqNodes struct { + respCh chan respNodes +} + +type querierStorage struct { + reqch chan reqStorage } -type querier struct { - reqch chan req +type querierNodes struct { + reqch chan reqNodes } -func newQuerier() querier { - return querier{ - reqch: make(chan req, 100), +func newQuerierStorage() querierStorage { + return querierStorage{ + reqch: make(chan reqStorage, 100), } } -func (c *querier) Query(ctx context.Context) ([]akashv2beta1.InventoryClusterStorage, error) { - r := req{ - respCh: make(chan resp, 1), +func newQuerierNodes() querierNodes { + return querierNodes{ + reqch: make(chan reqNodes, 100), + } +} + +func (c *querierStorage) Query(ctx context.Context) ([]akashv2beta2.InventoryClusterStorage, error) { + r := reqStorage{ + respCh: make(chan respStorage, 1), } select { @@ -75,8 +100,31 @@ func (c *querier) Query(ctx context.Context) ([]akashv2beta1.InventoryClusterSto } } -type Storage interface { - Query(ctx context.Context) ([]akashv2beta1.InventoryClusterStorage, error) +func (c *querierNodes) Query(ctx context.Context) (inventory.Nodes, error) { + r := reqNodes{ + respCh: make(chan respNodes, 1), + } + + select { + case c.reqch <- r: + case <-ctx.Done(): + return nil, ctx.Err() + } + + select { + case rsp := <-r.respCh: + return rsp.res, rsp.err + case <-ctx.Done(): + return nil, ctx.Err() + } +} + +type QuerierStorage interface { + Query(ctx context.Context) ([]akashv2beta2.InventoryClusterStorage, error) +} + +type QuerierNodes interface { + Query(ctx context.Context) (inventory.Nodes, error) } type Watcher interface { @@ -126,6 +174,7 @@ func InformKubeObjects(ctx context.Context, pubsub *pubsub.PubSub, informer cach } informer.Run(ctx.Done()) + return nil }) } diff --git a/operator/inventory/util.go b/operator/inventory/util.go index b5fb0d56..cc6c139a 100644 --- a/operator/inventory/util.go +++ b/operator/inventory/util.go @@ -93,11 +93,29 @@ func ErrGroupFromCtx(ctx context.Context) *errgroup.Group { return val.(*errgroup.Group) } -func StorageFromCtx(ctx context.Context) []Storage { +func StorageFromCtx(ctx context.Context) []QuerierStorage { val := ctx.Value(CtxKeyStorage) if val == nil { panic("context does not have storage set") } - return val.([]Storage) + return val.([]QuerierStorage) +} + +func FeatureDiscoveryFromCtx(ctx context.Context) QuerierNodes { + val := ctx.Value(CtxKeyFeatureDiscovery) + if val == nil { + panic("context does not have storage set") + } + + return val.(QuerierNodes) +} + +func HWInfoFromCtx(ctx context.Context) hwInfo { + val := ctx.Value(CtxKeyHwInfo) + if val == nil { + panic("context does not have file reader set") + } + + return val.(hwInfo) } diff --git a/operator/psutil.go b/operator/psutil.go new file mode 100644 index 00000000..d8c4fef5 --- /dev/null +++ b/operator/psutil.go @@ -0,0 +1,112 @@ +package operator + +import ( + "context" + "encoding/json" + "fmt" + "net" + "net/http" + + "github.com/gorilla/mux" + "github.com/jaypipes/ghw/pkg/cpu" + "github.com/jaypipes/ghw/pkg/gpu" + "github.com/jaypipes/ghw/pkg/memory" + "github.com/spf13/cobra" + "github.com/spf13/viper" +) + +const ( + flagApiPort = "api-port" +) + +func cmdPsutil() *cobra.Command { + cmd := &cobra.Command{ + Use: "psutil", + Short: "dump node hardware spec", + Args: cobra.ExactArgs(0), + SilenceUsage: true, + } + + cmd.AddCommand(cmdPsutilServe()) + + return cmd +} + +func cmdPsutilServe() *cobra.Command { + cmd := &cobra.Command{ + Use: "serve", + Short: "dump node hardware spec via REST", + Args: cobra.ExactArgs(0), + SilenceUsage: true, + RunE: func(cmd *cobra.Command, args []string) error { + router := mux.NewRouter() + router.HandleFunc("/cpu", cpuInfoHandler).Methods(http.MethodGet) + router.HandleFunc("/gpu", gpuHandler).Methods(http.MethodGet) + router.HandleFunc("/memory", memoryHandler).Methods(http.MethodGet) + + port := viper.GetUint16(flagApiPort) + + srv := &http.Server{ + Addr: fmt.Sprintf(":%d", port), + Handler: router, + BaseContext: func(_ net.Listener) context.Context { + return cmd.Context() + }, + } + + return srv.ListenAndServe() + }, + } + + cmd.Flags().Uint16(flagApiPort, 8080, "api port") + if err := viper.BindPFlag(flagApiPort, cmd.Flags().Lookup(flagApiPort)); err != nil { + panic(err) + } + + return cmd +} + +func cpuInfoHandler(w http.ResponseWriter, r *http.Request) { + res, err := cpu.New() + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + + writeJSON(w, res) +} + +func gpuHandler(w http.ResponseWriter, r *http.Request) { + res, err := gpu.New() + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + + writeJSON(w, res) +} + +func memoryHandler(w http.ResponseWriter, r *http.Request) { + res, err := memory.New() + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + + writeJSON(w, res) +} + +func writeJSON(w http.ResponseWriter, obj interface{}) { + bytes, err := json.Marshal(obj) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + + w.Header().Set("Content-Type", "application/json; charset=UTF-8") + + _, err = w.Write(bytes) + if err != nil { + return + } +} diff --git a/script/setup-kube.sh b/script/setup-kube.sh index 5fb001cd..ab70c9bd 100755 --- a/script/setup-kube.sh +++ b/script/setup-kube.sh @@ -120,7 +120,7 @@ install_crd() { set -x kubectl apply -f "$CRD_FILE" kubectl apply -f "$rootdir/_docs/kustomize/storage/storageclass.yaml" - kubectl patch node "${KIND_NAME}-control-plane" -p '{"metadata":{"labels":{"akash.network/storageclasses":"beta2.default"}}}' + kubectl patch node "${KIND_NAME}-control-plane" -p '{"metadata":{"labels":{"akash.network":"true","akash.network/storageclasses":"beta2.default"}}}' } install_metrics() { diff --git a/script/tools.sh b/script/tools.sh index 2d92baa3..5807313c 100755 --- a/script/tools.sh +++ b/script/tools.sh @@ -8,13 +8,24 @@ gomod="$SCRIPT_DIR/../go.mod" function get_gotoolchain() { local gotoolchain local goversion + local local_goversion gotoolchain=$(grep -E '^toolchain go[0-9]{1,}.[0-9]{1,}.[0-9]{1,}$' < "$gomod" | cut -d ' ' -f 2 | tr -d '\n') + goversion=$(grep -E '^go [0-9]{1,}.[0-9]{1,}(.[0-9]{1,})?$' < "$gomod" | cut -d ' ' -f 2 | tr -d '\n') if [[ ${gotoolchain} == "" ]]; then # determine go toolchain from go version in go.mod if which go > /dev/null 2>&1 ; then - goversion=$(GOTOOLCHAIN=local go version | cut -d ' ' -f 3 | sed 's/go*//' | tr -d '\n') + local_goversion=$(GOTOOLCHAIN=local go version | cut -d ' ' -f 3 | sed 's/go*//' | tr -d '\n') + if [[ $($SEMVER compare "v$local_goversion" v"$goversion") -ge 0 ]]; then + goversion=$local_goversion + else + local_goversion= + fi + fi + + if [[ "$local_goversion" == "" ]]; then + goversion=$(curl -s "https://go.dev/dl/?mode=json&include=all" | jq -r --arg regexp "^go$goversion" '.[] | select(.stable == true) | select(.version | match($regexp)) | .version' | head -n 1 | sed -e s/^go//) fi if [[ $goversion != "" ]] && [[ $($SEMVER compare "v$goversion" v1.21.0) -ge 0 ]]; then