FOPTS-5266 Added pagination to Kubecost Allocation API requests (#56)

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## v1.21.0
+
+-   Added pagination for Kubecost Allocation API requests.
+
 ## v1.20.0
 
 -   Setting the default value of the INCLUDE_PREVIOUS_MONTH to true.

README.md

@@ -182,7 +182,7 @@ You should see 200/201s in the logs, which indicates that the exporter is workin
 | flexera.vendorName | string | `"Kubecost"` | Vendor name for the Bill Connect. It is used when CREATE_BILL_CONNECT_IF_NOT_EXIST is set to true. |
 | image.pullPolicy | string | `"Always"` |  |
 | image.repository | string | `"public.ecr.aws/flexera/cbi-oi-kubecost-exporter"` |  |
-| image.tag | string | `"1.17"` |  |
+| image.tag | string | `"1.21"` |  |
 | imagePullSecrets | list | `[]` |  |
 | includePreviousMonth | bool | `true` | Indicates whether to collect and export previous month data. Default is true. Setting this flag to false will prevent collecting and uploading the data from previous month and only upload data for the current month. Partial Data (i.e. missing data for some days) for previous month will not be uploaded even if the flag value is set to true.|
 | kubecost.aggregation | string | `"pod"` | The level of granularity to use when aggregating the cost data. Valid values are namespace, controller, node, or pod. |
@@ -191,6 +191,7 @@ You should see 200/201s in the logs, which indicates that the exporter is workin
 | kubecost.idle | bool | `true` | Indicates whether to include cost of idle resources. |
 | kubecost.idleByNode | bool | `false` | Indicates whether idle allocations are created on a per node basis. |
 | kubecost.multiplier | float | `1` | Optional multiplier for costs. |
+| kubecost.pageSize | int | `500` | Indicates the pagination limit |
 | kubecost.shareIdle | bool | `false` | Indicates whether allocate idle cost proportionally across non-idle resources. |
 | kubecost.shareNamespaces | string | `"kube-system,cadvisor"` | Comma-separated list of namespaces to share costs with the remaining non-idle, unshared allocations. |
 | kubecost.shareTenancyCosts | bool | `true` | Indicates whether to share the cost of cluster overhead assets across tenants of those resources. |
@@ -210,6 +211,7 @@ Below are the parameters used in the Helm configuration along with the environme
 | kubecost.shareIdle | SHARE_IDLE | `shareIdle` | - | Indicates whether to allocate idle cost proportionally across non-idle resources. |
 | kubecost.shareNamespaces | SHARE_NAMESPACES | `shareNamespaces` | - | Specifies a comma-separated list of namespaces to share costs with the remaining non-idle, unshared allocations. |
 | kubecost.shareTenancyCosts | SHARE_TENANCY_COSTS | `shareTenancyCosts` | - | Indicates whether to share the cost of cluster overhead assets across tenants of those resources. |
+| kubecost.pageSize | int | `500` | Indicates the pagination limit |
 
 ## License
 

helm-chart/Chart.yaml

@@ -6,10 +6,10 @@ description: Kubecost exporter helm chart for Kubernetes
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 1.20.0
+version: 1.21.0
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.
 # It is recommended to use it with quotes.
-appVersion: "1.20"
+appVersion: "1.21"

helm-chart/README.md

@@ -1,6 +1,6 @@
 # cbi-oi-kubecost-exporter
 
-![Version: 1.20.0](https://img.shields.io/badge/Version-1.20.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.20](https://img.shields.io/badge/AppVersion-1.20-informational?style=flat-square)
+![Version: 1.21.0](https://img.shields.io/badge/Version-1.21.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.21](https://img.shields.io/badge/AppVersion-1.21-informational?style=flat-square)
 
 ### Kubecost exporter helm chart for Kubernetes
 
@@ -100,7 +100,7 @@ You should see 200/201s in the logs, which indicates that the exporter is workin
 | flexera.vendorName | string | `"Kubecost"` | Vendor name for the Bill Connect. It is used when CREATE_BILL_CONNECT_IF_NOT_EXIST is set to true. |
 | image.pullPolicy | string | `"Always"` |  |
 | image.repository | string | `"public.ecr.aws/flexera/cbi-oi-kubecost-exporter"` |  |
-| image.tag | string | `"1.20"` |  |
+| image.tag | string | `"1.21"` |  |
 | imagePullSecrets | list | `[]` |  |
 | includePreviousMonth | bool | `true` | Indicates whether to collect and export previous month data. Default is true. Setting this flag to false will prevent collecting and uploading the data from previous month and only upload data for the current month. Partial Data (i.e. missing data for some days) for previous month will not be uploaded even if the flag value is set to true. |
 | kubecost.aggregation | string | `"pod"` | The level of granularity to use when aggregating the cost data. Valid values are namespace, controller, node, or pod. |
@@ -109,9 +109,11 @@ You should see 200/201s in the logs, which indicates that the exporter is workin
 | kubecost.idle | bool | `true` | Indicates whether to include cost of idle resources. |
 | kubecost.idleByNode | bool | `false` | Indicates whether idle allocations are created on a per node basis. |
 | kubecost.multiplier | float | `1` | Optional multiplier for costs. |
+| kubecost.pageSize | int | `500` | Indicates the pagination limit |
 | kubecost.shareIdle | bool | `false` | Indicates whether allocate idle cost proportionally across non-idle resources. |
 | kubecost.shareNamespaces | string | `"kube-system,cadvisor"` | Comma-separated list of namespaces to share costs with the remaining non-idle, unshared allocations. |
 | kubecost.shareTenancyCosts | bool | `true` | Indicates whether to share the cost of cluster overhead assets across tenants of those resources. |
 | persistentVolume.enabled | bool | `true` | Enable Persistent Volume. Recommended setting is true to prevent loss of historical data. |
 | persistentVolume.size | string | `"1Gi"` | Persistent Volume size. |
 | requestTimeout | int | `5` | Indicates the timeout per each request in minutes. |
+

helm-chart/templates/cronjob.yaml

@@ -63,6 +63,8 @@ spec:
                 value: "{{ .Values.kubecost.shareTenancyCosts }}"
               - name: MULTIPLIER
                 value: "{{ .Values.kubecost.multiplier }}"
+              - name: PAGE_SIZE
+                value: "{{ .Values.kubecost.pageSize }}"
               - name: FILE_ROTATION
                 value: "{{ .Values.fileRotation }}"
               - name: FILE_PATH

helm-chart/values.yaml

@@ -5,7 +5,7 @@
 image:
   repository: public.ecr.aws/flexera/cbi-oi-kubecost-exporter
   pullPolicy: Always
-  tag: "1.20"
+  tag: "1.21"
 
 imagePullSecrets: []
 
@@ -67,6 +67,8 @@ kubecost:
   shareTenancyCosts: true
   # -- Optional multiplier for costs.
   multiplier: 1.0
+  # -- Indicates the pagination limit
+  pageSize: 500
 
 # -- Indicates whether to delete files generated for previous months. Note: current and previous months data is kept.
 fileRotation: true

index.yaml

@@ -1,9 +1,19 @@
 apiVersion: v1
 entries:
   cbi-oi-kubecost-exporter:
+  - apiVersion: v2
+    appVersion: "1.21"
+    created: "2024-10-18T11:40:00.962201-06:00"
+    description: Kubecost exporter helm chart for Kubernetes
+    digest: 9889df72b4f9d1ea6557027558c8bbe2bad98677e73b8c250d74f9e46a3e4722
+    name: cbi-oi-kubecost-exporter
+    type: application
+    urls:
+    - helm-chart/cbi-oi-kubecost-exporter-1.21.0.tgz
+    version: 1.21.0
   - apiVersion: v2
     appVersion: "1.20"
-    created: "2024-10-11T12:29:59.261971-04:00"
+    created: "2024-10-18T11:40:00.961959-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 739eb4727978b2a4dfcff39866da362b64346f58928abcc14deeb469a732b13b
     name: cbi-oi-kubecost-exporter
@@ -13,7 +23,7 @@ entries:
     version: 1.20.0
   - apiVersion: v2
     appVersion: "1.19"
-    created: "2024-10-11T12:29:59.261443-04:00"
+    created: "2024-10-18T11:40:00.961225-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: b21fe46b0ea4590e67e38057751252fd3d3b07926adf2c2428d7f536afb3b9d5
     name: cbi-oi-kubecost-exporter
@@ -23,7 +33,7 @@ entries:
     version: 1.19.0
   - apiVersion: v2
     appVersion: "1.18"
-    created: "2024-10-11T12:29:59.260996-04:00"
+    created: "2024-10-18T11:40:00.960836-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: a78276488d28e78b40d2c38ab00217a2eeef34920ecfd13d794030843da4985d
     name: cbi-oi-kubecost-exporter
@@ -33,7 +43,7 @@ entries:
     version: 1.18.0
   - apiVersion: v2
     appVersion: "1.17"
-    created: "2024-10-11T12:29:59.258907-04:00"
+    created: "2024-10-18T11:40:00.960444-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: d672d4fdee8ca56f97faacaef48bcb246276433512fd3e53880a5496c37d0397
     name: cbi-oi-kubecost-exporter
@@ -43,7 +53,7 @@ entries:
     version: 1.17.0
   - apiVersion: v2
     appVersion: "1.16"
-    created: "2024-10-11T12:29:59.258361-04:00"
+    created: "2024-10-18T11:40:00.960041-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 676832f616592285b1db455ed66620c0453e7f506fc7fc46bf8214abe15efd3b
     name: cbi-oi-kubecost-exporter
@@ -53,7 +63,7 @@ entries:
     version: 1.16.0
   - apiVersion: v2
     appVersion: "1.15"
-    created: "2024-10-11T12:29:59.257986-04:00"
+    created: "2024-10-18T11:40:00.959659-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 1a717c15b88f6e6fb9d4fe083ca6517eb78258b160c064c13e7804e7f659e530
     name: cbi-oi-kubecost-exporter
@@ -63,7 +73,7 @@ entries:
     version: 1.15.0
   - apiVersion: v2
     appVersion: "1.14"
-    created: "2024-10-11T12:29:59.257507-04:00"
+    created: "2024-10-18T11:40:00.959285-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 781f81b9f708f8b86cb215bd5bd5ddf7a5782195b4f0e4b85a0265b0cb35c862
     name: cbi-oi-kubecost-exporter
@@ -73,7 +83,7 @@ entries:
     version: 1.14.0
   - apiVersion: v2
     appVersion: "1.13"
-    created: "2024-10-11T12:29:59.257034-04:00"
+    created: "2024-10-18T11:40:00.958999-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: bd1994ff4b3b30c623efb05b784f3851f7cb4583479b7c670dd19316cd8a4c02
     name: cbi-oi-kubecost-exporter
@@ -83,7 +93,7 @@ entries:
     version: 1.13.0
   - apiVersion: v2
     appVersion: "1.12"
-    created: "2024-10-11T12:29:59.256612-04:00"
+    created: "2024-10-18T11:40:00.958581-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: d1bfa13ac76037b1cc9c2223826056f1da801e59d4d54348504469f4760c458e
     name: cbi-oi-kubecost-exporter
@@ -93,7 +103,7 @@ entries:
     version: 1.12.0
   - apiVersion: v2
     appVersion: "1.11"
-    created: "2024-10-11T12:29:59.256257-04:00"
+    created: "2024-10-18T11:40:00.958167-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: f12e36c016cb0b3bca7d75efd69414004c4a0f238e550f74fd13daf015305370
     name: cbi-oi-kubecost-exporter
@@ -103,7 +113,7 @@ entries:
     version: 1.11.1
   - apiVersion: v2
     appVersion: "1.11"
-    created: "2024-10-11T12:29:59.255834-04:00"
+    created: "2024-10-18T11:40:00.957717-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: a079003a1bcc3207e1c4f93653caca2234212e2bebafae7acde25a9a2d185bc5
     name: cbi-oi-kubecost-exporter
@@ -113,7 +123,7 @@ entries:
     version: 1.11.0
   - apiVersion: v2
     appVersion: "1.10"
-    created: "2024-10-11T12:29:59.255355-04:00"
+    created: "2024-10-18T11:40:00.957308-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: e7f8ac15bb87b78cc14aa3a63a2e37af9db427ad1a059621958430b8121156cd
     name: cbi-oi-kubecost-exporter
@@ -123,7 +133,7 @@ entries:
     version: 1.10.0
   - apiVersion: v2
     appVersion: 1.9.1
-    created: "2024-10-11T12:29:59.265266-04:00"
+    created: "2024-10-18T11:40:00.96635-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 773cc828783dd12211cccf149cda5b3c95e5d5bf7bf507d73559ef5f0f66f13a
     name: cbi-oi-kubecost-exporter
@@ -133,7 +143,7 @@ entries:
     version: 1.9.1
   - apiVersion: v2
     appVersion: "1.9"
-    created: "2024-10-11T12:29:59.265038-04:00"
+    created: "2024-10-18T11:40:00.966005-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 312f1c8f469adbed664530059526b69c023c017f71cc6c7b8bd9aed523ef22d6
     name: cbi-oi-kubecost-exporter
@@ -143,7 +153,7 @@ entries:
     version: 1.9.0
   - apiVersion: v2
     appVersion: "1.8"
-    created: "2024-10-11T12:29:59.264809-04:00"
+    created: "2024-10-18T11:40:00.965644-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 3fc398d0ae6d2c44c9196d262042394763fc7df49f05bd97764637a9becaad11
     name: cbi-oi-kubecost-exporter
@@ -153,7 +163,7 @@ entries:
     version: 1.8.0
   - apiVersion: v2
     appVersion: "1.7"
-    created: "2024-10-11T12:29:59.264596-04:00"
+    created: "2024-10-18T11:40:00.965309-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 90fbd649c651fa2ad66d0705aced6cbecd908206c59c135ee585ce12188cd287
     name: cbi-oi-kubecost-exporter
@@ -163,7 +173,7 @@ entries:
     version: 1.7.0
   - apiVersion: v2
     appVersion: "1.6"
-    created: "2024-10-11T12:29:59.264379-04:00"
+    created: "2024-10-18T11:40:00.964996-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: f998790d9229fce36398ef487edf6685a9a29e359dd2c78191c4d16344d47acd
     name: cbi-oi-kubecost-exporter
@@ -173,7 +183,7 @@ entries:
     version: 1.6.0
   - apiVersion: v2
     appVersion: "1.5"
-    created: "2024-10-11T12:29:59.264157-04:00"
+    created: "2024-10-18T11:40:00.964611-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: a0503a1af7b86a8f0769009de5a71a3613e8503a5f59cf86874d483f0f5f2abb
     name: cbi-oi-kubecost-exporter
@@ -183,7 +193,7 @@ entries:
     version: 1.5.0
   - apiVersion: v2
     appVersion: "1.4"
-    created: "2024-10-11T12:29:59.263682-04:00"
+    created: "2024-10-18T11:40:00.964105-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 5b296e921178d84717345be19b92a4eea0c9ca1c65410128dec596692220c8e2
     name: cbi-oi-kubecost-exporter
@@ -193,7 +203,7 @@ entries:
     version: 1.4.2
   - apiVersion: v2
     appVersion: "1.4"
-    created: "2024-10-11T12:29:59.263326-04:00"
+    created: "2024-10-18T11:40:00.963654-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: ab6992a0af9f24d5c090db71679b7f274f3576b1cfbe91325e0d6e9af317657e
     name: cbi-oi-kubecost-exporter
@@ -203,7 +213,7 @@ entries:
     version: 1.4.1
   - apiVersion: v2
     appVersion: "1.4"
-    created: "2024-10-11T12:29:59.263032-04:00"
+    created: "2024-10-18T11:40:00.96338-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 870030e989ba274f8b0f9d122a4ada24a4d29ef2c071408bf46d86e18942effd
     name: cbi-oi-kubecost-exporter
@@ -213,7 +223,7 @@ entries:
     version: 1.4.0
   - apiVersion: v2
     appVersion: "1.3"
-    created: "2024-10-11T12:29:59.262815-04:00"
+    created: "2024-10-18T11:40:00.963058-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: 0d967927bfab5f4bb6d40ce062710679e98b145c4b3f38709c0bf5d216ad8606
     name: cbi-oi-kubecost-exporter
@@ -223,7 +233,7 @@ entries:
     version: 1.3.0
   - apiVersion: v2
     appVersion: "1.2"
-    created: "2024-10-11T12:29:59.261739-04:00"
+    created: "2024-10-18T11:40:00.961571-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: d0c724c7bb085a1801e27f60ff748bc69aba14425cfd27fd132f99966692ed80
     name: cbi-oi-kubecost-exporter
@@ -233,12 +243,12 @@ entries:
     version: 1.2.0
   - apiVersion: v2
     appVersion: "1.1"
-    created: "2024-10-11T12:29:59.254953-04:00"
+    created: "2024-10-18T11:40:00.955115-06:00"
     description: Kubecost exporter helm chart for Kubernetes
     digest: c6f2681575b704b5934efea2357921896d55c4ad5e09692f31ab92b9e614cfea
     name: cbi-oi-kubecost-exporter
     type: application
     urls:
     - helm-chart/cbi-oi-kubecost-exporter-1.1.0.tgz
     version: 1.1.0
-generated: "2024-10-11T12:29:59.254375-04:00"
+generated: "2024-10-18T11:40:00.954235-06:00"