Auto switch use iam-role/irsa when aws_access_key, aws_secret_key is …

…leaved null (#13)

* options usage for iam role, irsa and add examples use with irsa

* rename example k8s

* example: update image

* update function's name - get_aws_account_session_via_iam_user

* change function default's name

* update: boto3's behavior in exporter

---------

Co-authored-by: phat.ntp <phat.ntp@xyz>

.gitignore

@@ -1,3 +1,4 @@
 __pycache__
 *.pyc
 .DS_Store
+.venv

README.md

@@ -29,6 +29,18 @@ AWS Cost Metrics Exporter fetches cost data from a list of AWS accounts, each of
 
 ![aws-cost-exporter-design](doc/images/aws-cost-exporter-design.png)
 
+## How Does Exporter Use AWS Credentials
+This exporter works base on [Boto3 SDK](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials), with order is changed a littel as below:
+- Passing credentials as parameters in the boto.client() method, these parameters are defined in the `exporter_config.yaml` file as `aws_access_key` and `aws_secret_key`.
+- When both `aws_access_key` and `aws_secret_key` are set to null values in the `exporter_config.yaml` file, the subsequent priority order will be:
+   - Environment variables when export enviroment variables with `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`
+   - Shared credential file (~/.aws/credentials)
+   - AWS config file (~/.aws/config)
+   - Assume Role provider
+   - Assume Role With Web Identity Provider: example use IRSA on EKS
+   - Boto2 config file (/etc/boto.cfg and ~/.boto)
+   - Instance metadata service on an Amazon EC2 instance that has an IAM role configured.
+
 ## Setup AWS IAM User, Role, and Policy
 
 Note that if there is a list of AWS accounts for cost data collection, only **ONE** user needs to be created. This user is usually created in the AWS account where the exporter is deployed (an EKS cluster). This can be done from the AWS console - IAM portal or by terraform code.

app/exporter.py

@@ -42,7 +42,7 @@ def run_metrics_loop(self):
         while True:
             # every time we clear up all the existing labels before setting new ones
             self.aws_daily_cost_usd .clear()
-
+            
             for aws_account in self.targets:
                 logging.info("querying cost data for aws account %s" % aws_account["Publisher"])
                 try:
@@ -52,7 +52,7 @@ def run_metrics_loop(self):
                     continue
             time.sleep(self.polling_interval_seconds)
 
-    def get_aws_account_session(self, account_id):
+    def get_aws_account_session_via_iam_user(self, account_id):
         sts_client = boto3.client(
             "sts",
             aws_access_key_id=self.aws_access_key,
@@ -65,6 +65,17 @@ def get_aws_account_session(self, account_id):
 
         return assumed_role_object["Credentials"]
 
+    def get_aws_account_session_default(self, account_id):
+        sts_client = boto3.client(
+            "sts",
+        )
+
+        assumed_role_object = sts_client.assume_role(
+            RoleArn=f"arn:aws:iam::{account_id}:role/{self.aws_assumed_role_name}",RoleSessionName="AssumeRoleSession1"
+        )
+
+        return assumed_role_object["Credentials"]
+
     def query_aws_cost_explorer(self, aws_client, group_by):
         end_date = datetime.today()
         start_date = end_date - relativedelta(days=1)
@@ -83,7 +94,10 @@ def query_aws_cost_explorer(self, aws_client, group_by):
         return response["ResultsByTime"]
 
     def fetch(self, aws_account):
-        aws_credentials = self.get_aws_account_session(aws_account["Publisher"])
+        if self.aws_access_key == "" and self.aws_access_secret == "":
+            aws_credentials = self.get_aws_account_session_default(aws_account["Publisher"])
+        else:
+            aws_credentials = self.get_aws_account_session_via_iam_user(aws_account["Publisher"])
 
         aws_client = boto3.client(
             "ce",

deployment/k8s-with-eks/configmap.yaml

@@ -0,0 +1,41 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: aws-cost-exporter-config
+data:
+  exporter_config.yaml: |
+    exporter_port: $EXPORTER_PORT|9090 # the port that exposes cost metrics
+    polling_interval_seconds: $POLLING_INTERVAL_SECONDS|28800 # by default it is 8 hours because for daily cost, AWS only updates the data once per day
+    metric_name: aws_daily_cost_usd # change the metric name if needed
+    
+    aws_access_key: $AWS_ACCESS_KEY|"" # for prod deployment, DO NOT put the actual value here
+    aws_access_secret: $AWS_ACCESS_SECRET|"" # for prod deployment, DO NOT put the actual value here
+    aws_assumed_role_name: example-assumerole
+    
+    group_by:
+      enabled: true
+      # Cost data can be groupped using up to two different groups: DIMENSION, TAG, COST_CATEGORY.
+      # ref: https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_GetCostAndUsageWithResources.html
+      # note: label_name should be unique, and different from the labes in target_aws_accounts
+      groups:
+        - type: DIMENSION
+          key: SERVICE
+          label_name: ServiceName
+        - type: DIMENSION
+          key: REGION
+          label_name: RegionName
+      merge_minor_cost:
+        # if this is enabled, minor cost that is below the threshold will be merged into one group
+        enabled: false
+        threshold: 10
+        tag_value: other
+    
+    target_aws_accounts:
+      # here defines a list of target AWS accounts
+      # it should be guaranteed that all the AWS accounts have the same set of keys (in this example they are Publisher, ProjectName, and EnvironmentName)
+      - Publisher: 234567890123
+        ProjectName: dev-team-1
+        EnvironmentName: dev
+      - Publisher: 321645789123
+        ProjectName: dev-team-2
+        EnvironmentName: dev

deployment/k8s-with-eks/deployment.yaml

@@ -0,0 +1,59 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: aws-cost-exporter
+  labels:
+    app.kubernetes.io/name: aws-cost-exporter
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: aws-cost-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: aws-cost-exporter
+    spec:
+      serviceAccount: aws-cost-exporter
+      serviceAccountName: aws-cost-exporter
+      containers:
+        - name: aws-cost-exporter
+          image: "opensourceelectrolux/aws-cost-exporter:v1.0.1"
+          command: [ "python", "main.py", "-c", "/exporter_config.yaml" ]
+          imagePullPolicy: Always
+          ports:
+            - containerPort: 9090
+              name: metrics
+              protocol: TCP
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: metrics
+            failureThreshold: 10
+            initialDelaySeconds: 180
+            periodSeconds: 10
+            successThreshold: 1
+            timeoutSeconds: 1
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: metrics
+            failureThreshold: 10
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            successThreshold: 1
+            timeoutSeconds: 1
+          resources:
+            limits:
+              memory: 500Mi
+            requests:
+              cpu: 50m
+              memory: 300Mi
+          volumeMounts:
+          - name: config-volume
+            mountPath: /exporter_config.yaml
+            subPath: exporter_config.yaml
+      volumes:
+        - name: config-volume
+          configMap:
+            name: aws-cost-exporter-config

deployment/k8s-with-eks/sa.yaml

@@ -0,0 +1,7 @@
+apiVersion: v1
+automountServiceAccountToken: true
+kind: ServiceAccount
+metadata:
+  annotations:
+    eks.amazonaws.com/role-arn: arn:aws:iam::135468794354:role/aws-cost-exporter-irsa
+  name: aws-cost-exporter

deployment/k8s-with-eks/service.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1	
+kind: Service
+metadata:
+  name: aws-cost-exporter
+spec:
+  ports:
+  - port: 80
+    protocol: TCP
+    targetPort: metrics
+  selector:
+    app.kubernetes.io/name: aws-cost-exporter
+  type: ClusterIP

deployment/k8s/deployment.yaml

@@ -73,4 +73,4 @@ spec:
       volumes:
         - name: config-volume
           configMap:
-            name: aws-cost-exporter-config
+            name: aws-cost-exporter-config

exporter_config.yaml

@@ -2,9 +2,9 @@ exporter_port: $EXPORTER_PORT|9090 # the port that exposes cost metrics
 polling_interval_seconds: $POLLING_INTERVAL_SECONDS|28800 # by default it is 8 hours because for daily cost, AWS only updates the data once per day
 metric_name: aws_daily_cost_usd # change the metric name if needed
 
-aws_access_key: $AWS_ACCESS_KEY # for prod deployment, DO NOT put the actual value here
-aws_access_secret: $AWS_ACCESS_SECRET # for prod deployment, DO NOT put the actual value here
-aws_assumed_role_name: phoenix-service-cloud-cost-role
+aws_access_key: $AWS_ACCESS_KEY|"" # for prod deployment, DO NOT put the actual value here or default is null ("") to use iam-role/irsa
+aws_access_secret: $AWS_ACCESS_SECRET|"" # for prod deployment, DO NOT put the actual value here or default is set null ("") to use iam-role/irsa
+aws_assumed_role_name: example-assumerole
 
 group_by:
   enabled: true
@@ -27,6 +27,9 @@ group_by:
 target_aws_accounts:
   # here defines a list of target AWS accounts
   # it should be guaranteed that all the AWS accounts have the same set of keys (in this example they are Publisher, ProjectName, and EnvironmentName)
-  - Publisher: 123456789012
-    ProjectName: myproject
+  - Publisher: 234567890123
+    ProjectName: dev-team-1
+    EnvironmentName: dev
+  - Publisher: 321645789123
+    ProjectName: dev-team-2
     EnvironmentName: dev