resource creation failing with securityContext "readOnlyRootFilesystem: true" option in controllerconfig

[Swapnil-CSI]

What happened?

I am using below controllerconfig with aws official provider.

[swapnil@bharshankar aws-official]$ cat controllerconfig.yaml
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: aws-official
  annotations:
    eks.amazonaws.com/role-arn: "<role arn>"
spec:
  args:
  - --debug
  serviceAccountName: aws-provider-official
  podSecurityContext:
    fsGroup: 2000
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    readOnlyRootFilesystem: true
    runAsNonRoot: true
  resources:
    limits:
      cpu: 200m
      memory: 512Mi
    requests:
      cpu: 200m
      memory: 512Mi
[swapnil@bharshankar aws-official]$

When I tried to create a resource it showing me below error in pod logs

2023-09-21T13:50:54Z    INFO    Starting workers        {"controller": "managed/autoscaling.aws.upbound.io/v1beta1, kind=autoscalinggroup", "worker count": 10}
2023-09-21T13:56:07Z    DEBUG   provider-aws    Calling the inner handler for Create event.     {"gvk": "autoscaling.aws.upbound.io/v1beta1, Kind=GroupTag", "name": "dev-nodegroup-8b8nq-vstsm", "queueLength": 0}
2023-09-21T13:56:07Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/autoscaling.aws.upbound.io/v1beta1, kind=grouptag", "request": {"name":"dev-nodegroup-8b8nq-vstsm"}}
2023-09-21T13:56:08Z    DEBUG   provider-aws    Cannot connect to provider      {"controller": "managed/autoscaling.aws.upbound.io/v1beta1, kind=grouptag", "request": {"name":"dev-nodegroup-8b8nq-vstsm"}, "uid": "aa53fb50-c50f-4a68-ae8e-ee6e9c2ab41b", "version": "596853", "external-name": "", "error": "cannot get a terraform workspace for resource: cannot create directory for workspace: mkdir /tmp/aa53fb50-c50f-4a68-ae8e-ee6e9c2ab41b: read-only file system", "errorVerbose": "mkdir /tmp/aa53fb50-c50f-4a68-ae8e-ee6e9c2ab41b: read-only file system\ncannot create directory for workspace\ngithub.com/upbound/upjet/pkg/terraform.(*WorkspaceStore).Workspace\n\tgithub.com/upbound/[email protected]/pkg/terraform/store.go:227\ngithub.com/upbound/upjet/pkg/controller.(*Connector).Connect\n\tgithub.com/upbound/[email protected]/pkg/controller/external.go:117\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:244\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:839\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226\nruntime.goexit\n\truntime/asm_amd64.s:1594\ncannot get a terraform workspace for resource\ngithub.com/upbound/upjet/pkg/controller.(*Connector).Connect\n\tgithub.com/upbound/[email protected]/pkg/controller/external.go:119\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:244\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:839\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226\nruntime.goexit\n\truntime/asm_amd64.s:1594"}
2023-09-21T13:56:08Z    DEBUG   events  cannot get a terraform workspace for resource: cannot create directory for workspace: mkdir /tmp/aa53fb50-c50f-4a68-ae8e-ee6e9c2ab41b: read-only file system   {"type": "Warning", "object": {"kind":"GroupTag","name":"dev-nodegroup-8b8nq-vstsm","uid":"aa53fb50-c50f-4a68-ae8e-ee6e9c2ab41b","apiVersion":"autoscaling.aws.upbound.io/v1beta1","resourceVersion":"596853"}, "reason": "CannotConnectToProvider"}

I tried to create a sample file on /tmp dir on pod it is showing readonly file system error.

aws-official-0db1298ae3e5-d598dd569-l2j4p:/$ touch /tmp/test
touch: /tmp/test: Read-only file system
aws-official-0db1298ae3e5-d598dd569-l2j4p:/$

It works fine if I do not use "readOnlyRootFilesystem" option in controllerconfig.

[swapnil@bharshankar awsnodegroup]$ kubectl get grouptag
NAME                        READY   SYNCED   EXTERNAL-NAME                                                          AGE
dev-nodegroup-qrg7l-lqjpp   True    True     xxxxxxxxxx-eac55e22-c650-33fb-d698-a4c45c414e5c,Name   3m33s
[swapnil@bharshankar awsnodegroup]$

Also, I am able to create a file in /tmp dir.

aws-official-0db1298ae3e5-68d4455459-69drk:/tmp$ touch test
aws-official-0db1298ae3e5-68d4455459-69drk:/tmp$ ls
6246027a-8154-4fa5-b039-5dda5708582e  test                                     plugin3029160287
aws-official-0db1298ae3e5-68d4455459-69drk:/tmp$

How can we reproduce it?

use securityContext readOnlyRootFilesystem: true option in controllerconfig

What environment did it happen in?

• Crossplane Version: 1.13.2
• Provider Version: 0.40.0
• Kubernetes Version:
• Kubernetes Distribution: EKS

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions]

This issue is being closed since there has been no activity for 14 days since marking it as stale. If you still need help, feel free to comment or reopen the issue!