ecs: Service Resource in a constant update loop

[dverveiko]

What happened?

While testing aws_ecs_service resource with Uptest, I observe constant update loop which leads to failing test.

Part of provider logs which represents an issue:

1.6777432133929799e+09	DEBUG	provider-aws	External resource is up to date	{"controller": "managed/ecs.aws.upbound.io/v1beta1, kind=service", "request": "/sample-service", "uid": "f7336824-e8f1-44e0-b92c-5407c9a91491", "version": "760940", "external-name": "sample-service", "requeue-after": 1677743813.3929758}
1.677743213414793e+09	DEBUG	provider-aws	Reconciling	{"controller": "managed/ecs.aws.upbound.io/v1beta1, kind=service", "request": "/sample-service"}
1.677743222428406e+09	DEBUG	provider-aws	refresh ended	{"workspace": "/var/folders/10/f_jw8phd19v96gx9s4dr763c0000gn/T/f7336824-e8f1-44e0-b92c-5407c9a91491", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.2.1\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:46:53.529016+02:00\",\"terraform\":\"1.2.1\",\"type\":\"version\",\"ui\":\"1.0\"}\n{\"@level\":\"info\",\"@message\":\"aws_ecs_service.sample-service: Refreshing state... [id=arn:aws:ecs:REDACTED:153891904029:service/example/sample-service]\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:01.554971+02:00\",\"hook\":{\"resource\":{\"addr\":\"aws_ecs_service.sample-service\",\"module\":\"\",\"resource\":\"aws_ecs_service.sample-service\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_ecs_service\",\"resource_name\":\"sample-service\",\"resource_key\":null},\"id_key\":\"id\",\"id_value\":\"arn:aws:ecs:REDACTED:153891904029:service/example/sample-service\"},\"type\":\"refresh_start\"}\n{\"@level\":\"info\",\"@message\":\"aws_ecs_service.sample-service: Refresh complete [id=arn:aws:ecs:REDACTED:153891904029:service/example/sample-service]\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:02.373876+02:00\",\"hook\":{\"resource\":{\"addr\":\"aws_ecs_service.sample-service\",\"module\":\"\",\"resource\":\"aws_ecs_service.sample-service\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_ecs_service\",\"resource_name\":\"sample-service\",\"resource_key\":null},\"id_key\":\"id\",\"id_value\":\"arn:aws:ecs:REDACTED:153891904029:service/example/sample-service\"},\"type\":\"refresh_complete\"}\n{\"@level\":\"info\",\"@message\":\"Plan: 0 to add, 0 to change, 0 to destroy.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:02.387590+02:00\",\"changes\":{\"add\":0,\"change\":0,\"remove\":0,\"operation\":\"plan\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Apply complete! Resources: 0 added, 0 changed, 0 destroyed.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:02.415503+02:00\",\"changes\":{\"add\":0,\"change\":0,\"remove\":0,\"operation\":\"apply\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Outputs: 0\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:02.415549+02:00\",\"outputs\":{},\"type\":\"outputs\"}\n"}
1.677743222433541e+09	DEBUG	provider-aws	External resource is up to date	{"controller": "managed/ecs.aws.upbound.io/v1beta1, kind=service", "request": "/sample-service", "uid": "f7336824-e8f1-44e0-b92c-5407c9a91491", "version": "760951", "external-name": "sample-service", "requeue-after": 1677743822.433538}
1.677743222542592e+09	DEBUG	provider-aws	Reconciling	{"controller": "managed/ecs.aws.upbound.io/v1beta1, kind=service", "request": "/sample-service"}
1.677743238283701e+09	DEBUG	provider-aws	refresh ended	{"workspace": "/var/folders/10/f_jw8phd19v96gx9s4dr763c0000gn/T/f7336824-e8f1-44e0-b92c-5407c9a91491", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.2.1\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:02.690592+02:00\",\"terraform\":\"1.2.1\",\"type\":\"version\",\"ui\":\"1.0\"}\n{\"@level\":\"info\",\"@message\":\"aws_ecs_service.sample-service: Refreshing state... [id=arn:aws:ecs:REDACTED:153891904029:service/example/sample-service]\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:17.340786+02:00\",\"hook\":{\"resource\":{\"addr\":\"aws_ecs_service.sample-service\",\"module\":\"\",\"resource\":\"aws_ecs_service.sample-service\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_ecs_service\",\"resource_name\":\"sample-service\",\"resource_key\":null},\"id_key\":\"id\",\"id_value\":\"arn:aws:ecs:REDACTED:153891904029:service/example/sample-service\"},\"type\":\"refresh_start\"}\n{\"@level\":\"info\",\"@message\":\"aws_ecs_service.sample-service: Refresh complete [id=arn:aws:ecs:REDACTED:153891904029:service/example/sample-service]\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:18.224213+02:00\",\"hook\":{\"resource\":{\"addr\":\"aws_ecs_service.sample-service\",\"module\":\"\",\"resource\":\"aws_ecs_service.sample-service\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_ecs_service\",\"resource_name\":\"sample-service\",\"resource_key\":null},\"id_key\":\"id\",\"id_value\":\"arn:aws:ecs:REDACTED:153891904029:service/example/sample-service\"},\"type\":\"refresh_complete\"}\n{\"@level\":\"info\",\"@message\":\"Plan: 0 to add, 0 to change, 0 to destroy.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:18.238743+02:00\",\"changes\":{\"add\":0,\"change\":0,\"remove\":0,\"operation\":\"plan\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Apply complete! Resources: 0 added, 0 changed, 0 destroyed.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:18.271153+02:00\",\"changes\":{\"add\":0,\"change\":0,\"remove\":0,\"operation\":\"apply\"},\"type\":\"change_summary\"}\n{\"@level\":\"info\",\"@message\":\"Outputs: 0\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:18.271198+02:00\",\"outputs\":{},\"type\":\"outputs\"}\n"}
1.677743248467985e+09	DEBUG	provider-aws	plan ended	{"workspace": "/var/folders/10/f_jw8phd19v96gx9s4dr763c0000gn/T/f7336824-e8f1-44e0-b92c-5407c9a91491", "out": "{\"@level\":\"info\",\"@message\":\"Terraform 1.2.1\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:18.342925+02:00\",\"terraform\":\"1.2.1\",\"type\":\"version\",\"ui\":\"1.0\"}\n{\"@level\":\"info\",\"@message\":\"aws_ecs_service.sample-service: Plan to update\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:28.453806+02:00\",\"change\":{\"resource\":{\"addr\":\"aws_ecs_service.sample-service\",\"module\":\"\",\"resource\":\"aws_ecs_service.sample-service\",\"implied_provider\":\"aws\",\"resource_type\":\"aws_ecs_service\",\"resource_name\":\"sample-service\",\"resource_key\":null},\"action\":\"update\"},\"type\":\"planned_change\"}\n{\"@level\":\"info\",\"@message\":\"Plan: 0 to add, 1 to change, 0 to destroy.\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-03-02T09:47:28.454478+02:00\",\"changes\":{\"add\":0,\"change\":1,\"remove\":0,\"operation\":\"plan\"},\"type\":\"change_summary\"}\n"}
1.677743248468914e+09	DEBUG	provider-aws	Successfully requested update of external resource	{"controller": "managed/ecs.aws.upbound.io/v1beta1, kind=service", "request": "/sample-service", "uid": "f7336824-e8f1-44e0-b92c-5407c9a91491", "version": "760962", "external-name": "sample-service", "requeue-after": 1677743848.468889}

What environment did it happen in?

• Universal Crossplane Version:
• Provider Version: 4.56.0

This bug relates to: #575 (PR #577)

[ulucinar]

Hi @dverveiko,
Could you please also link this issue to the version bump PR where we observed it? Thank you very much in advance.

[svscheg]

The issue can be reproduced.

[dverveiko]

Unfortunately can't fix.

[mhoshi-vm]

As written in #825 the issue seems to happen when using spec.taskDefinitionRef
The resolved taskDefintion currently is not an arn nor it includes the revision number.

In this situation I see that the at the terraform plan phase it is always trying to remove the revision from the taskDefinition.
This is leading to the situation on the service gets recreated on every reconcile loop. (This leading to a service getting restarted on every 5min)

  # aws_ecs_service.test2-ecs-ns1 will be updated in-place
  ~ resource "aws_ecs_service" "test2-ecs-ns1" {
        id                                 = "arn:aws:ecs:us-west-2:XXXXXXX:service/hoge/test2-ecs-ns1"
        name                               = "test2-ecs-ns1"
        tags                               = {
            "crossplane-kind"           = "service.ecs.aws.upbound.io"
            "crossplane-name"           = "test2-ecs-ns1"
            "crossplane-providerconfig" = "aws-provider"
        }
      ~ task_definition                    = "test2-ecs-ns1:16" -> "test2-ecs-ns1"
        # (16 unchanged attributes hidden)



        # (3 unchanged blocks hidden)
    }

I have raised PR #826 where it tries to return the ARN for external name.
With the help of this, the taskDefinition is now resolved to an ARN.

      taskDefinition: arn:aws:ecs:us-west-2:XXXXXX:task-definition/test2-ecs-ns1:59
      taskDefinitionRef:
        name: test2-ecs-ns1
        policy:
          resolution: Required
          resolve: Always

With the help of this I now see the service running more than the reconcile loop (at least for 21 hours in the following screenshot)

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions]

This issue is being closed since there has been no activity for 14 days since marking it as stale. If you still need help, feel free to comment or reopen the issue!