Pulumi Refresh Unexpectedly Removing Datadog Resources

[andrew-purdin]

What happened?

Pulumi refresh is removing our IntegrationLambdaArn and IntegrationLogCollection across all of our stacks. The resources exist and have not been altered via the Datadog UI. This wasn't happening as of about a month ago, though we can observe the behavior on @pulumi/datadog 4.23 as well as 4.21.

Example

Output of pulumi about

I'm not comfortable posting all the information from pulumi about, the issue is consistent between my local machine and a remote pipeline however. Below is relevant version and plugin information.

pulumi plugin ls from local:

NAME          KIND      VERSION  SIZE    INSTALLED     LAST USED
aws           resource  6.14.0   562 MB  23 hours ago  23 hours ago
aws           resource  6.13.3   556 MB  5 days ago    2 days ago
aws           resource  6.12.3   551 MB  2 weeks ago   2 weeks ago
aws           resource  6.12.1   551 MB  1 week ago    1 week ago
aws           resource  6.9.0    594 MB  1 month ago   1 month ago
aws           resource  6.8.0    589 MB  1 month ago   1 month ago
aws           resource  6.7.0    585 MB  5 days ago    2 days ago
aws           resource  6.6.0    583 MB  1 month ago   1 month ago
aws           resource  6.5.0    580 MB  2 months ago  2 months ago
aws           resource  6.4.1    580 MB  1 month ago   1 month ago
aws           resource  6.2.1    576 MB  2 months ago  2 months ago
aws           resource  6.2.0    574 MB  3 months ago  3 months ago
aws           resource  5.42.0   542 MB  4 months ago  4 months ago
aws           resource  5.41.0   542 MB  6 months ago  5 months ago
aws           resource  5.40.0   526 MB  7 months ago  7 months ago
aws           resource  5.39.0   524 MB  7 months ago  7 months ago
aws           resource  5.38.0   524 MB  7 months ago  7 months ago
aws           resource  5.35.0   515 MB  8 months ago  8 months ago
aws           resource  5.34.0   514 MB  7 months ago  7 months ago
aws           resource  5.31.0   510 MB  7 months ago  7 months ago
aws           resource  5.16.2   419 MB  7 months ago  7 months ago
datadog       resource  4.23.0   72 MB   5 days ago    2 days ago
datadog       resource  4.21.0   69 MB   4 months ago  4 months ago
datadog       resource  4.20.0   66 MB   1 day ago     1 day ago

TOTAL plugin cache size: 13 GB

Snippets from pulumi about:

CLI          
Version      3.97.0
Go Version   go1.21.5
Go Compiler  gc

Plugins
NAME    VERSION
nodejs  unknown

Host     
OS       darwin
Version  14.2.1
Arch     arm64

This project is written in nodejs: executable='/Users/APurdin/.nvm/versions/node/v18.16.0/bin/node' version='v18.16.0'

Package.json dependencies:

{
  "engines": {
    "node": "18"
  },
  "name": "aws-datadog",
  "devDependencies": {
    "@types/node": "^18.16.2",
    "@typescript-eslint/eslint-plugin": "^6.8.0",
    "@typescript-eslint/parser": "^6.8.0",
    "eslint": "^8.51.0",
    "eslint-config-prettier": "^9.0.0",
    "eslint-plugin-jsdoc": "^46.8.2",
    "eslint-plugin-prettier": "^5.0.1",
    "husky": "^8.0.3",
    "prettier": "^3.0.3",
    "typescript": "^5.2.2"
  },
  "dependencies": {
    "@pulumi/aws": "^6.5.0",
    "@pulumi/datadog": "^4.23.0",
    "@pulumi/eks": "^1.0.3",
    "@pulumi/pulumi": "^3.88.1",
    "config": "^3.3.9"
  },

Additional context

This was working consistently for some time, so I suspect something has changed either in the upstream Terraform provider, or in the Datadog API.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[iwahbe]

Hi @andrew-purdin. I'm sorry to hear that. To confirm, are you saying that pulumi refresh used to work as expected (on v4.21.0), but now does not work as expected? Do you know when this stopped working?

---

May be related to DataDog/terraform-provider-datadog#2219.

[andrew-purdin]

Yeah @iwahbe that's correct. We first saw the breaking behavior via refresh on December 7th and our pipeline ran last before that on November 6th without the refresh causing any removals. That run did not include any version changes of the provider nor any other dependencies.

[iwahbe]

Thanks for confirming @andrew-purdin.

[andrew-purdin]

Hi @iwahbe, we were doing some more digging on this and we think the problem is caused by some API changes that were made by DataDog. (See 2146 and 2200 on the TF provider).

I'm not 100% certain, but I think we are unable to work around this until version 3.32.0 of the TF provider is integrated to the Pulumi provider because of this PR. I saw there is an open PR to jump to 3.33.0, but it's been stalled for a long time. Can we expect to see that released soon?

We have some Datadog integration resources we need to provision very soon, so I'd appreciate your timely help with this.

[andrew-purdin]

@iwahbe can you provide an update on this please?

[iwahbe]

Hi @andrew-purdin. Sorry for the lack of update. We are working on this now.

[andrew-purdin]

Thanks @iwahbe, I see your PR, do you think it will be merged soon?

[iwahbe]

Hey @andrew-purdin. We're going to merge today.

We were hesitant to merge before, since the change added about 1.5 million lines of generated code to the pulumi-datadog SDK. That makes certain libraries very heavy, which is not the experience we hope to give users.

We were hoping that we would have time to reduce the size of the SDK (this issue: pulumi/pulumi-terraform-bridge#1468), but that has proven harder then expected. Given the breaking change introduced by Datadog, we have decided to merge now, and then follow up when we have time to reduce the SDK size.