serverless-plugin-split-stacks: AppSync Api not found in stack. Did you forget to deploy?

[owain68]

I am getting the error AppSync Api not found in stack. Did you forget to deploy?.

Here is the environment and output.

✖ Stack tech-xxxxxx-dev failed to deploy (1353s)
Environment: darwin, node 18.13.0, framework 3.27.0 (local), plugin 6.2.3, SDK 4.3.2
Credentials: Local, "wgv-app-dev" profile
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues

Error:
AppSync Api not found in stack. Did you forget to deploy?

I am trying to migrate from V1 of the plugin using the migration guide.

All the resources appear to deploy for a short period of time before a ROLLBACK is issued. And then they disappear. I suspect that the Rollback is initiated by the serverless framework itself since the appsync plugin throws this error.

async getApiId() {
   this.loadConfig();

   if (!this.naming) {
     throw new this.serverless.classes.Error(
       'Could not find the naming service. This should not happen.',
     );
   }

   const logicalIdGraphQLApi = this.naming.getApiLogicalId();

   const { StackResources } = await this.provider.request<
     DescribeStackResourcesInput,
     DescribeStackResourcesOutput
   >('CloudFormation', 'describeStackResources', {
     StackName: this.provider.naming.getStackName(),
     LogicalResourceId: logicalIdGraphQLApi,
   });

   const apiId = last(StackResources?.[0]?.PhysicalResourceId?.split('/'));

   if (!apiId) {
     throw new this.serverless.classes.Error(
       'AppSync Api not found in stack. Did you forget to deploy?',
     );
   }

   return apiId;
 }

So apiId is not found for some reason which throws an exception.

I am probably missing something in my serverless.yml or do I need to explicitly Output some CloudFormation values to fix this.

Here is the header of my serverless yml file.

appSync:
  name: "${self:service}-${self:provider.stage}"
  authentication:
    type: AMAZON_COGNITO_USER_POOLS
    config:
      awsRegion: ${self:provider.region}
      defaultAction: ALLOW
      userPoolId:
        Fn::ImportValue: ${self:custom.resources.PROJECT_PREFIX}-UserPoolId
  additionalAuthentications:
    - type: API_KEY
  apiKeys:
    - name: Default-${self:provider.stage}
      description: Default-${self:provider.stage} generated by Serverless deployment
      expiresAfter: 1y
      apiKeyId: ${self:custom.resources.APPSYNC_API_KEY}
....

It would be very helpful if there is a working example available to compare with.

Thanks in advance.

O.,

[owain68]

I have managed to run this through the debugger and the following screenshot shows that describeStackResources returns an empty array.

I am using serverless-plugin-split-stacks so might this be the call pointing to the wrong Stack?

aws cloudformation describe-stack-resources --stack-name my-stack-root-name returns an array of 100 the max for this call so I am not sure why the call is not returning anything. And yes I am in the correct AWS account!

[bboure]

Hi,

Thanks for reporting this and debugging

The split stack plugin is likely the problem here.
your API resource is probably in a substack, which causes this issue.

I see two options to fix this:

1. Maybe we can first find in which stack the main API resource is located, then described that stack to extract the data (not sure how to do that yet)
2. If we can't easily detect which stack it is in, we're going to have to describe all stacks until we find the right one.

[owain68]

@bboure Thank you for your response. Indeed I agree it is an incompatibility with serverless-plugin-split-stacks

❯ aws cloudformation describe-stack-resources --stack-name my-root-stack --logical-resource-id GraphQlApi
{
    "StackResources": []
}

Which is what the plugin is executing, whereas this:

❯ aws cloudformation describe-stack-resources --stack-name my-root-stack-GraphQlNestedStack-8IL3PSMRG63O --logical-resource-id GraphQlApi
{
    "StackResources": [
        {
            "StackName": "tech-wegive-dev-GraphQlNestedStack-8IL3PSMRG63O",
            "StackId": "arn:aws:cloudformation:eu-west-1:XXXXXXXXX:stack/my-root-stack-GraphQlNestedStack-8IL3PSMRG63O/e1770f10-db7b-11ed-ab88-02c5b9160411",
            "LogicalResourceId": "GraphQlApi",
            "PhysicalResourceId": "arn:aws:appsync:eu-west-1:XXXXXXXX:apis/azeyoqvmxfdh3nlydysuqzsroq",
            "ResourceType": "AWS::AppSync::GraphQLApi",
            "Timestamp": "2023-04-15T10:54:32.852000+00:00",
            "ResourceStatus": "CREATE_COMPLETE",
            "DriftInformation": {
                "StackResourceDriftStatus": "NOT_CHECKED"
            }
        }
    ]
}

For the serverless-plugin-split-stacks this approach might work.

{
    "StackResources": [
        {
            "StackName": "tech-wegive-dev",
            "StackId": "arn:aws:cloudformation:eu-west-1:XXXXXXXXX:stack/my-root-stack/8fcf1640-db7a-11ed-88b5-0224c27dc929",
            "LogicalResourceId": "GraphQlNestedStack",
            "PhysicalResourceId": "arn:aws:cloudformation:eu-west-1:XXXXXXXX:stack/my-root-stack-GraphQlNestedStack-8IL3PSMRG63O/e1770f10-db7b-11ed-ab88-02c5b9160411",
            "ResourceType": "AWS::CloudFormation::Stack",
            "Timestamp": "2023-04-16T11:12:51.674000+00:00",
            "ResourceStatus": "UPDATE_COMPLETE",
            "DriftInformation": {
                "StackResourceDriftStatus": "NOT_CHECKED"
            }
        }
    ]
}

We can see he plugin creates a nested stack under LogicalResourceId of GraphQlNestedStack which gives you the nested stack name in PhysicalResourceId (with a bit split('/') calculation). Then this could be fed into the the "second" describeStackResources with the nestedStackId.

One risk is that you are building a dependency on the serverless-plugin-split-stacks if they change. Another point occurred to me is the limitation of describeStackResources in only returning 100 items. I am not clear if by specifying LogicalResourceId this limit is never being approached or is it a limit on the search? Perhaps aws cloudformation list-stack-resources as described in the doc

Obviously this problem has broken my deployment pipeline - I did the upgrade to V2 in order to start implementing Javascript resolvers - for obvious reasons.

Thank you for your help.

O.

[owain68]

@bboure I have tweaked some of the settings on serverless.yml for some of the serverless-plugin-split-stacks to

  splitStacks:
    perFunction: false
    perType: false
    perGroupFunction: true
    nestedStackCount: 20
    stackConcurrency: 5 # Controls if enabled and how much stacks are deployed in parallel. Disabled if absent.
    resourceConcurrency: 10 # Controls how much resources are deployed in parallel. Disabled if absent.

and there still remains a LogicalResourceId of GraphQlNestedStack so it looks like this remains constant. I have had a look at the source of serverless-plugin-split-stacks but I can't work out how this gets set. @dougmoscrop.

[dougmoscrop]

I don't have the bandwidth to provide technical support, but keep in mind:

• an already migrated resource in an existing nested stack doesn't get (re/de)-migrated; when this plugin was written CloudFormation imports didn't exist and so there was no way to go about that
• you can override migration logic for new resources or new stages using stacks-map.js
• CloudFormation has vastly increased limits, consider not using split-stacks anymore, and if you are adding more than 500 resources to a stack you are asking for a Bad Time. You can alternatively use my bootstrap plugin to just move some resources to a single prereq stack, usually stateful things like database tables, queues, and buckets.

[bboure]

Thanks both for all the details.

I wonder where the GraphQlNestedStack name comes from, but there is no guarantee that it will always be the same, depending on configuration.

I will try to investigate this and find a way to solve this that does not imply taking assumptions based on what plugin-split-stacks does.

The dirty workaround would be to walk through the main stack first and then all the nested stacks ("ResourceType": "AWS::CloudFormation::Stack",) until we find what we need.

In your case, I think the problem occurs when the plugin tries to describe the stack to show you some details (API id, URL, API keys, etc). In that particular case, we could also swallow the error and skip that part instead of failing and forcing SF to roll everything back.

[dougmoscrop]

If you add Outputs to the template in this plugin, and it's run first, the split stack plugin should fix the output to point to the nested stack output when it migrates

[bboure]

Ah that's a good info. Thanks @owain68

If I remember correctly that's how v1 used to do this. That would explain why this issue is new.

[owain68]

@dougmoscrop

Thank you very much for your advice. I have removed serverless-plugin-split-stacks as you have suggested. I hadn;t realised the CF limits were now 500 and I now come in well below that. I already have my persistence layers (Cognito and DynamoDb) in other stacks so I was able to drop "the functions" and recreate without any major issues.

@bboure I am not sure whether we should close this issue, I don't mind but maybe it might be worth including in the Upgrade Guide. I am happy to raise a PR in the doc for it - it might save someone about seven days of debugging! As my deployment is large, it takes forever to debug since the feedback loop is so long!

O.

[ghost]

@dougmoscrop
I currently have 614 resources deployed.
The content deployments of the said Issues are failing.
(There is no error indication on the stack, but the CLI execution result shows an error)

If we remove serverless-plugin-split-stacks, the number of resources will be exceeded and deployments will not communicate, so we would like to continue using serverless-plugin-split-stacks.

Is there any effective solution?
It would be preferable for the plugin to be compatible, but if this is difficult, we would like to try to solve the problem somehow by using our own debugging.