[FEAT] Build functions individually

[tstackhouse]

Describe the solution you'd like
Serverless supports the package.individually flag, however during the build step, all functions are passed through webpack together, resulting in a single dist build that the packaging step then pulls from. Without adding separate include/exclude rules to each function, this results in deployment artifacts that are identical (and can cause memory exhaustion in the packaging step as each artifact is built in complete parallel). Adding a global exclude and including individual function handlers to each artifact helps with this, however the generated node_modules includes deps for all the functions, not just the one that is pulled in.

Describe alternatives you've considered

• Not using package.individually results in each lambda having the same artifact deployed
• Using the flag will create artifacts for each lambda, but they are wtill identical
• Adding individual include declarations will pull only the compiled handler, but node_modules is still identical
• Refactoring to make smaller apps is also a possibility, but that would still face the same kinds of shortcomings of packaging individually.

Check which provider is affected:
[X] AWS
[?] Azure
[?] Google Cloud Platform

Check which framework is affected:
[] Angular
[x] Nodejs
[x] Serverless
[] Lambda
[] Infrastructure as a code

Additional context
I'm working on a project that we're using nx-serverless in production, and currently have ~60 lambda handlers in our API.

[wickstargazer]

Hi, i did make a compile executor which just compiles the typescript into javascript that works. But i think i might need more education or dig in on how serverless individual flag works. I am imagining it only pull certain entry files into the compiler

With that set to each deploy configuration we might be able to achieve the smaller size effect. Do you have any insights?

the function getEntryForFunction in file libs\nx-serverless\src\utils\normalize.ts does this thing. If we can refactor it to accomodate the scenario i think we can achieve the same effect. Because i took the logic from serverless-webpack plugin and i think i ommitted the individual check.

export const getEntryForFunction = (
  name,
  serverlessFunction,
  serverless,
  sourceroot,
  root
) => {
  const handler = serverlessFunction.handler;

  const handlerFile = getHandlerFile(handler);
  if (!handlerFile) {
    _.get(serverless, 'service.provider.name') !== 'google' &&
      serverless.cli.log(
        `\nWARNING: Entry for ${name}@${handler} could not be retrieved.\nPlease check your service config if you want to use lib.entries.`
      );
    return {};
  }
  const ext = getEntryExtension(handlerFile, serverless);

  // Create a valid entry key
  let handlerFileFinal = `${sourceroot.replace(
    '/src',
    ''
  )}/${handlerFile}${ext}`;

  if (handlerFile.match(/src/)) {
    handlerFileFinal = `${sourceroot}/${handlerFile.replace('src/', '')}${ext}`;
  }

  return {
    [handlerFile]: resolve(root, `${handlerFileFinal}`),
  };
};

[tstackhouse]

I'd need to experiment with that a bit, it's an interesting thought. We had used the compile executor at one point I believe, but missing out on all the bundling was causing other issues.

I've been elbow deep trying to triage some really esoteric build failures in our project for the last few days and will probably file a bug report after getting it working with my fork. I want to submit a PR, but with all the changes from the hotfix/0.5.3 branch other things are breaking in completely different ways, probably related to config schema changes.

[wickstargazer]

@tstackhouse Lets move this onto 1.0.0-beta i think its an easier code approach and easy to read as well.

Sorry i am shuffling it around ... had to refactor to support the new workspace.

If you do not want to update your workspace, maybe we can do a 0.5.4 fix for the solution? I can help you through it.
lets exchange discord/slack

[tstackhouse]

No worries. I may have gotten a bandaid on my issue for the moment, I'd need to upgrade to serverless 2 in order to use the beta, as I get an error when attempting to run the deploy command:

10035$ NODE_OPTIONS="--max-old-space-size=7168" npx nx run my-api-api:deploy --stage=my-stage

> nx run my-api-api:deploy --stage=my-stage
Starting to Initiate Serverless Instance
Loading Environment Variables
Environment variables set according to env.json
Initiating Serverless Instance
Disable "Resolve Configuration Internally" for serverless 2.0+.
Serverless: Deprecation warning: Serverless constructor expects service configuration details to be provided.
            Starting from next major Serverless will no longer auto resolve it internally.
            More Info: https://www.serverless.com/framework/docs/deprecations/#MISSING_SERVICE_CONFIGURATION
Serverless: Deprecation warning: Serverless constructor expects resolved CLI commands and options to be provided via "config.commands" and "config.options".
            Starting from next major Serverless will no longer auto resolve CLI arguments internally.
            More Info: https://www.serverless.com/framework/docs/deprecations/#MISSING_COMMANDS_OR_OPTIONS_AT_CONSTRUCTION
Serverless: Running "serverless" installed locally (in service node_modules)
Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
Serverless: Load command dev
The Serverless version (2.44.0) does not satisfy the "frameworkVersion" (>=1.58.0 <2.0.0) in serverless.yml

It's still on my radar and want to keep the discussion going, but I may have at least gotten the immediate fire out by increasing my memory limit to the GH actions max.

[warrickhill]

Any update on when this issue might be resolved?

[wickstargazer]

Hi, i will eye this on our next release. thanks for the bump @warrickhill

[mvdornellas]

Good afternoon guys, @tstackhouse how are you?
Do you have any updates on when this feature will be released?

[wickstargazer]

@dornellas13 this is fixed in the current version.
check out this PR #97

let me know if you get stuck anywhere, there is a bug for stage which i will be releasing in a few days.

[wickstargazer]

@dornellas13 we have released 1.1.0 and it can deploy using stage options and function option according to #97