feat: remove dev node_modules deps #242

khuezy · 2023-09-20T21:49:24Z

fixes: #166

This reduces my zipped lambda from 31MB => 10MB

I investigated the use of next.config's outputFileTracingExcludes but that is hard to merge w/ the user's config file since the config is js and may contain functions - which we can't easily stringify. Doing a complex regex to replace is very tricky and brittle if the user's config file is non standard too...

So the solution is to do the node_modules cleanup AFTER next has built.

vercel · 2023-09-20T21:49:27Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
open-next	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 20, 2023 9:50pm

changeset-bot · 2023-09-20T21:49:28Z

⚠️ No Changeset found

Latest commit: e8e8ea7

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

sladg · 2023-09-21T16:24:19Z

packages/open-next/src/build.ts

+  if (options.minify) {
+    removeNodeModule(path.join(outputPath, "node_modules"), [
+      "@esbuild",
+      "prisma/libquery_engine-darwin-arm64.dylib.node",


can we include all prisma engines?

meaning all other OS packages not compatible with Lambda

Do you know where I can get a list of all non-lambda engines?
I might need to update this and use micromatch to allow for negation...

I have some generic prisma node_modules cleanup code https://github.com/jetbridge/sst-prisma/blob/master/stacks/resources/prismaLayer.ts#L90

Honestly, what the hell is wrong with prisma?

RDIL · 2023-11-01T02:42:41Z

packages/open-next/src/build.ts

+      // "react", // TODO: remove react/react-dom when nextjs updates its precompile versions
+      // "react-dom",
+      "@webassemblyjs",
+      "uglify-js",


Duplicate here, it's also listed above!

Thx, I'll have to retest this w/ Next 14, which seems to have fixed the dev deps getting traced to the standalone output.

Been busy last week and caught a cold this week... I'll eventually test this to confirm... @mathisobadia was still seeing these dev modules but I didn't...

Here's my server function lambda zip, on Next 14.0.2:

@revmischa This is inside your server-function lambda ? There is a lot of things that you should be able to remove, you can use experimental.outputFileTracingExcludes from next config to remove all of those.
You should probably create an issue in next repo as well, those dev deps should have been removed.

mathisobadia · 2023-11-06T14:32:55Z

@khuezy The docs mention that next 14+ doesn't have the issue anymore but I that doesn't seem to be the case for me. So this PR is still valuable

seawatts · 2023-12-11T03:50:20Z

Any progress on this?

khuezy · 2023-12-11T20:46:22Z

Any progress on this?

Not much, for now, you can try using outputFileTracingExcludes:

 outputFileTracingExcludes: {
   '*': [
     '**@swc/core**',
     '**esbuild**',
    '**uglify**',
      'webassemblyjs',
     '**sass**',
     '**caniuse-lite**',
     "@aws-sdk"
   ],
},

It's hard to know which to remove so it might be better for the user to manage that. For example, if you add "watchpack", it'll break server actions even though it looks like a dev dep.

revmischa · 2023-12-30T22:58:37Z

Any progress on this?

Not much, for now, you can try using outputFileTracingExcludes:

 outputFileTracingExcludes: {
   '*': [
     '**@swc/core**',
     '**esbuild**',
    '**uglify**',
      'webassemblyjs',
     '**sass**',
     '**caniuse-lite**',
     "@aws-sdk"
   ],
},

I'm trying to use this but I get a "RangeError: Maximum call stack size exceeded" error: vercel/next.js#42641 (comment)

khuezy · 2023-12-30T23:24:49Z

Something change in the later version of next (not sure if bug or by design.) Using the "**" causes an infinite loop. You have remove the wildcard.

revmischa · 2023-12-31T01:52:04Z

Something change in the later version of next (not sure if bug or by design.) Using the "**" causes an infinite loop. You have remove the wildcard.

Hm, I tried

    outputFileTracingExcludes: {
      "*": [
        "@swc/core",
        "webpack",
        "docx",
       ]

And get the same error

enroly-mike · 2024-01-31T16:45:17Z

Maybe im doing something weird here, but on nextjs 14.1 and pnpm ... I've tried to manually remove the node-modules from the server function you have listed above and my lambda fails pretty hard.

{ "errorType": "Error", "errorMessage": "Cannot find module 'styled-jsx/package.json'\nRequire stack:\n- /var/task/typescript/frontend/insights/node_modules/next/dist/server/require-hook.js\n- /var/task/typescript/frontend/insights/node_modules/next/dist/server/next-server.js", "code": "MODULE_NOT_FOUND", "requireStack": [ "/var/task/typescript/frontend/insights/node_modules/next/dist/server/require-hook.js", "/var/task/typescript/frontend/insights/node_modules/next/dist/server/next-server.js" ], "stack": [ "Error: Cannot find module 'styled-jsx/package.json'", "Require stack:", "- /var/task/typescript/frontend/insights/node_modules/next/dist/server/require-hook.js", "- /var/task/typescript/frontend/insights/node_modules/next/dist/server/next-server.js", " at Module._resolveFilename (node:internal/modules/cjs/loader:1134:15)", " at resolve (node:internal/modules/helpers:188:19)", " at Object.<anonymous> (/var/task/typescript/frontend/insights/node_modules/next/dist/server/require-hook.js:38:32)", " at Module._compile (node:internal/modules/cjs/loader:1356:14)", " at Module._extensions..js (node:internal/modules/cjs/loader:1414:10)", " at Module.load (node:internal/modules/cjs/loader:1197:32)", " at Module._load (node:internal/modules/cjs/loader:1013:12)", " at Module.require (node:internal/modules/cjs/loader:1225:19)", " at require (node:internal/modules/helpers:177:18)", " at Object.<anonymous> (/var/task/typescript/frontend/insights/node_modules/next/dist/server/next-server.js:13:1)" ] }

khuezy · 2024-02-04T01:12:32Z

I've tried to manually remove the node-modules

Did you remove the entire folder? You should only remove the dev subfolders, not the entire node_modules

r34son · 2024-02-20T21:37:26Z

packages/open-next/src/build.ts

+
+  if (options.minify) {
+    removeNodeModule(path.join(outputPath, "node_modules"), [
+      "@esbuild",


my logs:

can we remove some of them also?

Very likely yes, but you'll need to test for yourself. You can probably remove :

swc core

sharp

esbuild

caniuse lite

uglify-js

terser

AndreyMay · 2024-07-02T04:13:49Z

nextjs v14.2.4 / open-next 3.0.2 -- still includes dev dependencies in standalone build.

This next config works and excludes unwanted dependencies (note: the setup for monorepo):

    outputFileTracingRoot: path.join(__dirname, '../../'),
    outputFileTracingExcludes: {
      '*': [
        './**/.prisma/client/libquery_engine-darwin*', // prisma mac binary
        './**/@swc/core-linux-x64-gnu*',
        './**/@swc/core-linux-x64-musl*',
        './**/@esbuild*',
        './**/webpack/',
        './**/rollup*',
        './**/terser*',
        './**/sharp*',
      ],
    },

feat: remove dev node_modules deps

e8e8ea7

vercel bot deployed to Preview September 20, 2023 21:50 View deployment

sladg reviewed Sep 21, 2023

View reviewed changes

RDIL reviewed Nov 1, 2023

View reviewed changes

revmischa mentioned this pull request Dec 30, 2023

Large bundle size as of NextJS 13.0.2 due to Prisma + 2x @swc/core (musl + gnu) vercel/next.js#42641

Closed

1 task

r34son reviewed Feb 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: remove dev node_modules deps #242

feat: remove dev node_modules deps #242

khuezy commented Sep 20, 2023 •

edited

Loading

vercel bot commented Sep 20, 2023 •

edited

Loading

changeset-bot bot commented Sep 20, 2023

sladg Sep 21, 2023 •

edited

Loading

khuezy Sep 21, 2023 •

edited

Loading

revmischa Nov 15, 2023

khuezy Nov 15, 2023

RDIL Nov 1, 2023

khuezy Nov 1, 2023

khuezy Nov 15, 2023

revmischa Nov 15, 2023

conico974 Nov 17, 2023

mathisobadia commented Nov 6, 2023

seawatts commented Dec 11, 2023

khuezy commented Dec 11, 2023

revmischa commented Dec 30, 2023

khuezy commented Dec 30, 2023

revmischa commented Dec 31, 2023

enroly-mike commented Jan 31, 2024

khuezy commented Feb 4, 2024

r34son Feb 20, 2024

conico974 Feb 20, 2024

AndreyMay commented Jul 2, 2024 •

edited

Loading

feat: remove dev node_modules deps #242

Are you sure you want to change the base?

feat: remove dev node_modules deps #242

Conversation

khuezy commented Sep 20, 2023 • edited Loading

vercel bot commented Sep 20, 2023 • edited Loading

changeset-bot bot commented Sep 20, 2023

⚠️ No Changeset found

sladg Sep 21, 2023 • edited Loading

Choose a reason for hiding this comment

khuezy Sep 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mathisobadia commented Nov 6, 2023

seawatts commented Dec 11, 2023

khuezy commented Dec 11, 2023

revmischa commented Dec 30, 2023

khuezy commented Dec 30, 2023

revmischa commented Dec 31, 2023

enroly-mike commented Jan 31, 2024

khuezy commented Feb 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndreyMay commented Jul 2, 2024 • edited Loading

khuezy commented Sep 20, 2023 •

edited

Loading

vercel bot commented Sep 20, 2023 •

edited

Loading

sladg Sep 21, 2023 •

edited

Loading

khuezy Sep 21, 2023 •

edited

Loading

AndreyMay commented Jul 2, 2024 •

edited

Loading