fix(webpack): Aliased module paths now properly map to the correct aurelia-loader module id [SIMPLE] #139

pat841 · 2018-02-26T17:14:50Z

Reference: #122
@jods4 @EisenbergEffect
Information:

/**
 *  The purpose of this plugin is to track down where exactly each included dependency lives and build a module
 *  name from that. Since projects and webpack configurations can be vary, we do our best to guess but expect edge-cases
 *  to be hit and changes needed.
 *
 *  Process:
 *  - Each AureliaDependency contains the preserveModuleName symbol to notify this plugin to track the dependency
 *  - AureliaDependenciesPlugin searches and includes all PLATFORM.moduleName() dependencies  and stores as
 *    an AureliaDependency
 *  - ConventionDependenciesPlugin searches for all relative includes and stores as an AureliaDependency
 *  - At this point, webpack has resolved all included modules regardless of using a relative or absolute path
 *  - Now we want to normalize each include so that we can reliably replace the include string to match the webpacks
 *    module id
 *  - We then unwrap all the modules from ModuleConcatenationPlugin to get the raw dependencies
 *  - For each raw dependency that is included via node_modules/, read from its location:
 *      - Example Path: /home/usr/pkg/node_modules/mod/dir/file.js
 *          - The path to the module itself: /home/usr/pkg/node_modules
 *          - The module name: mod
 *          - The relative path: dir/file.js
 *      - Map the parsed path data to _nodeModuleResourcesMap by the parsed module name and its resource location
 *          - Example Map:
 *              {
 *                'aurelia-templating-router': {
 *                  '/home/usr/pkg/node_modules/aurelia-templating-router/dist/native-modules/router-view.js': {
 *                    path: '/home/usr/pkg/node_modules/aurelia-templating-router',
 *                    name: 'aurelia-templating-router',
 *                    relative: '/dist/native-modules/router-view.js',
 *                  },
 *                  '/home/usr/pkg/node_modules/aurelia-templating-router/dist/native-modules/route-href.js': {
 *                    path: '/home/usr/pkg/node_modules/aurelia-templating-router',
 *                    name: 'aurelia-templating-router',
 *                    relative: '/dist/native-modules/route-href.js',
 *                  },
 *                  '/home/usr/pkg/node_modules/aurelia-templating-router/dist/native-modules/aurelia-templating-router.js': {
 *                    path: '/home/usr/pkg/node_modules/aurelia-templating-router',
 *                    name: 'aurelia-templating-router',
 *                    relative: '/dist/native-modules/aurelia-templating-router.js',
 *                  },
 *                  '/home/usr/pkg/node_modules/aurelia-templating-router/dist/native-modules/route-loader.js': {
 *                    path: '/home/usr/pkg/node_modules/aurelia-templating-router',
 *                    name: 'aurelia-templating-router',
 *                    relative: '/dist/native-modules/route-loader.js',
 *                  },
 *                }
 *              }
 *  - For each mapped node_module, look at the included resources and normalize:
 *      - Look at the modules included resources and find a common path and its entry point
 *          - Entry point conditions:
 *              - Resource name matches 'index'
 *              OR Resource name matches the module name
 *              OR It is the only included module resource
 *          - If there are multiple entry points:
 *              - Pick the most shallow resource
 *              - If they are both as shallow as possible choose index over module name match
 *      - Map the normalized module id to _nodeModuleResourceIdMap by the resource file
 *          - Example Map:
 *              {
 *                '/home/usr/pkg/node_modules/aurelia-templating-router/dist/native-modules/router-view.js': 'aurelia-templating-router/router-view.js',
 *                '/home/usr/pkg/node_modules/aurelia-templating-router/dist/native-modules/route-href.js': 'aurelia-templating-router/route-href.js',
 *                '/home/usr/pkg/node_modules/aurelia-templating-router/dist/native-modules/aurelia-templating-router.js': 'aurelia-templating-router',
 *                '/home/usr/pkg/node_modules/aurelia-templating-router/dist/native-modules/route-loader.js': 'aurelia-templating-router/route-loader.js',
 *              }
 *  - Look at all webpack modules and for each module that includes or has a dependency that includes an AureliaDependency:
 *      - Handling module ids can be a bit tricky. Modules can be included in any of the following ways:
 *          import { Module } from 'module'
 *                                 'module/submodule'
 *                                 './module'
 *                                 'folder/module'
 *                                 'alias/folder/module'
 *                                 'alias$'
 *                                 '@scope/module'
 *          @decorator(PLATFORM.moduleName('module'))
 *                                 ...
 *      - The problem arises when aurelia-loader has to know the module to use at runtime. Webpack Mappings:
 *          - Absolute Module: 'module' -> 'module'
 *          - Relative Module: './module' -> 'folder/module'
 *          - Absolute Path: 'folder/module' -> 'folder/module'
 *          - Aliased Path: 'alias/folder/module' -> 'alias/folder/module'
 *          - Absolute Alias Path: 'alias$' -> 'alias$'
 *      - In order to have the aurelia-loader work correctly, we need to coerce everything to normalized ids
 *          - If the module is in the node_module/ map, use the normalized module id
 *          - If the module exists inside a webpack resolver path, use the relative path as the module id
 *          - If the module exists inside a webpack alias path, use the relative path from the alias path as the module id
 *      - Set the modules preserveModuleName Symbol export so the AureliaDependenciesPlugin can read it later
 *  - In AureliaDependenciesPlugin, for each AureliaDependency, replace the original and now incorrect PLATFORM.moduleName()
 *    path with the normalized path.
 *  - Source files now contain the normalized module paths with map correctly to the webpack module ids so that the
 *    aurelia-loader can find them during runtime.
 */

CLAassistant · 2018-02-26T17:14:57Z

All committers have signed the CLA.

jods4 · 2018-02-26T18:24:04Z

Thanks for creating this PR!

I feel bad about what I'm going to say but the timing is unfortunate...
I just created branch v2.0-rc6, which contains lots of modifications to make the plugin compatible with Webpack 4. As you might know, Webpack 4 was just released last week and it contains breaking API changes that I had to fix.

Could you please rebase your changes on top of v2.0-rc6 and check that they work with Webpack 4? Also please don't use deprecated APIs that generate warnings. Hopefully you won't have too much to change and you can use my own fixes as reference.

pat841 · 2018-02-28T14:04:09Z

@jods4 I went ahead and rebased but I have yet to test it as I have not yet upgraded my own build to webpack 4. If you have a working webpack 4 build, would you be able to test on your build for a reference against my webpack 3 build while I work on getting my build on webpack 4.

rek72-zz · 2018-06-12T20:47:48Z

@pat841 Hi Pat. This still shows open. Where are we at with this? Thanks.

Alexander-Taran · 2018-06-13T22:27:05Z

@pat841 have you had time to test your rebase?

pat841 · 2018-06-14T17:40:35Z

@rek72 and @Alexander-Taran I have verified that this now works on both Windows, MacOS, and Linux using Webpack 4.

rek72-zz · 2018-06-14T18:36:58Z

Are you testing with the fixed version or what's been committed?

jods4 · 2018-06-16T14:26:31Z

So I wanted to review this and possibly merge but it seems that it wasn't rebased properly.
Somehow it contains one of my own commits e49fb8d, which (1) is already on master and (2) creates a lot of noise in the diff and makes it harder for me to review.

Can this be rebased on master please?

pat841 · 2018-06-26T05:19:06Z

@jods4 I went ahead and re-based on master. Should be good to go.

jods4

This is a huge PR to the most complex part of this plugin so it'll take some time for me to review entirely.

First pass is mostly coding style...

I'm trying to get a good grasp of the modifications. Can you provide a quick summary of what behavior you intended to change with this PR? What specific use-case works differently than before?

jods4 · 2018-07-02T09:15:55Z

src/PreserveModuleNamePlugin.ts

-      return !!value;
+
+      // Preserve the module if its dependencies are also preserved
+      const reasons = (m.reasons && Array.isArray(m.reasons)) ? m.reasons : [];


Is there a reason for this defensive code?
It was not in my code and I've never had an issue filed about m.reasons not being an array.
Is there a case where it happens?

I am unsure, I usually add safety checks and assume to worst. If you need me to I can remove it.

The problem I have with defensive code is that it creates headache for future maintainers.

When I see this code it tells me "m.reasons could be something else than an array, we better be careful" and I immediately ask myself: "how so? what am I missing?".

If this m.reasons can be null we should document how / why, it's useful information.
If it cannot, then it's misleading bloat that I'd rather remove.

When unsure, someone needs to figure this out and it's courteous that you do it rather than deferring the analysis to future maintainers.

jods4 · 2018-07-02T09:17:25Z

src/PreserveModuleNamePlugin.ts

+
+      // Preserve the module if its dependencies are also preserved
+      const reasons = (m.reasons && Array.isArray(m.reasons)) ? m.reasons : [];
+      return reasons.some((reason) => Boolean(reason.dependency && reason.dependency[preserveModuleName]));


Nit: no need for parenthesis when there's a single argument. No need to convert to bool explicitely as it's done by some.

jods4 · 2018-07-02T09:19:38Z

src/PreserveModuleNamePlugin.ts

@@ -57,11 +147,13 @@ export class PreserveModuleNamePlugin {
          if (/^async[?!]/.test(realModule.rawRequest)) 
            id = "async!" + id;

-          id = id.replace(/\\/g, "/");
+          id = id.replace(/\\/g, '/');


All strings use double-quotes, why did you change this one to single-quote?
Also: consistency is good, make sure your new code uses double-quotes (I spotted some single-quotes).

jods4 · 2018-07-02T09:24:40Z

src/PreserveModuleNamePlugin.ts

+ *  @return {string|null} The module id if available, null otherwise
+ */
+function getModuleId(module: Webpack.Module, paths: string[], aliases: { [key: string]: string } | null): string | null {
+  if (!module) {


Why? This is an internal API and you've typed module: Webpack.Module.
We compile with strict null checks enabled, so this is not possible.

jods4 · 2018-07-02T09:26:20Z

src/PreserveModuleNamePlugin.ts

-                 aliasRelative(alias, realModule.resource);
-
+          // Get the module id
+          let id = getModuleId(realModule, roots, alias);


If we factor out getModuleId, we might as well take the if (!id) throw below with it.
That changes the return type to a non-null value.

What do you mean by this?

I mean on next line (140) you throw if the id is null.
You can make getModuleId throw if it can't find an id; and have a non-nullable return.

Non-nulls are better coding style and it makes getModuleId easier to re-use by not having to check its result. Mainline code is simpler as well.

jods4 · 2018-07-02T09:36:49Z

src/AureliaDependenciesPlugin.ts

+    // Get the module id, fallback to using the module request
+    let moduleId: string = dep.request;
+    if (dep.module && typeof dep.module[preserveModuleName] === 'string') {
+      moduleId = dep.module[preserveModuleName];


I am not sure what the impact of this change is.
Can you explain briefly, maybe with an example that behaves differently before/after this change?

AureliaDependenciesPlugin uses the preserveModuleName symbol to set the generated moduleId. We then check that it is set here and use that for the webpack moduleId, otherwise fallback to the default usage.

OK, I see.
I had this idea of rewriting user code with Webpack generated id but it breaks quite a few things (such as conventions).

You take a middle ground where you rewrite user's code with our normalized id.
That seems safe and would solve a few issue, I think, as aurelia-loader will do the same normalization anyway.., and in the rare cases where it doesn't (there are a few) it would break at runtime anyway.

Looks like a good change to me 👍

Out of curiosity: does this fix a specific problem you have?

Yes it does see #121

jods4 · 2018-07-02T09:37:21Z

src/AureliaDependenciesPlugin.ts

@@ -41,7 +48,7 @@ class ParserPlugin {
    hooks.evaluateIdentifier.tap("imported var.moduleName", TAP_NAME, (expr: Webpack.MemberExpression) => {
      if (expr.property.name === "moduleName" &&
          expr.object.name === "PLATFORM" &&
-          expr.object.type === "Identifier") {
+          String(expr.object.type) === "Identifier") {


Why? What can expr.object.type be that is not a string but would convert to "Identifier"?

OK, I see now that newer TS shows an error here.
A cast is a hack, the real problem lies in my poor type definition of expr.object.
Would be nice to fix it there instead.

Same for the later occurances of String(expr.object.type)

jods4 · 2018-07-02T09:38:01Z

src/AureliaDependenciesPlugin.ts

@@ -55,8 +62,8 @@ class ParserPlugin {
    //    PLATFORM.moduleName("id");
    hooks.evaluate.tap("MemberExpression", TAP_NAME, expr => {
      if (expr.property.name === "moduleName" &&
-         (expr.object.type === "MemberExpression" && expr.object.property.name === "PLATFORM" ||
-          expr.object.type === "Identifier" && expr.object.name === "PLATFORM")) {
+         (String(expr.object.type) === "MemberExpression" && expr.object.property.name === "PLATFORM" ||


Same question here and on next line: what can expr.object.type be that is not a string and would convert to "MemberExpression"?

jods4 · 2018-07-02T09:40:30Z

src/PreserveModuleNamePlugin.ts

+ *  @return {string|null} The escaped string
+ */
+function escapeString(str: string): string | null {
+  if (typeof str !== 'string') {


internal API with type string...

jods4 · 2018-07-02T09:48:31Z

src/PreserveModuleNamePlugin.ts

+ *
+ *  @return {string|null} The module id if available, null otherwise
+ */
+function getModuleId(module: Webpack.Module, paths: string[], aliases: { [key: string]: string } | null): string | null {


When extracting this function, we lost one bit of functionality.
If module[preserveModuleNode] was defined it would be used without looking at anything else.
This can be set by some plugins when creating the dependency, if I recall correctly.

Is this intentionnal?

I was unaware that other modules could set it. The AureliaDependenciesPlugin uses it to explicitly set the generated moduleId.

See for example (with some comments on why) ConventionDependenciesPlugin

pat841 · 2018-08-27T20:09:17Z

@jods4

I went ahead and updated the PR, apologies on the delay. I commented with any questions I had and updated some things to conform to your style guide.

pat841 · 2018-09-26T14:49:53Z

@jods4 Have you had a chance to review this yet?

pat841 · 2018-10-10T13:09:39Z

Any update on this @jods4 @EisenbergEffect

EisenbergEffect · 2018-10-11T00:07:49Z

@jods4 Can you look into this?

jods4 · 2018-10-11T21:26:39Z

src/webpack.d.ts

@@ -58,7 +58,7 @@ declare namespace Webpack {
    // Those types are not correct, but that's enough to compile this project
    property: IdentifierExpression;
    object: { name: string; type: string; } & MemberExpression;
-    type: "MemberExpression";
+    type: "MemberExpression" | "Identifier";


This seems very wrong MemberExpression is defined by type: "MemberExpression", it can't be Identifier.
The problem is rather on previous line where object: { ... } & MemberExpression;

I noticed that newer TS version create error because of this, we need to fix it.
I'm not sure why I wrote & MemberExpression in the first place here, because it's obviously not 100% correct.

jods4 · 2018-10-11T21:47:26Z

src/PreserveModuleNamePlugin.ts

 export const preserveModuleName = Symbol();

+// node_module maps
+const _nodeModuleResourcesMap: NodeModule.ResourcesMap = {};
+const _nodeModuleResourceIdMap: NodeModule.ResourceIdMap = {};


nit (style): other private members are not _ prefixed in this project.

jods4 · 2018-10-11T21:48:37Z

src/PreserveModuleNamePlugin.ts

@@ -30,6 +123,10 @@ export class PreserveModuleNamePlugin {
            modulesBeforeConcat.splice(i--, 1, ...m["modules"]);
        }

+        // Map and parse the modules if needed
+        modulesBeforeConcat.forEach((m) => mapNodeModule(m));


nit: modulesBeforeConcat.forEach(mapNodeModule), no need for lambda.

src/PreserveModuleNamePlugin.ts

jods4 · 2018-10-11T22:04:42Z

src/PreserveModuleNamePlugin.ts

-          return true;
-        }
+      if (m[preserveModuleName]) {
+        return true;


It seems to me that you have significantly changed the behavior of this.
We did not just check if some dependency had preserveModuleName set.

If the dependency used an absolute module id, we would use it as a module name, as this is more robust than trying to figure out the normalized id ourselves. See line 82 before PR.

Unless you have a good reason to remove that behavior, I would like to revert this change.

I am confused by this. Why would we set a modules id based on one of its dependencies? ModuleA could require ModuleB but we dont want to set ModuleA's id based on ModuleB which might or might not live in the same module directory.

Well I said it here:
If the dependency used an absolute module id, we would use it as a module name, as this is more robust than trying to figure out the normalized id ourselves.

Not sure how to say it better.
Normalizing the module names for aurelia-loader is a can of worms. It works rather well but there are lots of edge cases, this PR is proof of that.
So when we are handed out an absolute module id, that's great because it means we don't have to make guesses at normalization and it's guaranteed to work.
It's also marginally faster.

jods4 · 2018-10-11T22:19:01Z

I'm sorry I did not come back to this earlier. It needs a lot of focus and I've been working a little relentlessly those past months.

Thanks for the changes you've done so far, it's already easier to look at for me (less modifications).

The general changes I am fine with, modulo:

2 or 3 subtle behavior changes that might be unintended. I've pointed them out in comments, just clarify them for me. If they're unintended a revert should be quick.
some style nitpicking, nothing important.

I like the change where you modify user code to inject the normalized module id instead of original one. I think that's a good change 👍

Then there's the node_modules stuff. It's a big change with lots of complicated new code.
I'll try to read it again over the week-end, I'm too tired to do that now. I need to ensure that it works in many situations such as Windows vs linux (path manipulation), nested node_modules, symlinks... those are all common problems with node modules.

It would really help me a lot if you could summarize in a few word what use case you're trying to improve here. Can you provide an example of something that didn't work before and works with the new code?

I'm sorry it takes so long and is so hard to merge but that's a big change to the most tricky parts of this plugin.

pat841 · 2018-10-16T13:00:20Z

@jods4 PR updated with changes.

As for WHY this PR is needed, you can see the original PR #122. This PR is a simplified version of that which fixes the original bug #121.

jods4 · 2018-10-17T19:32:30Z

@pat841 I read issue #121 again and I'm confused.
There is a huge change in this PR related to how node_modules is handled, yet the issue is all about how aliases are handled.
Did I miss something?
Or what did you change in node_modules that makes it better for aliases specifically?

pat841 · 2019-02-20T16:38:40Z

@jods4 The issue was aliases were never handled properly when mixed with relative/absolute paths. This new approach creates a mapping of all normalized paths and replaces the alias/relative/absolute path in code with the normalized path.

Maybe we could tag this as beta/possible breaking and pre-release until were certain its stable? I have been using this PR in multiple projects of mine without issue .

EisenbergEffect · 2019-02-27T20:18:48Z

@pat841 We appreciate the awesome work you've put into this PR. One of the things that is holding this up is it's raw size. We usually like to work in smaller increments when we can. I wonder if there's a way that we could break this up into a set of successive, smaller PR, each addressing one issue. That would make the process go faster, it would be easier to review, test, etc.

Any thoughts on this? @bigopon and @fkleuver have done a good job of this on a few of our other libraries, so maybe they have some specific recommendations in this case.

fkleuver · 2019-02-28T02:54:11Z

I must admit that I often fail at breaking things up into smaller chunks as well. However, what I do to mitigate risks is add lots and lots of tests.

Tests are usually easier to review than code. You can look at simple inputs and outputs and verify whether they are appropriate, instead of having to reverse-engineer logic and guessing whether it might work or not.

So my recommendation would be to add tests. Whatever makes sense. A few unit tests could be good where possible (e.g. there's no need for a full project structure) and for the large areas it could be a full "e2e" tests on some real physical project structures where you have an input project and compare the processed output to a pre-generated one. Just to give an example. That way you can cover a lot of ground with relatively little work

…relia-loader module id [SIMPLE]

pat841 mentioned this pull request Feb 26, 2018

fix(webpack): Aliased module paths now properly map to the correct aurelia-loader module id #122

Open

pat841 force-pushed the simple branch from 3a48255 to fe5933d Compare February 26, 2018 17:24

pat841 force-pushed the simple branch from fe5933d to e48aaff Compare February 26, 2018 18:45

pat841 force-pushed the simple branch 3 times, most recently from 15eb138 to 568dc37 Compare June 26, 2018 05:17

pat841 force-pushed the simple branch from 568dc37 to 146dbc8 Compare June 26, 2018 05:21

jods4 suggested changes Jul 2, 2018

View reviewed changes

pat841 mentioned this pull request Sep 13, 2018

Alias is not resolved for compose view-model with class reference #152

Open

jods4 reviewed Oct 11, 2018

View reviewed changes

src/PreserveModuleNamePlugin.ts Show resolved Hide resolved

jods4 reviewed Oct 11, 2018

View reviewed changes

pat841 force-pushed the simple branch 2 times, most recently from 1e634ff to 103d9b7 Compare October 16, 2018 12:54

pat841 force-pushed the simple branch from 103d9b7 to 219ae08 Compare October 16, 2018 12:55

pat841 force-pushed the simple branch from 219ae08 to 5426b42 Compare October 16, 2018 13:26

fix(webpack): Aliased module paths now properly map to the correct au…

2299267

…relia-loader module id [SIMPLE]

pat841 force-pushed the simple branch from 5426b42 to 2299267 Compare November 8, 2019 18:02

bigopon force-pushed the master branch from 24764d0 to e028469 Compare September 1, 2021 11:25

fix(webpack): Aliased module paths now properly map to the correct aurelia-loader module id [SIMPLE] #139

Are you sure you want to change the base?

fix(webpack): Aliased module paths now properly map to the correct aurelia-loader module id [SIMPLE] #139

Conversation

pat841 commented Feb 26, 2018

CLAassistant commented Feb 26, 2018 • edited Loading

jods4 commented Feb 26, 2018

pat841 commented Feb 28, 2018

rek72-zz commented Jun 12, 2018

Alexander-Taran commented Jun 13, 2018

pat841 commented Jun 14, 2018

rek72-zz commented Jun 14, 2018

jods4 commented Jun 16, 2018

pat841 commented Jun 26, 2018

jods4 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pat841 commented Aug 27, 2018

pat841 commented Sep 26, 2018

pat841 commented Oct 10, 2018

EisenbergEffect commented Oct 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jods4 commented Oct 11, 2018

pat841 commented Oct 16, 2018

jods4 commented Oct 17, 2018

pat841 commented Feb 20, 2019 • edited Loading

EisenbergEffect commented Feb 27, 2019 • edited Loading

fkleuver commented Feb 28, 2019

CLAassistant commented Feb 26, 2018 •

edited

Loading

pat841 commented Feb 20, 2019 •

edited

Loading

EisenbergEffect commented Feb 27, 2019 •

edited

Loading