-
-
Notifications
You must be signed in to change notification settings - Fork 531
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support .mjs
output [also a TypeScript issue]
#436
Comments
@daflair I honestly haven't looked into what supporting ES6 modules natively in node.js will entail yet. Since it doesn't use any of the module loading legacy, I'm not sure how we can hook into it. If you have details, feel free to post them - I'm on vacation one more week and can review at a later date. I'll always accept a PR also 😄 |
I really wish I was suited to PR, but I am a lone developer at the moment. I've played around with ts-node a little and even looked into Besides, the more I think about it, I realize that this would require a different mechanism than legacy, I would be very interested in more discussion and even collaborating any time. I guess what I am trying to say is that having a solid understanding of something like ts-node is essential to architect a new mechanism and avoid the hassles of lessons learned. So please enjoy your vacation, but let's revisit this on your schedule :) [I'll update notes in this thread in meanwhile] |
Hi, |
It seems that with the exposed (not behind flags except for --experimental-modules) basically mean that you can intercept the first call to Module._load and do whatever you need (like copy or symlink with .mjs) but unfortunately, once you pass off the mjs to the _load, loading is done internally inside an internal Loader that is not accessible. So I have a few points of discussion on some potential solutions that could be refined with some discussions when possible. |
There is interesting discussion, and it seems that node is coming with a completely new "ES-module-based hooks" module: resolve and instantiate loader pipeline hooks #15445 From my testing, the existing internal and exposed aspects of the loader seem to reset their prototype chain inside each import's execution context, so a patch was only good for imports that occur directly, but imports by imports did not trigger the hooks as intended (or at least I could not get them to) So the great news is while I was going all out trying to figure out this intentionally inextensible hell they have been working a opening it up anyways :) When this lands, it looks like it will make life so much better moving forward with standard modules. Here is what a custom-loader might look like: // node/test/fixtures/es-module-loaders/example-loader.mjs
import url from 'url';
import path from 'path';
import process from 'process';
const builtins = new Set(
Object.keys(process.binding('natives')).filter((str) =>
/^(?!(?:internal|node|v8)\/)/.test(str))
);
const JS_EXTENSIONS = new Set(['.js', '.mjs']);
export function resolve(specifier, parentModuleURL/*, defaultResolve */) {
if (builtins.has(specifier)) {
return {
url: specifier,
format: 'builtin'
};
}
if (/^\.{0,2}[/]/.test(specifier) !== true && !specifier.startsWith('file:')) {
// For node_modules support:
// return defaultResolve(specifier, parentModuleURL);
throw new Error(
`imports must begin with '/', './', or '../'; '${specifier}' does not`);
}
const resolved = new url.URL(specifier, parentModuleURL);
const ext = path.extname(resolved.pathname);
if (!JS_EXTENSIONS.has(ext)) {
throw new Error(
`Cannot load file with non-JavaScript file extension ${ext}.`);
}
return {
url: resolved.href,
format: 'esm'
};
} |
I checked-out guy's branch (which is already 50% approved and CI-checked and I actually got it to work. Of course this is only a custom 9.0.0-pre build of node on my own Mac but I am running node's tests all the same just to see how stable this is for others to actually play around with. But here is the output:
Documentationfor reference only Loader hooksTo customize the default module resolution, loader hooks can optionally be When hooks are used they only apply to ES module loading and not to any Resolve hookThe resolve hook returns the resolved module file URL for a given module import url from 'url';
export async function resolve(specifier, parentModuleURL, defaultResolver) {
return {
url: new URL(specifier, parentModuleURL).href,
format: 'esm'
};
} The default NodeJS ES module resolution function is provided as a third In addition to returning the resolved file URL value, the resolve hook also For example a dummy loader to load JavaScript restricted to browser resolution import url from 'url';
import path from 'path';
import process from 'process';
const builtins = new Set(
Object.keys(process.binding('natives')).filter((str) =>
/^(?!(?:internal|node|v8)\/)/.test(str))
);
const JS_EXTENSIONS = new Set(['.js', '.mjs']);
export function resolve(specifier, parentModuleURL/*, defaultResolve */) {
if (builtins.has(specifier)) {
return {
url: specifier,
format: 'builtin'
};
}
if (/^\.{0,2}[/]/.test(specifier) !== true && !specifier.startsWith('file:')) {
// For node_modules support:
// return defaultResolve(specifier, parentModuleURL);
throw new Error(
`imports must begin with '/', './', or '../'; '${specifier}' does not`);
}
const resolved = new url.URL(specifier, parentModuleURL);
const ext = path.extname(resolved.pathname);
if (!JS_EXTENSIONS.has(ext)) {
throw new Error(
`Cannot load file with non-JavaScript file extension ${ext}.`);
}
return {
url: resolved.href,
format: 'esm'
};
} With this loader, running:
would load the module Dynamic instantiate hookTo create a custom dynamic module that doesn't correspond to one of the export async function dynamicInstantiate(url) {
return {
exports: ['customExportName'],
execute: (exports) => {
// get and set functions provided for pre-allocated export names
exports.customExportName.set('value');
}
};
} With the list of module exports provided upfront, the |
Hiya! Let me know if you are looking for any other hooks in particular. The loader is actually much more relaxed that I planned on making it initially to avoid people intentionally spilling internals; but, am open to figuring out hooks for official ways to do use cases. |
Amazing… :) I'm kind of exploring quickly some ideas for abstractions on top of that PR to see if there will be any snags. Cheers |
Here is a gist (yes, still using gists) of what I have so far… A simple TypeScript loader using NodeJS 9+ declarative loader API |
@bmeck I just have to clarify, I'm simply a user of ts-node with a very strong motivation to get to a point where ES2015 modules finally deliver what they really promised. Please don't misunderstand what I've been posting here as the opinion of TypeStrong or the ts-node folks. That said, I am very happy with the Some Thoughts I realize that many loaders will be design-time-relevant, so at this point it is best hidden behind a CLI/bin, but now what happens if more than one loader is to be used? Some production-relevant loaders can also be important to certain applications, so what happens then, can they be imperatively loaded? |
Per package loaders are being designed, but global loader was first step towards that. You need a concept of per package loaders to be guard against when people are using non-standard behavior, and a mechanism to default to the parent loader (so that APM/code coverage/testing hooks still can be applied).
No plans to do so. Is there an example of a loader that is not known ahead of time that you are thinking of? If nothing else, those loaders could always be loaded and we could have a |
@bmeck Okay, so I realize that this will seem like a request to expose ModuleWrap (which I think you are not in favour of) but this really is not, we just need to be able to createDynamicModules from raw ESM source code on the fly. Here's why… Currently dynamic modules (designed to facilitate wrapping cjs) work well because cjs source can be evaluated before wrapping, and returns a living exports object that can then be reflected. Any linking is really done in response to the actual require calls that occur while evaluating the actual module. However, when ES code is transpiled by a loader, the only way to replicate the above mechanics would be to statically analyze imports and exports (ASTing and stuff) since the code can't be evaluated until it is a wrapped Module. So potentially, the loader might actually need to mirror ModuleJob's behaviour in order to resolve links, which are potentially also dynamic modules or not… So forget about the performance cost of statically analyzing, probing loader states and mimicking loader pipelines, at the end of the day, all it takes is one wrong assumption about the real loader's logic and/or the changes in it's behaviour introduced by x other custom loader(s) to make loaders break one-another. Currently, I avoided all this hassle by simply transpiling to a dot file right next to the actual file, and then switched the url's pathname to the ES substitute, adding a horrific process.on('exit') delete dot files handler. Now for the down sides of using files… who's file is it anyway, and can it be deleted or overwritten, do we have write permission, what happens if the filename was actually a more natural way of naming more important things for this user, and oh, what if it was not even a file to begin with. :) Proposal Would it not be elegant enough to pass ES code as string for a module. So a custom-loader's job would be to load, not to link and instantiate, just load. If it needs to transpile, so be it, whatever that takes, even if it assumes certain linking and instantiation behaviour based on norms that are configured following certain design-time parameters, it is all done in isolation from the runtime. Extension I guess an extension on the idea would to let loaders pass various metadata in the object that would be passed along the loader pipeline. Currently, resolvers already return an object with So to accomplish the ES code as string, one would need an ESSourceLoader, and that is most likely going to be a node-supplied loader because of it's internal requirements. Now a TranpilingLoader returns The biggest downside of this is security, HackingLoader can simply do what it wants, but that can be accomplished regardless of this approach and the safe guards needed at this point should be sufficient with this extension in my opinion. |
I completely agree and am stetting up standards groundwork to let us use const compile = (pathname) => {
const input = `${fs.readFileSync(pathname)}`;
const {
outputText, diagnostics, sourceMapText
} = ts.transpileModule(input, { compilerOptions });
const blob = new Blob([outputText], {type: 'text/javascript'});
return URL.createObjectURL(blob);
} This has required me going around and making some changes in MIME types, https://github.com/bmeck/I-D/tree/master/javascript-mjs + nodejs/TSC#371 . It also will probably mean minor tweaks to the loader APIs to be MIME based and uniform across different APIs. But I think solves your use case of needing to keep the source text in memory. I have been avoiding transformation hooks precisely because you have multiple loaders writing to the same URL location. That would mean loaders cannot rely on the source text at that URL. As for meta-data, it is interesting but can be done with a a Map like you are doing right now. I'm open to things if they appear to be blockers over time, but for now I think getting the MIME work done and |
While I have someone looking into things, have you looked at cross worker requirements you might have? |
I recommend staying away from this pattern, blobs worked well in the web for pseudo-static content (predefined data that is loaded on demand, or dynamic generated data like processed images that may only interdepend on previously generated or static data) but did not offer much for modules (or we would have seen the new WebBlobs or webpack-es-blob-loader by now). When I explored using that pattern for dynamic modules in the browser, I quickly found several pain-points which cannot be avoided and make them very limiting. So I can't imagine how this would pan out for node, not to mention the additional requirements and pitfalls for the rich body of loaders that drive node applications today. Assuming a simple graph, loaders will have an additional responsibility of creating the dynamic urls, then surgically replacing So yes, theoretically it works if we only care about defining a one-off module using a But in reality, module graphs are a lot more complex, which is what we wanted from ES modules. In that case, they cannot work because unlike how ModuleJobs deal with mutable states, a url-from-blob is an immutable value that cannot be reserved ahead of time, it cannot be promisified or redefined. So once you create it, it must be it, that module needs to be fully-linked and ready to instantiate as is. Any way to work around the immutable nature of object-urls is way too messy, it opens up too many possibilities for leaks, and I imagine that is one reason why whatwg is pursuing the topic of loaders. That was what I learned going the long and hard, but maybe I missed something, can blob contents be changed? or can the createObjectURL url's be coerced to be something other than random-unqiue value that is returned after creating and passing the blob to which it would refer. |
I also have to mention one advantage that exists in node which mediates some but not all of the potential challenges for using The flip-side of course is that with chained loaders, each will have to create a FileReader, read the blob, decide if they have something to change, then create a new Blob, a new URL and return that. At this point, I have to ask, why not allow My underlaying assumption when I think multiple-loaders, which might help clarify my position, is that the order of loaders when it comes to resolving modules would be predetermined. For instance, I have a first loader that does global grooming (like injecting dynamic or sensitive content), then if it is .tsx? that loader compiles it, at this point, it might pass through a loader that does general es-related changes. The order of those loaders for each individual module should be predictable and even better explicitly configurable. However, I also assume that while some module's dependencies might still be passing through their first loader, some will be far ahead, maybe even passed through all loaders, so a loader should treat each module independently enough to allow for performance gains from running modules in parallel. |
@daflair I am not sure the complexity is true since the loader would be given the import specifiers and not need to rewrite the code. Lets make a more concrete example of:
As a pipeline and a given text of:
The composition aspect is still being worked out but lets assume a Using static URLs
... ESM mechanics start from spec ...
If we were to do source code in place
... ESM mechanics start from spec ...
I think both work fine, but one has explicit URLs for each step so that loaders don't have to worry about mutation from other loaders. In the mutating URL example any location data needs to be excluded from |
@bmeck I can definitely see the benefits from node's perspective… I can also see now how that can also be leveraged from a loader's perspective, different paradigm, different pros/cons. So I'll keep thinking of potential things that may need guards below:
Let me think on this a little throughout the day. |
@bmeck There is currently a difference between how resolve and dynamicInstantiate are hooked, one exposes the loader and the other does not, is that intentional? Is there a sure way to access the module being instantiated without having to crawl the actual loader instance. Or maybe this is intentional, which would make sense if you are trying to ensure static loaders are lightweight and dynamic loaders are as flexible as necessary, but that's just me speculating at this point. |
@SMotaal Neither exposes the loader. Are you talking about the reflective API for exports in our ESM facade?
No, by design; problems around doing so since it isn't actually mutable like a regular JS Object (hence the funky reflection API).
Intentional yes, but also there are questions about removing
They could, but it probably wouldn't be reliable since once something is in the Module Map, it is idempotent (either the local one per ESM, or the global one).
For now I would say yes. May change in the future, but see problems with Java's URLConnection. Things like TLS CA management, cookie management, mandated HTTP proxy etc. are example problems for us trying to make a generic approach until further research to solve these problems is figured out. If we provide one for
They are completely independent. In this case a |
So if dynamicInstantiate hook is kept [please keep it] We should assume a change like |
Yup
On Nov 1, 2017 12:14 PM, "Saleh Abdel Motaal" <[email protected]> wrote:
So if dynamicInstantiate hook is kept [please keep it]
Regarding:
https://github.com/nodejs/node/blob/18df171307495e639be214917d2a24
7163a21268/lib/internal/loader/Loader.js#L37-L40
We should assume that a change like this.dynamicInstantiate =
dynamicInstantiate.bind(null); which would isolate the method from the
lexical context of the loader?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#436 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAOUo0Z6Y-qNMb-VSadDDhboJgzOdbDFks5syKbfgaJpZM4PnDYm>
.
|
for the meantime, how can I get ts-node to transpile to commonjs/node style modules, that should fix these errors right
Update: Seems the issue was that I was not also doing |
@blakeembrey… thank you for leaving this open all this time. 🎉This meant a lot to me personally — thanks!
|
…ode), but ts-node wouldn't have it. (see issue: TypeStrong/ts-node#436) * Converted last of .js files in Scripts, to .ts. * MS own-modules are always loaded from root. (I manage them, so it's easy for me to make sure they all work together -- ie. the advantage of honoring semver for lib subdependencies is lower, when those subdependencies are my own modules) BTW, note that latest chrome (79) has issue when debugging source-mapped file: hovering over variables in the Sources text-editor doesn't display anything. I know this is a Chrome bug, since it works fine in Chromium, and the bug shows up in Chrome even on the actual canonicaldebate.com, which I haven't touched in months. I'll just wait for the fix rather than trying to work around.
The code in the comment just above can be done via the code below
|
I recently implemented and released experimental ESM support. Feedback is being tracked in #1007. Please take it for a spin and let us know if it works. |
I guess we are kind of being forced a little here but someone has to be the bigger guy here :)
Can you please make it possible to plug into node's new mjs ESMLoader workflow?
There is a TypeScript issue open already.
Some notes:
must runnode --experimental-modules something.mjs
(yes mjs)that--experimental-modules
will go away obviously.we can't really force it on non-mjs, unless we get the nodejs team is willing to open things a little.npm install renamer -g
thenrenamer -regex --find '\.js^' --replace '.mjs' './outDir/**/*.js'
as a solid contribution to making things just work.You can transpile inline and produce standard-compliant javascript that does not get shredded into commonjs, as long as you can name it mjs.Further references:
{ ModuleWrap } = process.bindings('module_wrap')
so not really internal "no-touch" but not officially supported and can break.The text was updated successfully, but these errors were encountered: