Avoid repeating npm install on each exercise

[rchavarria]

Some people find it tedious and a waste of time to have to install NodeJS packages everytime they want to solve an exercise (take a look at exercism.io#2972).

In that discussion, we reached to a solution: improve the documentation on /docs/INSTALATION.md to guide users to install all dependencies on <EXERCISM_HOME>/ecmascript and add that folder to the PATH system variable. This way, all mandatory tools to run tests will be available on the path and there will be no need to npm install on each exercise.

[tejasbubane]

Adding PATH (and NODE_PATH as some solutions suggest) did not work for me.

Changing directory is not and won't ever be supported I suppose: npm/npm#775

[tejasbubane]

Can we switch to yarn? It is crazy fast (more so with cached packages). Here is a comparison of npm and yarn on one exercise:

$ time npm install
....
npm install  31.45s user 9.02s system 34% cpu 1:58.32 total

PS: Notice the total time there^

without cache:

$ time yarn install --force
......
✨  Done in 32.49s.
yarn install --force  20.74s user 10.64s system 93% cpu 33.392 total

with cache:

$ rm -rf node_modules
$ time yarn install 
.....
✨  Done in 15.03s.
yarn install  9.37s user 5.12s system 90% cpu 15.977 total

Considerably less frustrating than waiting for npm install to complete for every exercise.

[rchavarria]

I see. Maybe, installing dependencies in a different directory that the working directory is not a good idea, based on npm/npm#775.

Yep, I've heard yarn is much faster installing packages than npm. But I'm a bit afraid about forcing users to install yet another tool. From a novice exercism user, it can be overwhelming.

Is yarn 100% compatible with npm's package.json file? At least about installing dependencies and running scripts? If so, documentation could be updated with a note about the possibility of using yarn. Does installing yarn require any change? If not, it's just a matter of taste. Exercism users can use whatever tool they want.

[tejasbubane]

Yarn is compatible with npm in terms of pulling packages from npm registry and running scripts, although it does not have all commands/options that npm does.
ref: https://yarnpkg.com/en/docs/migrating-from-npm I think in that way it is not a drop-in replacement for any project. But for our purposes, installing packages and running tests - it can be treated as npm replacement. I have been using it with ecmascript exercises for a while now and works without a hassle.

I get that it will be an added step for beginners since npm comes bundled with node. We can mention it as a faster alternative to npm and they can use either of the two.

[rchavarria]

Nice.

You could modify the /SETUP.md file, add some text under the Requirements chapter about the posibility to use yarn as an alternative to npm, and create a PR with your changes. The conversation can continue in that PR.

@exercism/javascript Any ideas on this npm Vs. yarn subject?

[masters3d]

I am using yarn in the TypeScript track. In addition to make being fast at install it also locks the packages so that everybody is using the exact version.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[joeljuca]

Is there any specific case that we would need specific packages for an exercise? I understand that students should have an easy way to test their solutions, but we solved a similar problem on exercism/xjavascript by using a global installation of Jasmine.

Since Exercism is supposed to teach a programming language and not libraries, platforms and/or tools, why not use a simpler approach?

[matthewmorgan]

@joelwallis at the moment (I believe) we need some sort of configuration file in each directory for the test runner (Jest.) I don't think we need an npm install for each exercise anymore. This is something that was needed in the past with the way we had the transpile-test pipeline set up.

Currently, we're using Jest with the babel-jest plugin to transpile and build just using the jest command.

If you're interested in tackling it, you would be more than welcome. If I can help in ay way, let me know.

[rchavarria]

I don't think we need an npm install for each exercise anymore.

Really? It'd be awesome. But, users need to install Jest globally I think. How would be the process right now? (just a few words, not a thorough description 😛 )

[matthewmorgan]

@rchavarria I think we can have a JSON file in each exercise that points jest at the specfiles in the current directory. That should be enough to make jest work.

But, I am wondering about the "grains" exercise, which requires the BigInt package. We might need to modify the makefile to still copy that file into the grains exercise dir. I'm not sure.

[joeljuca]

I don't know much about Jest. I'll take a closer look this weekend.

[tejasbubane]

I don't think we need an npm install for each exercise anymore. This is something that was needed in the past with the way we had the transpile-test pipeline set up.

@matthewmorgan If you mean installing jest globally and running it on each exercise, I don't think that will work. Since we still have more packages than jest itself, like babel-jest, preset-env. That means we will need to install all these packages globally.

That is, unless you have a better way that I am unaware of.

[rchavarria]

Closing and labeling as won't fix, agreed on #182

[aeamaea]

It's kinda hokey, but I just move the node modules from the completed exercise to the new one before running npm test for the first time. In all honestly, the base module set could just be cached and moved to each exercise as it's fetched and any deltas would be downloaded by npm anyway. seems to work for me. It's a huge waste and has an unacceptable environmental footprint as well. by the end of the JS track, each user would've download 9.5 gigabytes worth , multiply that with 86000 students (at the time of this writing) and you potentially have about 80 terabytes of node-modules down the wire that didn't really need to be downloaded. I'll leave the carbon footprint calculations for those who are interested in such things.

[aeamaea]

At 3kg / GB , it's about 3000kg / TB or 240,000 kilograms (half a million pounds) of carbon contributed to the atmosphere, just by the Javacript track on exercism.

It's probably peanuts compared to all the Netflix streaming that's going on, but I'm going to leave it out here anyway.

https://www.emergeinteractive.com/insights/detail/does-irresponsible-web-development-contribute-to-global-warming/

[SleeplessByte]

Peanuts or not, we can all do our part, yes? However, I don't get what you're trying to achieve with your comments. If you have something constructive or actionable, I'd like to hear it.

In all honestly, the base module set could just be cached and moved to each exercise as it's fetched and any deltas would be downloaded by npm anyway

"just" is relative and there is nothing simple about that setup for people new to this track. There are other options.

The JavaScript track is likely completely compatible with yarn 2's implementation of pnp, and to be honest, I was hoping by now that npm tink would be integrated and no longer experimental, but it's not.

• The reason we chose npm over yarn is pretty arbitrary, but the fact that npm comes with nodejs is a plus. For typescript it's less annoying to have the extra dependency, because typescript is one by itself (via tsc or babel).
• We probably want to stick with npm for now, until there is better/clearer onboarding for students.
• npm tink is still experimental.

For running tests in CLI, we actually do pretty much what you suggest:

1. it creates a workspace directory
2. it moves in all the files under test
3. it has a single copy of node_modules
4. it runs the tests

The above is conflict free (because the exercise slugs are unique), and is much fast than symlinking the node_modules directory everywhere. However, the set-up is non-trivial to understand, so it's unlikely we'll introduce it to the students at this time.

[aeamaea]

I'm new to JS, and it's a general comment, didn't mean to offend anyone. I think being aware of the cost of our day to day choices that seem small in the near term is a "good thing". I mentioned it for that very reason.

I tried to suggest what I'm doing , though it may not be a generalized solution (and therefore not actionable?) As I said, it's hokey but seems to work for me, for now.

I like the idea of single copy of node_modules. I'm not sure why it needs to be introduced to the students if the CLI can take care of it behind the scenes. Not trying to start another discussion on what the CLI should/shouldn't do. I'm happy with the CLI as it is.

P.S. I see the title of the article I posted, that was not my intent, I posted it as a reference to the number I used to do the carbon footprint calculation.

[SleeplessByte]

I think it's okay to suggest having a better way (aka revisiting this topic) with added benefit of trying to preserve the environment a bit more.

If you have the time, I wouldn't mind seeing your thoughts in a new issue, and then we/you/I/anyone can do further research to see what we can accomplish without too much pain and suffering for the students.

[Marloows]

Just moved node_modules and package-lock.json one folder up and everything seems magically to work :)

[SleeplessByte]

It does, and once #2548 lands, you don't need to do that anymore