Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

speed up the installation of dependencies #1030

Merged
merged 1 commit into from
Nov 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12,036 changes: 3,604 additions & 8,432 deletions package-lock.json

Large diffs are not rendered by default.

15 changes: 7 additions & 8 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"scripts": {
"build": "tsc --build && npm run copy-static && npm run build-bundles",
"copy-static": "copyfiles 'src/browser/client-scripts/*' build",
"build-node-bundle": "esbuild ./src/bundle/index.ts --outdir=./build/src/bundle --bundle --format=cjs --platform=node --target=ES2021",
"build-node-bundle": "esbuild ./src/bundle/cjs/index.ts --outdir=./build/src/bundle/cjs --bundle --format=cjs --platform=node --target=ES2021",
"build-browser-bundle": "node ./src/browser/client-scripts/build.js",
"build-bundles": "concurrently -c 'auto' 'npm:build-browser-bundle' 'npm:build-node-bundle --minify'",
"check-types": "tsc --project tsconfig.spec.json",
Expand Down Expand Up @@ -55,14 +55,9 @@
"@gemini-testing/commander": "2.15.4",
"@jspm/core": "2.0.1",
"@puppeteer/browsers": "2.4.0",
"@types/debug": "4.1.12",
"@types/yallist": "4.0.4",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these types packages should be declared in dev deps

"@vitest/spy": "2.1.4",
"@wdio/globals": "8.39.0",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not used anymore, so I remove it

"@wdio/protocols": "8.38.0",
"@wdio/types": "8.39.0",
"@wdio/utils": "8.39.0",
"@wdio/utils-cjs": "npm:@wdio/[email protected]",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move to dev deps, because now I bundle this package using esbuild

"bluebird": "3.5.1",
"chalk": "2.4.2",
"clear-require": "1.0.1",
Expand All @@ -78,7 +73,6 @@
"geckodriver": "4.5.0",
"gemini-configparser": "1.4.1",
"get-port": "5.1.1",
"glob-extra": "5.0.2",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move to dev deps. This package now also bundled

"import-meta-resolve": "4.0.0",
"local-pkg": "0.4.3",
"lodash": "4.17.21",
Expand Down Expand Up @@ -116,7 +110,7 @@
"@commitlint/config-conventional": "^19.0.3",
"@cspotcode/source-map-support": "0.8.0",
"@sinonjs/fake-timers": "10.3.0",
"@swc/core": "1.3.40",
"@swc/core": "1.9.2",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Install fresh version of swc/core. Because with old version I got errors in CI builds.

"@types/babel__code-frame": "7.0.6",
"@types/babel__core": "7.20.5",
"@types/bluebird": "3.5.38",
Expand All @@ -125,6 +119,7 @@
"@types/chai-as-promised": "7.1.5",
"@types/clear-require": "3.2.1",
"@types/cli-progress": "3.11.6",
"@types/debug": "4.1.12",
"@types/escape-string-regexp": "2.0.1",
"@types/fs-extra": "11.0.4",
"@types/lodash": "4.14.191",
Expand All @@ -137,8 +132,11 @@
"@types/sinonjs__fake-timers": "8.1.2",
"@types/urijs": "1.19.25",
"@types/url-join": "4.0.3",
"@types/yallist": "4.0.4",
"@typescript-eslint/eslint-plugin": "6.12.0",
"@typescript-eslint/parser": "6.12.0",
"@wdio/types": "8.39.0",
"@wdio/utils-cjs": "npm:@wdio/[email protected]",
"aliasify": "1.9.0",
"app-module-path": "2.2.0",
"browserify": "13.3.0",
Expand All @@ -151,6 +149,7 @@
"eslint": "8.25.0",
"eslint-config-gemini-testing": "2.8.0",
"eslint-config-prettier": "8.7.0",
"glob-extra": "5.0.2",
"husky": "0.11.4",
"jsdom": "^24.0.0",
"jsdom-global": "3.0.2",
Expand Down
6 changes: 3 additions & 3 deletions src/browser/browser.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
import crypto from "crypto";
import { RequestOptions } from "https";

import { RemoteCapability } from "@wdio/types/build/Capabilities";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found solution how to remove this hack

import _ from "lodash";

import { SAVE_HISTORY_MODE } from "../constants/config";
Expand All @@ -16,6 +14,8 @@ import { BrowserConfig } from "../config/browser-config";
import Callstack from "./history/callstack";
import type { WdProcess, WebdriverPool } from "../browser-pool/webdriver-pool";

import type { Capabilities } from "@wdio/types";

const CUSTOM_SESSION_OPTS = [
"outputDir",
"agent",
Expand Down Expand Up @@ -184,7 +184,7 @@ export class Browser {
return this._state;
}

get capabilities(): RemoteCapability {
get capabilities(): Capabilities.RemoteCapability {
return this.publicAPI.capabilities;
}

Expand Down
2 changes: 1 addition & 1 deletion src/browser/existing-browser.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ const url = require("url");
const Promise = require("bluebird");
const _ = require("lodash");
const webdriverio = require("webdriverio");
const { sessionEnvironmentDetector } = require("@wdio/utils-cjs");
const { sessionEnvironmentDetector } = require("../bundle/@wdio-utils");
const { Browser } = require("./browser");
const commandsList = require("./commands");
const Camera = require("./camera");
Expand Down
5 changes: 5 additions & 0 deletions src/bundle/@wdio-utils.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
/* eslint-disable @typescript-eslint/no-var-requires */
const bundle = require("./cjs");
Comment on lines +1 to +2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we not use require here?
If import does not work here, i think const bundle = import("./cjs") would.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can use import, but then typescript build this file and store it inside build folder. And it's not a big problem, but it will not be used in production code. In production code only final bundle file is used. So because of this, I use require here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And now typescript wouldn't build this file?


export const sessionEnvironmentDetector: typeof import("@wdio/utils-cjs").sessionEnvironmentDetector =
bundle.wdioUtils.sessionEnvironmentDetector;
3 changes: 3 additions & 0 deletions src/bundle/cjs/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export * from "./test-transformer";
export * from "./vendors/@wdio/utils-cjs";
export * from "./vendors/glob-extra";
193 changes: 193 additions & 0 deletions src/bundle/cjs/test-transformer.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
import * as nodePath from "node:path";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just moved from src/bundle/test-transformer.ts.

import * as babel from "@babel/core";
import { addHook } from "pirates";
import { requireModuleSync } from "../../utils/module";

import type { NodePath, PluginObj, TransformOptions } from "@babel/core";

const STYLE_EXTESTION_RE = /\.(css|less|scss|sass|styl|stylus|pcss)$/;
const IGNORE_STYLE_ERRORS = ["Unexpected token"];

export const TRANSFORM_EXTENSIONS = [".jsx", ".cjsx", ".mjsx", ".tsx", ".ctsx", ".mtsx"];
export const JS_EXTENSION_RE = /^\.([cm]?[tj]sx?|json)$/;

export const setupTransformHook = (opts: { removeNonJsImports?: boolean } = {}): VoidFunction => {
const transformOptions: TransformOptions = {
browserslistConfigFile: false,
babelrc: false,
configFile: false,
compact: false,
presets: [require("@babel/preset-typescript")],
sourceMaps: "inline",
plugins: [
[
require("@babel/plugin-transform-react-jsx"),
{
throwIfNamespace: false,
runtime: "automatic",
},
],
require("@babel/plugin-transform-modules-commonjs"),
],
};

const customIgnoreImportsPlugin = ({ types: t }: { types: typeof babel.types }): PluginObj => ({
name: "ignore-imports",
visitor: {
ImportDeclaration(path: NodePath<babel.types.ImportDeclaration>): void {
const extname = nodePath.extname(path.node.source.value);

if (extname && !extname.match(JS_EXTENSION_RE)) {
path.remove();
return;
}

try {
requireModuleSync(path.node.source.value);
} catch (err) {
if (shouldIgnoreImportError(err as Error)) {
path.remove();
return;
}

if ((err as NodeJS.ErrnoException).code === "ERR_REQUIRE_ESM") {
mockEsmModuleImport(t, path);
return;
}
}
},
},
});

if (opts.removeNonJsImports) {
transformOptions.plugins!.push([customIgnoreImportsPlugin]);
}

const revertTransformHook = addHook(
(originalCode, filename) => {
return babel.transform(originalCode, { filename, ...transformOptions })!.code as string;
},
{ exts: TRANSFORM_EXTENSIONS },
);

return revertTransformHook;
};

function shouldIgnoreImportError(err: Error): boolean {
const shouldIgnoreImport = IGNORE_STYLE_ERRORS.some(ignoreImportErr => {
return (err as Error).message.startsWith(ignoreImportErr);
});

if (!shouldIgnoreImport) {
return false;
}

const firstStackFrame = (err as Error).stack?.split("\n")[0] || "";
const filePath = firstStackFrame.split(":")[0];
const isStyleFilePath = STYLE_EXTESTION_RE.test(filePath);

return isStyleFilePath;
}

/**
* Replace esm module import with a Proxy.
* Examples:
* 1) `import pkg from "package"` -> `const pkg = new Proxy({}, {get: ..., apply: ...})`
* 2) `import {a, b as c} from "package"` -> `const {a, c} = new Proxy({}, {get: ..., apply: ...})`
*/
function mockEsmModuleImport(t: typeof babel.types, path: NodePath<babel.types.ImportDeclaration>): void {
const variableKey = genVarDeclKey(t, path.node);
const variableValue = genProxy(t, [
t.objectExpression([]),
t.objectExpression([genProxyGetHandler(t), genProxyApplyHandler(t)]),
]);

const variableDecl = t.variableDeclaration("const", [t.variableDeclarator(variableKey, variableValue)]);

path.replaceWith(variableDecl);
}

/**
* Generates the name of variables from the import declaration.
* Examples:
* 1) `import pkg from "package"` -> `pkg`
* 2) `import {a, b as c} from "package"` -> `const {a, с} `
*/
function genVarDeclKey(
t: typeof babel.types,
node: NodePath<babel.types.ImportDeclaration>["node"],
): babel.types.Identifier | babel.types.ObjectPattern {
if (node.specifiers.length === 1) {
if (["ImportDefaultSpecifier", "ImportNamespaceSpecifier"].includes(node.specifiers[0].type)) {
return t.identifier(node.specifiers[0].local.name);
}

return t.objectPattern([
t.objectProperty(
t.identifier(node.specifiers[0].local.name),
t.identifier(node.specifiers[0].local.name),
false,
true,
),
]);
}

const objectProperties = node.specifiers.map(spec => {
return t.objectProperty(t.identifier(spec.local.name), t.identifier(spec.local.name), false, true);
});

return t.objectPattern(objectProperties);
}

// Generates Proxy expression with passed arguments: `new Proxy(args)`
function genProxy(t: typeof babel.types, args: babel.types.Expression[]): babel.types.NewExpression {
return t.newExpression(t.identifier("Proxy"), args);
}

/**
* Generates "get" handler for Proxy:
*
* get: function (target, prop) {
* return prop in target ? target[prop] : new Proxy(() => {}, this);
* }
*/
function genProxyGetHandler(t: typeof babel.types): babel.types.ObjectProperty {
return t.objectProperty(
t.identifier("get"),
t.functionExpression(
null,
[t.identifier("target"), t.identifier("prop")],
t.blockStatement([
t.returnStatement(
t.conditionalExpression(
t.binaryExpression("in", t.identifier("prop"), t.identifier("target")),
t.memberExpression(t.identifier("target"), t.identifier("prop"), true),
genProxy(t, [t.arrowFunctionExpression([], t.blockStatement([])), t.thisExpression()]),
),
),
]),
),
);
}

/**
* Generates "apply" handler for Proxy:
*
* apply: function () {
* return new Proxy(() => {}, this);
* }
*/
function genProxyApplyHandler(t: typeof babel.types): babel.types.ObjectProperty {
return t.objectProperty(
t.identifier("apply"),
t.functionExpression(
null,
[],
t.blockStatement([
t.returnStatement(
genProxy(t, [t.arrowFunctionExpression([], t.blockStatement([])), t.thisExpression()]),
),
]),
),
);
}
2 changes: 2 additions & 0 deletions src/bundle/cjs/vendors/@wdio/utils-cjs.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
import * as wdioUtilsFns from "@wdio/utils-cjs";
export const wdioUtils = wdioUtilsFns;
2 changes: 2 additions & 0 deletions src/bundle/cjs/vendors/glob-extra.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
import * as globExtraFns from "glob-extra";
export const globExtra = globExtraFns;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and previous package also bundled by esbuild

2 changes: 0 additions & 2 deletions src/bundle/constants.ts

This file was deleted.

6 changes: 6 additions & 0 deletions src/bundle/glob-extra.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
/* eslint-disable @typescript-eslint/no-var-requires */
const bundle = require("./cjs");

export const expandPaths: typeof import("glob-extra").expandPaths = bundle.globExtra.expandPaths;
export const isMask: typeof import("glob-extra").isMask = bundle.globExtra.isMask;
export type { GlobOpts, ExpandOpts } from "glob-extra";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is used for reexport methods from glob-extra and specify type for them, because bundled file doesn't have types

2 changes: 0 additions & 2 deletions src/bundle/index.ts

This file was deleted.

Loading