NotADirectoryError when installing from source distribution

[thegamecracks]

When attempting to install the package specifically using its source distribution rather than the built wheels, the following error occurs:

>pip install reactpy --no-binary reactpy 
Collecting reactpy
  Using cached reactpy-1.0.2.tar.gz (123 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [26 lines of output]
      Traceback (most recent call last):
        File "..\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "..\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "..\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "..\AppData\Local\Temp\pip-build-env-qtfs6sj6\overlay\Lib\site-packages\hatchling\build.py", line 58, in build_wheel
          return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "..\AppData\Local\Temp\pip-build-env-qtfs6sj6\overlay\Lib\site-packages\hatchling\builders\plugin\interface.py", line 147, in build
          build_hook.initialize(version, build_data)
        File "..\AppData\Local\Temp\pip-build-env-qtfs6sj6\overlay\Lib\site-packages\hatch_build_scripts\plugin.py", line 51, in initialize
          run(cmd, cwd=str(work_dir), check=True, shell=True)  # noqa: S602
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "..\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 548, in run
          with Popen(*popenargs, **kwargs) as process:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "..\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 1026, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "..\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 1538, in _execute_child
          hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      NotADirectoryError: [WinError 267] The directory name is invalid
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

After some debugging, it appears that hatch is trying to build the JavaScript client and assumes src/js/ is present, but in the source distribution that directory is missing. Here is the build configuration in question:

reactpy/src/py/reactpy/pyproject.toml

Lines 127 to 136 in 3a3ad3f

	[[tool.hatch.build.hooks.build-scripts.scripts]]
	work_dir = "../../js"
	out_dir = "reactpy/_static"
	commands = [
	"npm ci",
	"npm run build"
	]
	artifacts = [
	"app/dist/"
	]

Right now, both the sdist and wheels uploaded on PyPI contain the reactpy/_static/ files generated by npm run build (inspector), but hatch is configured to build the client even when the source distribution already has the output files. This isn't a major issue since reactpy's wheels aren't dependent on system architecture, but it would be nice to have this part of the build process fixed (or at least documented).

The ideal solution would be to include the JS client's source code in the sdist and only build it when installing from source / generating the wheel. I thought of suggesting this:

+ [tool.hatch.build.targets.sdist.force-include]
+ "../../js" = "js"
+ 
+ [tool.hatch.build.targets.wheel.force-include]
+ "reactpy/_static" = "reactpy/_static"  # bypass .gitignore

- [[tool.hatch.build.hooks.build-scripts.scripts]]
- work_dir = "../../js"
+ [[tool.hatch.build.targets.wheel.hooks.build-scripts.scripts]]
+ work_dir = "js"

However it breaks installing from source (pip install src/py/reactpy) as pip skips building the sdist and therefore won't copy js/ to the correct place. I also don't know if there's an alternative to force-include which can map paths outside of the project root while respecting .gitignore. Absent that, it would be easy to accidentally pollute the sdist with node_modules.

In theory a symbolic link to src/js/ could remedy this along with slightly different changes to pyproject.toml, specifically the use of only-include:

+ [tool.hatch.build.targets.sdist]
+ only-include = ["js", "reactpy"]
+ 
+ [tool.hatch.build.targets.wheel.force-include]
+ "reactpy/_static" = "reactpy/_static"  # bypass .gitignore

- [[tool.hatch.build.hooks.build-scripts.scripts]]
- work_dir = "../../js"
+ [[tool.hatch.build.targets.wheel.hooks.build-scripts.scripts]]
+ work_dir = "js"

But while it works when generating the sdist / building from sdist (python -m build), the npm run build command fails when installing from source or building the wheel from source (hatch build/python -m build --wheel):

(.venv) ..\reactpy\src\py\reactpy>pip install .
Processing ..\reactpy\src\py\reactpy
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [74 lines of output]

      added 249 packages, and audited 258 packages in 5s

      89 packages are looking for funding
        run `npm fund` for details

      3 moderate severity vulnerabilities

      To address all issues, run:
        npm audit fix

      Run `npm audit` for details.

      > build
      > npm --workspaces run build


      > event-to-object@0.1.2 build
      > tsc -b


      > @reactpy/client@0.3.1 build
      > tsc -b

      src/reactpy-vdom.tsx(3,28): error TS2307: Cannot find module 'event-to-object' or its corresponding type declarations.
      npm ERR! Lifecycle script `build` failed with error:
      npm ERR! Error: command failed
      npm ERR!   in workspace: @reactpy/client@0.3.1
      npm ERR!   at location: ..\reactpy\src\py\reactpy\js\packages\@reactpy\client

      > build
      > vite build

      vite v3.2.7 building for production...
      transforming...
      ✓ 9 modules transformed.
      [vite]: Rollup failed to resolve import "event-to-object" from "../../../../js/app/node_modules/@reactpy/client/dist/reactpy-vdom.js".
      This is most likely unintended because it can break your application at runtime.
      If you do want to externalize this module explicitly add it to
      `build.rollupOptions.external`
      error during build:
      Error: [vite]: Rollup failed to resolve import "event-to-object" from "../../../../js/app/node_modules/@reactpy/client/dist/reactpy-vdom.js".
      This is most likely unintended because it can break your application at runtime.
      If you do want to externalize this module explicitly add it to
      `build.rollupOptions.external`
          at onRollupWarning (file:///../reactpy/src/js/node_modules/vite/dist/node/chunks/dep-2faf2534.js:45909:19)
          at onwarn (file:///../reactpy/src/js/node_modules/vite/dist/node/chunks/dep-2faf2534.js:45680:13)     
          at Object.onwarn (file:///../reactpy/src/js/node_modules/rollup/dist/es/shared/rollup.js:23263:13)    
          at ModuleLoader.handleResolveId (file:///../reactpy/src/js/node_modules/rollup/dist/es/shared/rollup.js:22158:26)
          at file:///../reactpy/src/js/node_modules/rollup/dist/es/shared/rollup.js:22119:26
      npm ERR! Lifecycle script `build` failed with error:
      npm ERR! Error: command failed
      npm ERR!   in workspace: undefined
      npm ERR!   at location: ..\reactpy\src\py\reactpy\js\app
      Traceback (most recent call last):
        File "..\reactpy\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "..\reactpy\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "..\reactpy\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "..\AppData\Local\Temp\pip-build-env-4ebt3ux8\overlay\Lib\site-packages\hatchling\build.py", line 58, in build_wheel
          return os.path.basename(next(builder.build(directory=wheel_directory, versions=['standard'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "..\AppData\Local\Temp\pip-build-env-4ebt3ux8\overlay\Lib\site-packages\hatchling\builders\plugin\interface.py", line 147, in build
          build_hook.initialize(version, build_data)
        File "..\AppData\Local\Temp\pip-build-env-4ebt3ux8\overlay\Lib\site-packages\hatch_build_scripts\plugin.py", line 51, in initialize
          run(cmd, cwd=str(work_dir), check=True, shell=True)  # noqa: S602
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "..\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 571, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command 'npm run build' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

FWIW, in an unrelated package, sardine-web, I structured it so the JS client's source code was inside the package itself, but setup.py would build the client and remove the source code afterwards, leaving only the source code in the sdist and its compiled result in the wheel. In hindsight, it probably would have been easier had I left the client source code outside of the package and only included it in the sdist via MANIFEST.in. Given hatch's own mechanisms for controlling distributions, plus the convenience of hatch-build-scripts, it might be easier to move src/js/ beside the src/py/reactpy/reactpy/ package and follow the previous suggestion to pyproject.toml, rather than try to find a workaround with the current layout.

[thegamecracks]

Given hatch's own mechanisms for controlling distributions, plus the convenience of hatch-build-scripts, it might be easier to move src/js/ beside the src/py/reactpy/reactpy/ package and follow the previous suggestion to pyproject.toml, rather than try to find a workaround with the current layout.

A word of caution, as of now this suggested layout would require js/node_modules/ to be deleted before creating a source distribution, as its presence can cause js/packages/ to be incorrectly omitted from the sdist (see pypa/hatch#1197 for more info). Turns out it wasn't easy...

[Archmonger]

@thegamecracks Would you be interested in PRing your suggested fix?

[thegamecracks]

Yup, #1183 implements the last suggestion along with a custom build script to work around the node_modules conflict.