Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swap to .spec files for PyInstaller #365

Merged
merged 20 commits into from
Sep 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/Build-and-deploy-mac.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ jobs:
- uses: conda-incubator/setup-miniconda@v2
with:
activate-environment: nwb-guide
#mamba-version: "*"
environment-file: environments/environment-MAC.yml
auto-activate-base: false

Expand Down
17 changes: 3 additions & 14 deletions .github/workflows/pyflask-build-and-dist-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,19 +30,16 @@ jobs:
os: ubuntu-latest
label: environments/environment-Linux.yml
prefix: /usr/share/miniconda3/envs/nwb-guide
shorthand: unix

- python-version: "3.10"
- python-version: "3.9"
Comment on lines -35 to +34
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be consistent with our environment files. Technically doesn't matter b/c the environment files override it, but still

os: macos-latest
label: environments/environment-Mac.yml
prefix: /Users/runner/miniconda3/envs/nwb-guide
shorthand: unix

- python-version: "3.10"
- python-version: "3.9"
os: windows-latest
label: environments/environment-Windows.yml
prefix: C:\Miniconda3\envs\nwb-guide
shorthand: win

steps:
- uses: actions/checkout@v3
Expand Down Expand Up @@ -88,15 +85,7 @@ jobs:
run: rm -f /Users/runner/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/sonpy/linux/sonpy.so

- name: Build PyFlask distribution
run: npm run build:flask:${{ matrix.shorthand }}

#- if: matrix.os == 'windows-latest'
# name: Run test on build executable
# run: node tests/testPyinstallerExecutable.js --script ./build/nwb-guide/nwb-guide.exe

#- if: matrix.os != 'windows-latest'
# name: Run test on build executable
# run: node tests/testPyinstallerExecutable.js --script ./build/nwb-guide/nwb-guide
run: npm run build:flask

- name: Run test on distributed executable
run: node tests/testPyinstallerExecutable.js
Expand Down
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,6 @@ semantic.json
build/
yarn.lock

*.spec

*.pyc
src/.DS_Store
.DS_Store
Expand Down
74 changes: 74 additions & 0 deletions nwb-guide.spec
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# -*- mode: python ; coding: utf-8 -*-
import sys
from pathlib import Path

sys.setrecursionlimit(sys.getrecursionlimit() * 5)

from PyInstaller.utils.hooks import collect_data_files
from PyInstaller.utils.hooks import collect_all

datas = [('./paths.config.json', '.'), ('./package.json', '.')]
binaries = []
hiddenimports = ['scipy._distributor_init', 'scipy._lib.messagestream', 'scipy._lib._ccallback', 'scipy._lib._testutils', 'email_validator']
datas += collect_data_files('jsonschema_specifications')
tmp_ret = collect_all('nwbinspector')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('neuroconv')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('pynwb')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('hdmf')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('ndx_dandi_icephys')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]
tmp_ret = collect_all('ci_info')
datas += tmp_ret[0]; binaries += tmp_ret[1]; hiddenimports += tmp_ret[2]


block_cipher = None


a = Analysis(
[f"{Path('pyflask') / 'app.py'}"],
pathex=[],
binaries=binaries,
datas=datas,
hiddenimports=hiddenimports,
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)

exe = EXE(
pyz,
a.scripts,
[],
exclude_binaries=True,
name='nwb-guide',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
console=True,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)
coll = COLLECT(
exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='nwb-guide',
)
13 changes: 7 additions & 6 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,14 @@
"dev:app": "vite src/renderer",
"dev:server": "cd pyflask && python -m flask run --port 4242",
"build": "npm run build:app",
"echo": "python -c \"print('hello')\"",
"build:app": "electron-vite build --outDir build",
"build:win": "npm run build && npm run build:flask:win && npm run build:electron:win",
"build:mac": "npm run build && npm run build:flask:unix && npm run build:electron:mac",
"build:linux": "npm run build && npm run build:flask:unix && npm run build:electron:linux",
"build:flask:base": "python -m PyInstaller --log-level DEBUG --name nwb-guide --onedir --clean --noconfirm ./pyflask/app.py --distpath ./build/flask --collect-data jsonschema_specifications --collect-all nwbinspector --collect-all neuroconv --collect-all pynwb --collect-all hdmf --collect-all ci_info --collect-all ndx_dandi_icephys --hidden-import scipy._distributor_init --hidden-import scipy._lib.messagestream --hidden-import scipy._lib._ccallback --hidden-import scipy._lib._testutils --hidden-import email_validator",
"build:flask:win": "npm run build:flask:base -- --add-data ./paths.config.json;. --add-data ./package.json;.",
"build:flask:unix": "npm run build:flask:base -- --add-data ./paths.config.json:. --add-data ./package.json:.",
"build:win": "npm run build && npm run build:flask && npm run build:electron:win",
"build:mac": "npm run build && npm run build:flask && npm run build:electron:mac",
"build:linux": "npm run build && npm run build:flask && npm run build:electron:linux",
"build:flask": "python -m PyInstaller nwb-guide.spec --log-level DEBUG --clean --noconfirm --distpath ./build/flask",
"build:flask:spec:base": "pyi-makespec --name nwb-guide --onedir --collect-data jsonschema_specifications --collect-all nwbinspector --collect-all neuroconv --collect-all pynwb --collect-all hdmf --collect-all ndx_dandi_icephys --collect-all ci_info --hidden-import scipy._distributor_init --hidden-import scipy._lib.messagestream --hidden-import scipy._lib._ccallback --hidden-import scipy._lib._testutils --hidden-import email_validator ./pyflask/app.py",
"build:flask:spec": "npm run build:flask:spec:base && python prepare_pyinstaller_spec.py",
"build:electron:win": "electron-builder build --win --publish never",
"build:electron:mac": "electron-builder build --mac --publish never",
"build:electron:linux": "electron-builder build --linux --publish never",
Expand Down
30 changes: 30 additions & 0 deletions prepare_pyinstaller_spec.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
"""
Calling `pyi-makespec` regenerates the base spec file, but we need to extend the recursion limit.

This script is run automatically as a part of `npm run build:flask:spec` after the `pyi-makespec` call.
"""
from pathlib import Path

with open(file=Path(__file__).parent / "nwb-guide.spec", mode="r") as io:
lines = io.readlines()

lines.insert(1, "import sys\n")
lines.insert(2, "from pathlib import Path\n")
lines.insert(3, "\n")
lines.insert(4, "sys.setrecursionlimit(sys.getrecursionlimit() * 5)\n")
lines.insert(5, "\n")

# Originally this was a separate `npm` command per platform to account for CLI syntax differences between ; and :
# The spec file is, however, the same across platforms
data_line_index = lines.index("datas = []\n")
lines[data_line_index] = "datas = [('./paths.config.json', '.'), ('./package.json', '.')]\n"

# Another platform specific difference is the app.py location
app_py_line_index = next(index for index, line in enumerate(lines) if "app.py" in line)
app_py_line = " [f\"{Path('pyflask') / 'app.py'}\"],\n"
lines[app_py_line_index] = app_py_line

with open(file=Path(__file__).parent / "nwb-guide.spec", mode="w") as io:
io.writelines(lines)

print("Sucessfully injected recursion depth extension and json paths!")