Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trying to fix Mac distributable #353

Merged
merged 26 commits into from
Sep 13, 2023
Merged

Trying to fix Mac distributable #353

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8f9f9f7
Testing
rly Sep 9, 2023
f49184d
Set pyinstaller log level to debug
rly Sep 9, 2023
7005c4b
Add sonpy.so explicitly
rly Sep 9, 2023
5ead011
Testing
rly Sep 9, 2023
0d346cf
Testing
rly Sep 9, 2023
53e62b2
Testing
rly Sep 9, 2023
e55b823
Testing
rly Sep 9, 2023
499dc92
Fix
rly Sep 9, 2023
074c2a6
Fix
rly Sep 9, 2023
e29e43e
Update arm64 env
rly Sep 9, 2023
d1be94a
Adjust all envs
rly Sep 9, 2023
18c363c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2023
0760fc4
Fix windows?
rly Sep 9, 2023
cd9e512
Fix mac build from cache
rly Sep 9, 2023
7f1d4d6
reset windows
rly Sep 9, 2023
0a5d9d0
reset windows
rly Sep 9, 2023
92e1f46
Fix missing import
rly Sep 9, 2023
31ac2dc
Merge branch 'main' into debug_mac_dist
rly Sep 12, 2023
9b84832
try neuroconv specific debug
CodyCBakerPhD Sep 12, 2023
c3658e7
Update pyflask-build-and-dist-tests.yml
CodyCBakerPhD Sep 12, 2023
e3d9eff
Update Build-and-deploy-mac.yml
CodyCBakerPhD Sep 12, 2023
97b1caa
Update pyflask-build-and-dist-tests.yml
CodyCBakerPhD Sep 12, 2023
97b8d8f
Update Build-and-deploy-mac.yml
CodyCBakerPhD Sep 12, 2023
942edbd
update to main post merge
CodyCBakerPhD Sep 12, 2023
ddcbc4b
Update package.json
CodyCBakerPhD Sep 12, 2023
82451bc
Merge pull request #358 from NeurodataWithoutBorders/more_debugs_m1
CodyCBakerPhD Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/Build-and-deploy-mac.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ jobs:
- name: Install package.json modules and their dependencies
run: npm install --ignore-scripts

- name: Remove bad sonpy file (might make Spike2 format unusable on Mac - should exclude from selection)
run: rm -f /usr/local/miniconda/envs/nwb-guide/lib/python3.9/site-packages/sonpy/linux/sonpy.so

- uses: apple-actions/import-codesign-certs@v1
with:
# https://developer.apple.com/account/resources/certificates/add
Expand All @@ -56,5 +59,4 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
appleId: ${{ secrets.APPLE_ID }} # currently this is set to Ryan's Apple ID and password
appleIdPassword: ${{ secrets.APPLE_PASSWORD }}
run: |
npm run deploy:mac
run: npm run deploy:mac
4 changes: 4 additions & 0 deletions .github/workflows/pyflask-build-and-dist-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,10 @@ jobs:

- run: npm ci

# fix for macos build
- if: matrix.os == 'macos-latest'
run: rm -f /Users/runner/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/sonpy/linux/sonpy.so

CodyCBakerPhD marked this conversation as resolved.
Show resolved Hide resolved
- name: Build PyFlask distribution
run: npm run build:flask:${{ matrix.shorthand }}

Expand Down
37 changes: 14 additions & 23 deletions environments/environment-Linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,31 +1,22 @@
name: nwb-guide
channels:
- defaults
- conda-forge
- defaults
dependencies:
- python = 3.9.17
- PyInstaller = 4.7
- nodejs = 16.14.2
- libgcc = 7.2.0
- git = 2.20.1
- python = 3.9.18
- PyInstaller = 5.13.0
- nodejs = 18.16.1
- numcodecs = 0.11.0
# install these from conda-forge so that dependent packages get included in the distributable
- jsonschema = 4.18.0 # installs jsonschema-specifications
- pydantic[email] = 2.0.2 # installs email-validator
- pip
- pip:
- chardet == 4.0.0
- pandas == 1.4.2
- openpyxl == 3.0.3
- pennsieve == 6.1.2
- configparser == 4.0.2
- python-docx == 0.8.10
- xlrd == 1.2.0
- biopython == 1.79
- flask == 2.0.2
- flask_restx
- protobuf == 3.20.0
- gevent == 21.12.0
- werkzeug == 2.0.2
- opencv-python == 4.5.3.56
- chardet == 5.1.0
- configparser == 6.0.0
- flask == 2.3.2
- flask-cors == 4.0.0
- flask_restx == 1.1.0
- neuroconv @ git+https://github.com/catalystneuro/neuroconv.git@main#neuroconv[full]
- hdmf >= 3.7.0
- pytest == 7.2.2
- pytest == 7.4.0
- pytest-cov == 4.1.0
- flask-cors === 3.0.10
12 changes: 7 additions & 5 deletions environments/environment-MAC-arm64.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,23 @@ channels:
- conda-forge
- defaults
dependencies:
- python = 3.9.17
- python = 3.9.18
- PyInstaller = 5.13.0
- nodejs = 18.16.1
- numcodecs = 0.11.0
- lxml = 4.9.3 # pypi build fails due to x64/arm64 mismatch so install from conda-forge
- pyedflib = 0.1.32 # pypi build fails due to x64/arm64 mismatch so install from conda-forge
- jsonschema = 4.18.0 # this installs jsonschema-specifications which is needed for validation
- pydantic[email] = 2.0.2 # email validator is used by dandi
- numpy # may have x64/arm64 mismatch issues so install from conda-forge
# install these from conda-forge so that dependent packages get included in the distributable
- jsonschema = 4.18.0 # installs jsonschema-specifications
- pydantic[email] = 2.0.2 # installs email-validator
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was wondering if the conda-forge installation pathway would resolve a lot of our challenges with this

Thanks for trying that out

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an alternative to pip install email_validator which is used in the win/linux envs. I don't know which one is better, but they both seem to work. Do you have a preference? We should probably be consistent with the envs as much as we can.

Copy link
Collaborator

@CodyCBakerPhD CodyCBakerPhD Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that conda-forge is more reliable when there are many nested (as in hidden imports) or non-Python (as in bindings) assets attached to the package

Also architecture specific releases - PyPI does a lot of that magically and you just cross your fingers but as you know, it's something you specifically setup in the feedstock (with or without certain amount of developer pains)

Though PyPI can host non-pure Python, in experience it tends to work best for pure Python code

So I'm fine with pydantic coming from conda-forge, yes. Also v2.x of Pydantic is partially written in Rust whereas v1.x is pure Python so setting up for conda-forge might be best here

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great points

- pip
- pip:
- chardet == 5.1.0
- configparser == 6.0.0
- flask == 2.3.2
- flask_restx == 1.1.0
- flask-cors == 4.0.0
- neuroconv @ git+https://github.com/catalystneuro/neuroconv.git@main#neuroconv[ecephys]
- flask_restx == 1.1.0
- neuroconv @ git+https://github.com/catalystneuro/neuroconv.git@main#neuroconv[full]
- pytest == 7.4.0
- pytest-cov == 4.1.0
16 changes: 8 additions & 8 deletions environments/environment-MAC.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
name: nwb-guide
channels:
- defaults
- conda-forge
- defaults
dependencies:
- python = 3.9.17
- nodejs = 18.16.1
- python = 3.9.18
- PyInstaller = 5.13.0
- nodejs = 18.16.1
- numcodecs = 0.11.0
- jsonschema = 4.18.0 # this installs jsonschema-specifications which is needed for validation
- pydantic[email] = 2.0.2 # email validator is used by dandi
# install these from conda-forge so that dependent packages get included in the distributable
- jsonschema = 4.18.0 # installs jsonschema-specifications
- pydantic[email] = 2.0.2 # installs email-validator
- pip
- pip:
- chardet == 5.1.0
- configparser == 6.0.0
- flask == 2.3.2
- flask-cors === 3.0.10
- flask-cors == 4.0.0
- flask_restx == 1.1.0
- neuroconv @ git+https://github.com/catalystneuro/neuroconv.git@main#neuroconv[full]
- hdmf >= 3.7.0
- pytest == 7.2.2
- pytest == 7.4.0
- pytest-cov == 4.1.0
6 changes: 3 additions & 3 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "nwb-guide",
"productName": "NWB GUIDE",
"version": "0.0.4",
"version": "0.0.5",
"description": "",
"main": "./build/main/main.js",
"scripts": {
Expand All @@ -15,9 +15,9 @@
"build:win": "npm run build && npm run build:flask:win && npm run build:electron:win",
"build:mac": "npm run build && npm run build:flask:unix && npm run build:electron:mac",
"build:linux": "npm run build && npm run build:flask:unix && npm run build:electron:linux",
"build:flask:base": "python -m PyInstaller --name nwb-guide --onedir --clean --noconfirm ./pyflask/app.py --distpath ./build/flask --collect-data jsonschema_specifications --collect-all nwbinspector --collect-all neuroconv --collect-all pynwb --collect-all hdmf --collect-all ci_info --collect-all ndx_dandi_icephys --hidden-import scipy._distributor_init --hidden-import scipy._lib.messagestream --hidden-import scipy._lib._ccallback --hidden-import scipy._lib._testutils --hidden-import email_validator",
"build:flask:base": "python -m PyInstaller --log-level DEBUG --name nwb-guide --onedir --clean --noconfirm ./pyflask/app.py --distpath ./build/flask --collect-data jsonschema_specifications --collect-all nwbinspector --collect-all neuroconv --collect-all pynwb --collect-all hdmf --collect-all ci_info --collect-all ndx_dandi_icephys --hidden-import scipy._distributor_init --hidden-import scipy._lib.messagestream --hidden-import scipy._lib._ccallback --hidden-import scipy._lib._testutils --hidden-import email_validator",
CodyCBakerPhD marked this conversation as resolved.
Show resolved Hide resolved
"build:flask:win": "npm run build:flask:base -- --add-data ./paths.config.json;. --add-data ./package.json;.",
"build:flask:unix": "npm run build:flask:base -- --add-data ./paths.config.json:. --add-data ./package.json:. --collect-all ndx_dandi_icephys",
"build:flask:unix": "npm run build:flask:base -- --add-data ./paths.config.json:. --add-data ./package.json:.",
"build:electron:win": "electron-builder build --win --publish never",
"build:electron:mac": "electron-builder build --mac --publish never",
"build:electron:linux": "electron-builder build --linux --publish never",
Expand Down
Loading