Installing gadma on Macs with the M1 chip

[burbrink]

Hello,

I was wondering if anyone has successfully installed gadma on a mac with an M1 (or M1 Ultra) chip? I have had no luck even after creating an conda environment.

Right now it seems I am stuck with this error after executing gadma:

File "/Users/frankburbrink/miniconda3/envs/gadma_try/lib/python3.10/site-packages/moments/init.py", line 13, in
from timestamp import Timestamp
ImportError: cannot import name 'Timestamp' from 'timestamp' (unknown location)

Any help would be appreciated!

Thanks!

Frank

[noscode]

Dear Frank,

Sorry for the long reply. Have you fixed the problem? From your error message, it looks like the issue is for moments library.

Best regards,
Ekaterina

[burbrink]

Hi Ekaterina,

Thanks for you help. I am still having an issue on my M1. Any help would be appreciated.

It looks like moments 2.0 installed ok and the gadma install says it is ok:
Successfully installed gadma-2.0.0rc27.dev12

However, I am not getting the following when opening or running the test on gadma:

(GADMA_RosettaX) frankburbrink@Mac-Studio GADMA % gadma --test
Traceback (most recent call last):
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/bin/gadma", line 5, in
from gadma.core import main
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/init.py", line 93, in
from .data import DataHolder, SFSDataHolder, VCFDataHolder # NOQA
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/data/init.py", line 1, in
from .data import DataHolder, SFSDataHolder, VCFDataHolder # NOQA
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/data/data.py", line 1, in
from ..utils import check_file_existence, ensure_file_existence
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/utils/init.py", line 1, in
from .variables import Variable, ContinuousVariable, DiscreteVariable # NOQA
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/utils/variables.py", line 2, in
from .distributions import uniform_generator, trunc_normal_sigma_generator,
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/utils/distributions.py", line 2, in
from .utils import get_correct_dtype
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/utils/utils.py", line 12, in
import allel
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/allel/init.py", line 4, in
from .model.ndarray import *
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/allel/model/ndarray.py", line 18, in
from allel.opt.model import genotype_array_pack_diploid, genotype_array_unpack_diploid,
ImportError: dlopen(/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/allel/opt/model.cpython-38-darwin.so, 0x0002): tried: '/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/allel/opt/model.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/allel/opt/model.cpython-38-darwin.so' (no such file), '/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/allel/opt/model.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

[noscode]

Thank you for an update. Now it looks like the import of allele package is failing due to architecture that is not supported. Could you check that you will have almost the same output if you try:
python -m "import allele"

If so, I am not sure why allele package is not supported on M1. Maybe it is possible to install it from the source.
Do you need to read VCF file with GADMA? This library is required for that case, and if you do not need it I can tell you how to exclude it from the headers.

[burbrink]

So, there is "No module named import allele." However, I have scikit-allel 1.3.7 successfully installed in my conda environment.

I generally make the VCF files outside of gadma using the easysfs package, so it is important that I can read them into gadma.

Any other thoughts?

Thanks!

Frank

[noscode]

Oh, I am sorry I mixed up the flag, what is the output for: python -c "import allel"?

As you do not need VCF reading, you can do the following:

1. in file /Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/utils/utils.py comment out line 12 (add # at the beginning of the line, import allel -> # import allel)
2. in file /Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/utils/data_utils.py comment out line 3
3. in file /Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/engines/moments_ld_engine.py comment out line 19

Please let me know if it helped.

[burbrink]

I am still working on this and getting errors:
Traceback (most recent call last):
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/bin/gadma", line 5, in
from gadma.core import main
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/init.py", line 94, in
from .engines import get_engine, all_engines # NOQA
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/engines/init.py", line 6, in
from .moments_ld_engine import MomentsLdEngine, extract_rec_map_name_and_extension # NOQA
File "/Users/frankburbrink/mambaforge/envs/GADMA_RosettaX/lib/python3.8/site-packages/gadma/engines/moments_ld_engine.py", line 25
"""
#Function for reading data using multiprocessing
"""
^
IndentationError: expected an indented block

This works perfectly on my non-M1 chip.

[noscode]

It becomes more interesting and unusual. Unfortunately, I do not know what is going on. It looks like Python does not like formatting of the file gadma/engines/moments_ld_engine.py, but I checked: it looks fine for me. I tried to find similar problems for M1 chip, but I did not find much. I do not have M1 processor to try it myself.

One idea: maybe switch to different version of Python? GADMA should work for python3.9 and python3.10 as well.

[burbrink]

Hmmm...it doesn't seem like it will work with python3.9 or 3.10. See the errors (first is cloned and pip install ., second is a conda install):

error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [2 lines of output]
Python version 3.9 is not supported!
Supported versions are 3.6, 3.7, 3.8
[end of output]

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

• gadma -> python[version='>=3.6,<3.9']

Your python: python==3.9

[noscode]

Oh, this one is on me. Maybe it can be installed using pip?

I will fix this restriction next release, GADMA can be used for any Python version starting from 3.6.

[burbrink]

I created a conda environment with python==3.10 and did the gadma pip install but the test keeps failing. I can easily run this on my cluster and other sources, but I will be teaching a workshop on speciation processes and likely many of the students attending will be using mac laptops with the m1/m2 chips. I totally realize this isn't your problem and there are tons of threads with python package and m1/m2 issues.

Anyhow, your program has been a life saver for our research and allows us to explore model space in a really comprehensive way. Thanks so much!

I am pasting the latest error in case you insight:

frankburbrink@Mac-Studio build % gadma --test
Traceback (most recent call last):
File "/Users/frankburbrink/mambaforge/envs/gadma_try3/bin/gadma", line 5, in
from gadma.core import main
File "/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/gadma/init.py", line 93, in
from .data import DataHolder, SFSDataHolder, VCFDataHolder # NOQA
File "/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/gadma/data/init.py", line 1, in
from .data import DataHolder, SFSDataHolder, VCFDataHolder # NOQA
File "/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/gadma/data/data.py", line 1, in
from ..utils import check_file_existence, ensure_file_existence
File "/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/gadma/utils/init.py", line 1, in
from .variables import Variable, ContinuousVariable, DiscreteVariable # NOQA
File "/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/gadma/utils/variables.py", line 2, in
from .distributions import uniform_generator, trunc_normal_sigma_generator,
File "/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/gadma/utils/distributions.py", line 2, in
from .utils import get_correct_dtype
File "/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/gadma/utils/utils.py", line 12, in
import allel
File "/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/allel/init.py", line 4, in
from .model.ndarray import *
File "/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/allel/model/ndarray.py", line 18, in
from allel.opt.model import genotype_array_pack_diploid, genotype_array_unpack_diploid,
ImportError: dlopen(/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/allel/opt/model.cpython-310-darwin.so, 0x0002): tried: '/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/allel/opt/model.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/allel/opt/model.cpython-310-darwin.so' (no such file), '/Users/frankburbrink/mambaforge/envs/gadma_try3/lib/python3.10/site-packages/allel/opt/model.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

[noscode]

Thank you very much for your kind words and I am glad that you have included GADMA in the workshop for students, that sounds fantastic! I hope it went well. I will try to fix issues with M1/M2 chips as soon as I will be able to do it, I appreciate all your feedback.

Best regards,
Ekaterina

[Enricobazzi]

Hi, just started using gadma2 on my mac with M1 chip. I was able to get it running following these steps (the order matters I think as some libraries/packages need to be of a specific version):

conda create -n gadma2 python=3.10
conda activate gadma2

pip install git+https://github.com/MomentsLD/moments.git
conda install -c conda-forge dadi
conda install -c conda-forge scikit-allel
pip install gadma
pip uninstall ruamel.yaml
pip install "ruamel.yaml<0.18.0"
pip uninstall matplotlib
pip install "matplotlib<3.5"

Hope this helps!

Enrico

[noscode]

Wow, thank you very much Enrico!