Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve documentation installation info #4419

Closed
choldgraf opened this issue Jul 24, 2017 · 42 comments
Closed

Improve documentation installation info #4419

choldgraf opened this issue Jul 24, 2017 · 42 comments
Milestone
0.16

Comments

@choldgraf
Copy link
Contributor

A few things I notice:

  • Do we really need to recommend python 2.7 to new users? IMO it makes MNE seem behind-the-times. Mayavi does technically work on python 3 now, though it's a pain to install IME.
  • The installation instructions are very text heavy. We should streamline.
  • Do we really need to explicitly call out matlab in the instructions?
  • I think we should have a separate page for installing the development version. If we have newbies coming to this page we don't want them seeing python setup.py develop IMO.
  • We should figure out who this set of instructions doesn't work for so that we can improve them.
@choldgraf
Copy link
Contributor Author

@kingjr @Eric89GXL

@larsoner larsoner added this to the 0.15 milestone Jul 24, 2017
@larsoner
Copy link
Member

I don't see a straightforward way to use Mayavi on Py3k until it works out of the box (conda-forge can also break things) so I'd rather wait until then to recommend 3+.

Otherwise I agree, do you have any time to try?

@choldgraf
Copy link
Contributor Author

I can make a PR with these improvements though let's get a little list together here we agree on first. Anything else pop out at you? @kingjr any specific feedback from people on things that didn't work?

@kingjr
Copy link
Member

kingjr commented Jul 24, 2017

Good. Perhaps clarifying the backends and environment variables in there.

@larsoner
Copy link
Member

@choldgraf any time to tackle this in the next week or so? Would be good for release.

@kingjr on Gitter mentioned adding a section for Mayavi tricks, which would be good (@kingjr feel free to add links to suggestions if you have them, otherwise I can try to dig some up)

@dengemann
Copy link
Member

dengemann commented Aug 16, 2017 via email

@dengemann
Copy link
Member

dengemann commented Aug 16, 2017 via email

@kingjr
Copy link
Member

kingjr commented Aug 16, 2017

@dengemann woaw; that's actually surprising to me ;); I regularly still run into a bunch of incompatibility issues between mayavi, xgboost, tensorflow etc for very unclear reason.

My only tips for now is

  • if conda install mayavi creates an incompatibility error, pip install mayavi doesn't
  • if, on ubuntu, mayavi freezes, install pythonwx, and launch ipython --pylab=wx (way better than qt in my experience)

@kingjr
Copy link
Member

kingjr commented Aug 16, 2017 •
edited
Loading

also, overall, I'm would be against env variables as much as possible, and show in examples/tutorials etc to add variables via os.env, such that we work in pure python

@dengemann
Copy link
Member

dengemann commented Aug 16, 2017 via email

@choldgraf
Copy link
Contributor Author

I can spend some time on this. Can people add to the list in the top-level comment anything they'd like to see included?

@agramfort
Copy link
Member

agramfort commented Aug 17, 2017 via email

@dengemann
Copy link
Member

dengemann commented Aug 17, 2017 via email

@agramfort
Copy link
Member

agramfort commented Aug 17, 2017 via email

@choldgraf
Copy link
Contributor Author

what is the main thing blocking us from using something like pythreejs or ipyvolume as a backend? The main drawback I see is that these are pure web packages, so it'll be clunkier to get them working in an old-fashioned (see what I did there @agramfort ;-) ) python workflow. For those who don't mind working within a browser context, it should be a great option for plotting! Maybe at least worth exploring the idea

@agramfort
Copy link
Member

agramfort commented Aug 19, 2017 via email

@larsoner
Copy link
Member

Removing the 0.15 milestone. @choldgraf if you think you can make the most critical changes in the next few days / week please do.

I think this will be facilitated greatly by #4523 once we get it done

@larsoner larsoner modified the milestones: 0.15, 0.16 Sep 14, 2017
@choldgraf
Copy link
Contributor Author

it looks like #4523 is closed but not merged...can you ping me whenever this gets fixed up so I can make changes?

I can put some time into this next week if things are in place for me to do it...let me know. Worst comes to worst we can always backport instructions

@larsoner
Copy link
Member

larsoner commented Sep 16, 2017 via email

@choldgraf
Copy link
Contributor Author

choldgraf commented Sep 16, 2017 •
edited
Loading

Lemme take a stab at it this week, when is .15 being released?

@kingjr
Copy link
Member

kingjr commented Sep 27, 2017

IMO, we need to fix/complement the install documentation before the release.

1. Mayavi:

On fresh mac os (El Capitain) install + anaconda with python 2.7, mayavi/pysurfer fail.
We seem to need to do conda remove navigator-updater; pip install vtk mayavi PySurfer.

On Ubuntu (16.04), (and, IIRC after matplotlib upgrade), I have repeatedly needed to do:
conda remove mayavi; install mayavi --upgrade

(and no, the maptloblib backend is not a good solution, as it can be mega slow)

2. environment variables, freesurfer and matlab dependencies

Ideally, we should have a script that automatically detects or setup the environment variables and paths related to Freesurfer, MNE C and Matlab. (Most users don't know what .bash_profile is, don't code in shell, and don't know the difference between sh and csh.)

We could have a pure python function that helps users set-up their environment:

def setup_paths(FREESURFER_DIR=None, MATLAB_ROOT=None, MNE_ROOT=None):
    if FREESURFER_DIR is None:
         if not 'FREESURFER_DIR' in os.environ:
             os.environ['FREESURFER_DIR'] = subprocess('which', 'recon-all')
...

In my experience, these two points are total blockers for newbies. Most I meet come from Fieldtrip where they just need to add the paths through the Matlab menu, and everything else works.

@larsoner
Copy link
Member

we need to fix/complement the install documentation before the release.

I agree it could be better. Someone needs to find the time to really beef it up. I won't have it because I'm working on too many other bugs and spread too thin currently. Personally I'm hoping #4570 and eventually #4523 will take care of it, because then the installation instructions can be:

conda env create -f environment.yml

... and this env will be tested on Linux, OSX, and Windows CIs. But maybe this won't happen soon enough :(

On fresh mac os (El Capitain) install + anaconda with python 2.7, mayavi/pysurfer fail.

Our CIs run OSX w/PySurfer. Can you clarify the problem?

On Ubuntu (16.04), (and, IIRC after matplotlib upgrade), I have repeatedly needed to do:

Anaconda unfortunately does not resolve dependencies properly all the time. If you want to upgrade a package, you're best off doing something like:

conda install numpy scipy matplotlib mayavi

To force it to look at all of those dependencies. If you just do:

conda install matplotlib

... it will often (always?) just install the latest matplotlib and pyqt5 instead of pyqt4, which will break some of Mayavi's dependencies. I wonder if there is an Anaconda bug about this (if there isn't, we should add one).

Ideally, we should have a script that automatically detects or setup the environment variables and paths related to Freesurfer, MNE C and Matlab.

The calls MNE-Python makes to MNE-C have been eliminated, and the ones to FreeSurfer are pretty limited. So I assume you're not talking about those? If you source these scripts in the Python environment and then exit, then the env vars don't persist AFAIK. So I assume your goal is to get the env vars set up in Python so you can do subprocess.Popen calls to e.g. set up your FreeSurfer reconstructions? If so it sounds more like the goal is to make a freesurfer-Python sort of package, so it seems a bit out of scope.

@choldgraf
Copy link
Contributor Author

choldgraf commented Sep 28, 2017 via email •
edited
Loading

@choldgraf
Copy link
Contributor Author

Also, can we come up with some actionable ways in which the documentation for installation should be improved? It seems like there is disagreement about how best to do this / what information needs to be in there. I am happy to give a shot at a PR, but it is hard to know what we should say as the "best" approach. (e.g. I still haven't really gotten mayavi working reliably on my system so am hesitant to be the one to recommend something that works all the time)

@mne-tools mne-tools deleted a comment from choldgraf Sep 28, 2017
@kingjr
Copy link
Member

kingjr commented Sep 28, 2017

Our CIs run OSX w/PySurfer. Can you clarify the problem?

installing pysurfer / mayavi through pip or mayavi crashes.

To force it to look at all of those dependencies. If you just do: conda install matplotlib

In my experience, this does not solve the dependencies issue. I had to manually remove navigator-updater else it wouldn't work. This happened on multiple platforms, including mac OS X and ubuntu with fresh anaconda installs.

The calls MNE-Python makes to MNE-C have been eliminated, and the ones to FreeSurfer are pretty limited. So I assume you're not talking about those?

Don't you need those for making the bems (make_watershed_bem) and the surface?

If you source these scripts in the Python environment and then exit, then the env vars don't persist AFAIK.

That's ok IMO, as long as the script is standalone and explicit (i.e. does not depend on preexisting shell variables).

So I assume your goal is to get the env vars set up in Python so you can do subprocess.Popen calls to e.g. set up your FreeSurfer reconstructions? If so it sounds more like the goal is to make a freesurfer-Python sort of package, so it seems a bit out of scope.

For the studies I have done, I systematically had to run the following anatomical pipeline in python (but scalp_surface) after Freesurfer's recon-all:

make_watershed_bem
mne.commands.mne_make_scalp_surfaces
setup_source_space
make_bem_model
make_bem_solution
coregistration

This required having freesurfer and MNE C setup properly, and thus repeatedly blocked students. But I guess there is a better and simpler way to do this?

@larsoner
Copy link
Member

None of those commands require MNE-C anymore. You are right that make_watershed_bem and mne_make_scalp_surfaces require FreeSurfer, but the others do not. So I think we're just talking about making these last two easier to work with in some way.

These two functions are actually just thin wrappers around FreeSurfer calls. (I think they're mostly provided for historical purposes?) If you are already doing recon-all, if you can add a mri_watershed and a mkheadsurf call to that same script, then you won't need FreeSurfer at all once you're working in Python. That seems like a better option to me, do you agree? That would make it a documentation problem, rather than trying to hack together Python calling of FreeSurfer functions (which, from PySurfer and MNE-C call-from-Python experiences, I know is non-trivial and can create library conflicts.)

@kingjr
Copy link
Member

kingjr commented Sep 28, 2017 •
edited
Loading

Ok; I agree with just adding recon-all mri_watershed and mkheadsurfto the documentation. The paths / env variables etc wouldn't be an issue anymore. Thanks

@choldgraf
Copy link
Contributor Author

choldgraf commented Sep 28, 2017 •
edited
Loading

another plug here to make documentation change suggestions as actionable and straightforward as possible, unless y'all wanna open the PR ;-)

@kingjr
Copy link
Member

kingjr commented Sep 28, 2017

@choldgraf you're right. IMO, two PRs would be useful:

1. Clarifiy dependencies

  • add a quick tutorial in Documentation > Source-level analysis > prepare anatomical data and/or in the cookbook, that includes a shell script including recon-all, mri_watershed and mkheadsurf for a given subject.
  • perhaps explicit in the install doc that MNE C should not be necessary for any processing
  • add in the installation doc that freesurfer is necessary for preprocessing the anatomical MRI, and that MNE doesn't do any segmentation/BEM etc

2. troubleshooting doc for mayavi

For Mayavi / PySurfer related issues, we should probably start a troubleshooting section that starts to list the issues with mayavi and their fixes. We could backport this documentation along the way, such that it's not a blocker for the release.

@larsoner
Copy link
Member

I can do (1) with @kingjr as a tester once the dust settles on the release. I won't have time before.

For (2) we could tell people to use our current environment.yml (#4570) with ETS_TOOLKIT=wx env var. It should work. But we should really test it for some months before advertising it.

@larsoner
Copy link
Member

Okay so over in #4626 I've confirmed that Linux/OSX/Windows can pass tests with the environment.yml. Locally I need to do ETS_TOOLKIT=wx before running any interactive Mayavi stuff to prevent conflicts with matplotlib. However, there is a way to automatically have the environment (when doing source activate mne) set env vars. I'm using it locally and it works. After 0.15 lands, we could modify the install page to recommend this procedure, and we could all try it.

Or we can wait for the 3.6 + PyQt5 environment to stabilize (@wmvanvliet has had good luck so far trying it today), which would be simpler and nicer. But I don't know how long it will take. In the meantime if others want to try it (or the 2.7 env), please do.

@kingjr
Copy link
Member

kingjr commented Sep 29, 2017 via email

@larsoner
Copy link
Member

I'm not sure what's the best decision with regard to waiting or not for pyqt5

The advantage is that you can use matplotlib 2+ which has some nice enhancements. Also numpy/scipy/scikit-learn etc. are all more up to date in the 3.6 environment because they aren't held back by old mayavi/vtk etc.

Here is some pseudo-documentation to get people off the ground if they want to try either environment. They can be tested with python -i tutorials/plot_mne_dspm_source_localization.py (or any other script that uses both matplotlib and mayavi.

Python 3 (experimental)

Use this file: https://raw.githubusercontent.com/agramfort/mne-python/6dfdfe805380c02813b6dad95d9b7e054215c465/environment.yml

conda env create -f environment3.yml -n mne3
source activate mne3

Python 2 (CIs use it)

Use this file: https://raw.githubusercontent.com/mne-tools/mne-python/57fe40c98a97db7d1be9143b859ff86d89e1a7d8/environment.yml

conda env create -f environment.yml -n mne2
pushd ~/anaconda2/envs/mne2/
mkdir ./etc/conda
mkdir ./etc/conda/activate.d
mkdir ./etc/conda/deactivate.d
printf '#!/bin/sh\n\nexport ETS_TOOLKIT=wx\n' > ./etc/conda/activate.d/env_vars.sh
printf '#!/bin/sh\n\nexport unset ETS_TOOLKIT\n' > ./etc/conda/deactivate.d/env_vars.sh
sed -i "s/ENABLE_USER_SITE = .*/ENABLE_USER_SITE = False/g" ./lib/python2.7/site.py
popd
source activate mne2

@choldgraf
Copy link
Contributor Author

Just a thought: can we make mayavi a soft dependency? As in, it doesn't get installed by default and throws an error if you try to do 3D viz without having it installed? I'm worried that, especially for beginners, it becomes a big blocker to any MNE functionality, even though there's tons of stuff you can do with MNE that has nothing to do with 3D viz...

@larsoner
Copy link
Member

larsoner commented Oct 3, 2017

Just a thought: can we make mayavi a soft dependency? As in, it doesn't get installed by default and throws an error if you try to do 3D viz without having it installed?

You mean in the environment.yml that we're trying to build? The only reason we really need it is to get Mayavi working. From what I've seen Mayavi causes 99% of the problems with Anaconda for us. If you don't want Mayavi, everything pretty much works with conda install numpy scipy matplotlib ... so I'm not sure we even need the environment.yml at that point (even the default Anaconda distribution is probably good enough!).

@choldgraf
Copy link
Contributor Author

gotcha - maybe this is just a documentation thing. Maybe just worth highlighting that mayavi is not required and may cause headaches if it is used. I don't like that we still have to recommend people use python 2 over python 3 at the very beginning of install instructions :-/

@kingjr
Copy link
Member

kingjr commented Oct 3, 2017 via email

@larsoner
Copy link
Member

larsoner commented Oct 4, 2017

Okay over on #4641 I think we've finally arrived at an environment that works. 3.6, latest numpy/scipy/matplotlib/mayavi, and CIs pass on Linux, OSX, and Windows. I propose that we add this to the doc, tagged as exerimental for 0.15. Then lots of us (and CIs) test it for a while and if it's all good, use it as the default instructions for 0.16. Thoughts?

@choldgraf
Copy link
Contributor Author

+1

@wmvanvliet
Copy link
Contributor

This has been working better for me than my previous mess with python2, so yes, +1!

@larsoner
Copy link
Member

larsoner commented Oct 4, 2017

Updated doc

https://6073-1301584-gh.circle-artifacts.com/0/tmp/circle-artifacts.2fIiuDA/html/install_mne_python.html

@larsoner larsoner mentioned this issue Oct 4, 2017
Merged
@choldgraf
Copy link
Contributor Author

choldgraf commented Oct 4, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.16
Development

No branches or pull requests
6 participants
@agramfort @wmvanvliet @choldgraf @dengemann @larsoner @kingjr