pyimagej image.init() FileNotFoundError: [Errno 2] No such file or directory: ~.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar

[benwilks]

Hi,
I'm having trouble running pyimagej image.init() installed with conda/mamba in a docker container.

mamba create -n pyimagej -c conda-forge python=3.8 pyimagej openjdk=11 ipykernel
wget https://downloads.imagej.net/fiji/latest/fiji-linux64.zip 
unzip fiji-linux64.zip 
rm fiji-linux64.zip

Importing imagej works but when i run:
ij = imagej.init(os.path.join(base_dir, 'Fiji.app'), mode='headless')

I get the following error related to

FileNotFoundError: No such file or directory: ~/.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar' -> ~/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/scifio-labeling-0.3.1.jar

---

Local path to ImageJ2 installation given: /home/jovyan/code/image-analysis/imagej/Fiji.app
Added 507 JARs to the Java classpath.
Adding jars from endpoints ['io.scif:scifio-labeling:0.3.1', 'net.imglib2:imglib2-imglyb:1.1.0']
Using endpoints ['io.scif:scifio-labeling:0.3.1', 'net.imglib2:imglib2-imglyb:1.1.0']
Getting verbose level: 2
Getting verbose level: 2
Returning expanded coordinate io.scif:scifio-labeling:0.3.1.
Returning expanded coordinate net.imglib2:imglib2-imglyb:1.1.0.
First time start-up may be slow. Downloaded dependencies will be cached for shorter start-up times in subsequent executions.
Executing: ('/opt/conda/envs/pyimagej/bin/mvn', '-B', '-f', '/home/jovyan/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/pom.xml', 'dependency:resolve', '-X')
Relevant mvn output: [INFO] io.scif:scifio-labeling:jar:0.3.1:compile -- module io.scif.labeling [auto]
Linking source /home/jovyan/.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar to target /home/jovyan/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/scifio-labeling-0.3.1.jar with link_type auto
Linking source /home/jovyan/.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar to target /home/jovyan/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/scifio-labeling-0.3.1.jar with link_type hard

---

FileNotFoundError Traceback (most recent call last)
Cell In[7], line 1
----> 1 ij = imagej.init(os.path.join(base_dir, 'Fiji.app'), mode='headless')

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/imagej/init.py:1200, in init(ij_dir_or_version_or_endpoint, mode, add_legacy, headless)
1197 raise EnvironmentError("Sorry, the interactive mode is not available on macOS.")
1199 if not sj.jvm_started():
-> 1200 success = _create_jvm(ij_dir_or_version_or_endpoint, mode, add_legacy)
1201 if not success:
1202 raise RuntimeError("Failed to create a JVM with the requested environment.")

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/imagej/init.py:1424, in _create_jvm(ij_dir_or_version_or_endpoint, mode, add_legacy)
1421 sj.config.endpoints.extend(original_endpoints)
1423 try:
-> 1424 sj.start_jvm()
1425 except subprocess.CalledProcessError as e:
1426 # Check to see if initialization failed due to "un-managed"
1427 # imagej-legacy
1428 err_lines = []

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/scyjava/_java.py:190, in start_jvm(options)
188 endpoints = endpoints[:1] + sorted(endpoints[1:])
189 _logger.debug("Using endpoints %s", endpoints)
--> 190 _, workspace = jgo.resolve_dependencies(
191 "+".join(endpoints),
192 m2_repo=scyjava.config.get_m2_repo(),
193 cache_dir=scyjava.config.get_cache_dir(),
194 manage_dependencies=scyjava.config.get_manage_deps(),
195 repositories=repositories,
196 verbose=scyjava.config.get_verbose(),
197 shortcuts=scyjava.config.get_shortcuts(),
198 )
199 jpype.addClassPath(os.path.join(workspace, "*"))
201 # HACK: Try to set JAVA_HOME if it isn't already.

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/jgo/jgo.py:712, in resolve_dependencies(endpoint_string, cache_dir, m2_repo, link_type, update_cache, force_update, manage_dependencies, repositories, shortcuts, verbose)
709 relevant_jars.append(jar_file_in_workspace)
711 try:
--> 712 link(
713 os.path.join(m2_repo, *g.split("."), a, version, jar_file),
714 jar_file_in_workspace,
715 link_type=link_type,
716 )
717 except FileExistsError:
718 # Do not throw exception if target file exists.
719 pass

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/jgo/jgo.py:222, in link(source, link_name, link_type)
220 except OSError as e:
221 if e.errno != 18:
--> 222 raise e
223 try:
224 return link(source=source, link_name=link_name, link_type="soft")

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/jgo/jgo.py:219, in link(source, link_name, link_type)
217 elif link_type.lower() == "auto":
218 try:
--> 219 return link(source=source, link_name=link_name, link_type="hard")
220 except OSError as e:
221 if e.errno != 18:

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/jgo/jgo.py:214, in link(source, link_name, link_type)
212 return os.symlink(source, link_name)
213 elif link_type.lower() == "hard":
--> 214 return os.link(source, link_name)
215 elif link_type.lower() == "copy":
216 return shutil.copyfile(source, link_name)

FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar' -> '/home/jovyan/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/scifio-labeling-0.3.1.jar'

Running ImageJ doctor doesn't show any problems
import imagej.doctor imagej.doctor.checkup()

Checking Python:

--> Python executable = /opt/conda/envs/pyimagej/bin/python

Checking environment:
--> CONDA_PREFIX = /opt/conda/envs/pyimagej
--> Python executable matches Conda environment.

Checking Python dependencies:
--> jgo: /opt/conda/envs/pyimagej/lib/python3.8/site-packages/jgo/init.py
--> scyjava: /opt/conda/envs/pyimagej/lib/python3.8/site-packages/scyjava/init.py
--> imglyb: /opt/conda/envs/pyimagej/lib/python3.8/site-packages/imglyb/init.py
--> pyimagej: /opt/conda/envs/pyimagej/lib/python3.8/site-packages/imagej/init.py

Checking Maven:
--> Maven executable = /opt/conda/envs/pyimagej/bin/mvn
$ mvn -v
Apache Maven 3.9.3 (246d85a6ebec21fd45a0f9abd2a99906f92a4707)
Maven home: /opt/conda/envs/pyimagej/opt/maven
Java version: 11.0.15-internal, vendor: Oracle Corporation, runtime: /opt/conda/envs/pyimagej/lib/jvm
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.1.36-unraid", arch: "amd64", family: "unix"

Checking Java:
--> JAVA_HOME = /opt/conda/envs/pyimagej/lib/jvm
--> Java executable = /opt/conda/envs/pyimagej/bin/java
$ java -version
openjdk version "11.0.15-internal" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15-internal+0-adhoc..src)
OpenJDK 64-Bit Server VM (build 11.0.15-internal+0-adhoc..src, mixed mode)

Great job! All looks good.

[ctrueden]

@benwilks Sorry for the slow reply. Something went sideways trying to download the scifio-labeling library from Maven, and then when PyImageJ (via jgo) tried to include it in the environment, it didn't work because the scifio-labeling-0.3.1.jar was not in the expected location of ~/.m2/repository. I wonder if this is because your Maven installation (which looks like it came from conda-forge) downloaded the library elsewhere without jgo understanding that.

With your pyimagej environment active, you can try to run:

$ mvn -X dependency:get -Dartifact=org.scijava:scijava-common:2.94.2 | grep url:
      url: file:///home/curtis/.m2/repository/
      url: https://repo.maven.apache.org/maven2

We want to see where the local .m2/repository resides. As you can see, mine is in my home directory, but maybe yours is somewhere else?

But you know, now that I wrote all this out, it actually might be something else going wrong: is it possible that Docker does not like the attempt to hard-link? From your log above, the attempt is:

ln /home/jovyan/.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar' '/home/jovyan/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/scifio-labeling-0.3.1.jar`

which only works when the source and destination are on the same physical disk partition. Maybe Docker does not like hard links in general? I'm not sure, but you can try adding the following configuration step to your script:

echo "[settings]" >> ~/.jgorc
echo "links = soft" >> ~/.jgorc

which will change jgo's preferred behavior from hard links (ln) to soft/symlinks (ln -s).

Happy to help advise further if you are still stuck. Let me know whether any of the above ideas enable any progress.

[ctrueden]

One other thing: I just stumbled over #280, which is likely to bite you right now since you are also trying to build a wrapped local Fiji installation. In the meantime, you might want to consider using sc.fiji:fiji:2.14.0 instead as your endpoint, rather than curling down fiji-linux64.zip, as long as you don't need to customize your installation with additional plugins afterward.

[ctrueden]

Finally, if you have sufficient Python programming experience, and you are able to figure this out, it would be awesome if you could think about how to improve the imagej.doctor to catch this situation in future in case other people have the same problem.