Accessing data from read-only cache directory

[jsiegle]

ONE currently assumes write access to the cache directory. However, there are some cases where it would be helpful to mount a database as a read-only input to an analysis environment. And ideally, if the analysis is being carried out on AWS, one should be able to read data from the original bucket (e.g. s3://ibl-brain-wide-map-public/data).

With a local install of the ONE package, I found two places where it tries to write to the cache directory:

• When caching the REST response in webclient.py line 138
• When downloading the remote cache tables in api.py line 1476

After commenting out those actions, I was able to successfully access data from a read-only S3 bucket. However, it failed when I tried to load the spike sorting results due to a file that wasn't downloaded when I first created the cache directory (electrodeSites.brainLocationIds_ccf_2017.npy).

Hopefully it's not too difficult to check whether ONE has write access to the cache directory and use that information to update its behavior.

[k1o0]

Hello, if you already have the data locally you can use ONE in local mode like this: from one.api import ONE; one = ONE(mode='local'). In local mode ONE will not download any data. If you still want to make REST queries, use the cache_rest=None argument. This will no longer save the REST responses locally.

[jsiegle]

Thanks! I'm able to initialize a ONE object in local mode, but it doesn't recognize cache_rest as an input argument (I'm using version 1.18.0).

I can load a trials object from the local directory, but SpikeSortingLoader fails because it's missing the pid2eid function. Any idea what's going on here?

[k1o0]

If you set up ONE with a local directory instead of a database URL it will return an instance of the One class which doesn't have any Alyx database functionality. To ensure you get an instance of OneAlyx (with REST functionality) pass in a database URL like so:

from one.api import ONE, OneAlyx

one = ONE(base_url='https://openalyx.internationalbrainlab.org', cache_rest=None, mode='local')

# Validation
print(one)  # One (offline, https://openalyx.internationalbrainlab.org)
print(type(one))  # <class 'one.api.OneAlyx'>
assert one.offline and isinstance(one, OneAlyx) and one.alyx.cache_mode is None

I'll add this to the ONE documentation FAQ. [Edit] Added here: https://one.internationalbrainlab.org/FAQ.html#how-do-i-use-one-in-a-read-only-environment

[jsiegle]

Thanks! This is working now, with a few minor caveats:

1. Installing ibllib using pip installs the minimum required version of one (1.16.1). I had to run pip install one-api --upgrade to get the latest version.
2. If I don't instantiate a "remote" One object before creating the local one, I have to enter a bunch to parameters upon initialization.
3. The ibl.atlas objects are still downloaded to the default location, rather than being read from the local cache (see Specify download location for BrainAtlas objects ibllib#548)
4. I had to call SpikeSortingLoader with alternate parameters (eid and pname, rather than just pid)

Once I figured these things out, everything went smoothly.