Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Access to large images (>2GB) #22

Open
perlman opened this issue Sep 28, 2020 · 1 comment
Open

Access to large images (>2GB) #22

perlman opened this issue Sep 28, 2020 · 1 comment

Comments

@perlman
Copy link

perlman commented Sep 28, 2020

The napari-omero plugin fails to load large images from OMERO.

This is not too surprising given the use of PixelService to read image data. This image size is 230144x96256x3.

I think this will be addressed with eventual pyramid support as mentioned in #1 .

napari Image:26434
14:18:46 ERROR Failed to getPlane() or getTile() from rawPixelsStore
Traceback (most recent call last):
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero/gateway/__init__.py", line 7466, in getTiles
    rawPlane = rawPixelsStore.getPlane(z, c, t)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero_api_RawPixelsStore_ice.py", line 1199, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
omero.ApiUsageException: exception ::omero::ApiUsageException
{
    serverStackTrace = ome.conditions.ApiUsageException: Converting Long 22152740864 to Integer is an overflow.
        at ome.io.nio.RomioPixelBuffer.safeLongToInteger(RomioPixelBuffer.java:123)
        at ome.services.RawPixelsBean.getPlane(RawPixelsBean.java:392)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.tools.hibernate.SessionHandler.doStateful(SessionHandler.java:216)
        at ome.tools.hibernate.SessionHandler.invoke(SessionHandler.java:200)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy113.getPlane(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy113.getPlane(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172)
        at ome.services.throttling.Callback.run(Callback.java:56)
        at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
        at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140)
        at ome.services.blitz.impl.RawPixelsStoreI.getPlane_async(RawPixelsStoreI.java:105)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at omero.cmd.CallContext.invoke(CallContext.java:85)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy114.getPlane_async(Unknown Source)
        at omero.api._RawPixelsStoreTie.getPlane_async(_RawPixelsStoreTie.java:150)
        at omero.api._RawPixelsStoreDisp.___getPlane(_RawPixelsStoreDisp.java:1107)
        at omero.api._RawPixelsStoreDisp.__dispatch(_RawPixelsStoreDisp.java:1639)
        at IceInternal.Incoming.invoke(Incoming.java:221)
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
        at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
        at Ice.ConnectionI.message(ConnectionI.java:1056)
        at IceInternal.ThreadPool.run(ThreadPool.java:395)
        at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
        at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
        at java.base/java.lang.Thread.run(Thread.java:834)

    serverExceptionClass = ome.conditions.ApiUsageException
    message = Converting Long 22152740864 to Integer is an overflow.
}
14:18:46 ERROR Unhandled exception:
Traceback (most recent call last):
  File "/Users/eric/anaconda3/envs/jax/bin/napari", line 8, in <module>
    sys.exit(main())
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/__main__.py", line 181, in main
    view_path(
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/view_layers.py", line 9, in view_path
    (where <layer_type> is replaced with one of the layer types):
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/components/add_layers_mixin.py", line 847, in open
    self._add_layers_with_plugins(
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/components/add_layers_mixin.py", line 914, in _add_layers_with_plugins
    new = self._add_layer_from_data(*_data)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/components/add_layers_mixin.py", line 986, in _add_layer_from_data
    layer = add_method(data, **(meta or {}))
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/components/add_layers_mixin.py", line 232, in add_image
    layer = self.add_layer(layers.Image(image, **i_kwargs))
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/layers/image/image.py", line 247, in __init__
    self._update_dims()
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/layers/base/base.py", line 406, in _update_dims
    self.refresh()
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/layers/base/base.py", line 655, in refresh
    self.set_view_slice()
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/layers/base/base.py", line 608, in set_view_slice
    self._set_view_slice()
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari/layers/image/image.py", line 554, in _set_view_slice
    image = np.asarray(self.data[self.dims.indices]).transpose(order)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/array/core.py", line 1364, in __array__
    x = self.compute()
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/base.py", line 167, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/base.py", line 452, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/threaded.py", line 76, in get
    results = get_async(
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/local.py", line 316, in reraise
    raise exc
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari_omero/utils.py", line 39, in wrapper_timer
    value = func(*args, **kwargs)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/napari_omero/plugins/loaders.py", line 119, in <lambda>
    get_plane = delayed(timer(lambda idx: pixels.getPlane(*idx)))
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero/gateway/__init__.py", line 7432, in getPlane
    planeList = list(self.getPlanes([(theZ, theC, theT)]))
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero/gateway/__init__.py", line 7499, in getTiles
    raise exc
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero/gateway/__init__.py", line 7466, in getTiles
    rawPlane = rawPixelsStore.getPlane(z, c, t)
  File "/Users/eric/anaconda3/envs/jax/lib/python3.8/site-packages/omero_api_RawPixelsStore_ice.py", line 1199, in getPlane
    return _M_omero.api.RawPixelsStore._op_getPlane.invoke(self, ((z, c, t), _ctx))
omero.ApiUsageException: exception ::omero::ApiUsageException
{
    serverStackTrace = ome.conditions.ApiUsageException: Converting Long 22152740864 to Integer is an overflow.
        at ome.io.nio.RomioPixelBuffer.safeLongToInteger(RomioPixelBuffer.java:123)
        at ome.services.RawPixelsBean.getPlane(RawPixelsBean.java:392)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.tools.hibernate.SessionHandler.doStateful(SessionHandler.java:216)
        at ome.tools.hibernate.SessionHandler.invoke(SessionHandler.java:200)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy113.getPlane(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy113.getPlane(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172)
        at ome.services.throttling.Callback.run(Callback.java:56)
        at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
        at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140)
        at ome.services.blitz.impl.RawPixelsStoreI.getPlane_async(RawPixelsStoreI.java:105)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at omero.cmd.CallContext.invoke(CallContext.java:85)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy114.getPlane_async(Unknown Source)
        at omero.api._RawPixelsStoreTie.getPlane_async(_RawPixelsStoreTie.java:150)
        at omero.api._RawPixelsStoreDisp.___getPlane(_RawPixelsStoreDisp.java:1107)
        at omero.api._RawPixelsStoreDisp.__dispatch(_RawPixelsStoreDisp.java:1639)
        at IceInternal.Incoming.invoke(Incoming.java:221)
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
        at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
        at Ice.ConnectionI.message(ConnectionI.java:1056)
        at IceInternal.ThreadPool.run(ThreadPool.java:395)
        at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
        at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
        at java.base/java.lang.Thread.run(Thread.java:834)

    serverExceptionClass = ome.conditions.ApiUsageException
    message = Converting Long 22152740864 to Integer is an overflow.
}
@joshmoore
Copy link
Collaborator

My assumption would also be that the strategy from #1 would solve this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@joshmoore @perlman