Image block layer #1827

aschampion · 2018-12-17T16:42:20Z

The is the second block of changes from #1801 following #1816.

The main new functionality of this branch is the direct rendering of volumetric image data to stack viewers via an ImageBlockLayer. Currently this only supports N5 volumes. Such image block layers are backed by a cache. Mutations of this cache dynamically trigger redrawing of image block layers via the event system. The cache also has hook events for eviction of mutated blocks, for write-back behavior.

Changes also include:

Refactoring to tile source types
Improvements to existing caches
Miscellaneous bugfixes for stack navigation

Remaining backlog:

Move block loading to web workers. For most real-world N5 datasets, existing performance is unusable. Parallelizing this via web workers will help.
- Introduce a promise-based web worker wrapper that can use subsets of CATMAID necessary for tile sources.
- Web worker pre-emption/request cancelling.
WebGL2 parameters and testing for pixel types in this table.
Make filters generic to pixel datatypes.
Update label stack behavior for pixel datatypes.
Squashing
Commit cleanup

aschampion · 2019-01-08T19:31:03Z

Since I may not have time to work on this for a few weeks, I'd rather land it without any of the TODOs other than the cleanup and squashing so that it doesn't rot. The only thing I'd like to wait on, then, is the next wasm-bindgen release so that I can take advantage of some performance improvements in n5-wasm. That shouldn't affect anything else, so I'll go ahead and mark this for review.

tomka · 2019-01-09T13:32:45Z

Great! I try to go through the changes later today. Since the TODOs are still useful changes, we should probably move them to two or three new issues.

django/applications/catmaid/static/js/image-block.js

django/applications/catmaid/static/js/layers/pixi-image-block-layer.js

tomka · 2019-01-11T16:23:33Z

django/applications/catmaid/static/js/layers/pixi-image-block-layer.js

+      let texture = baseTex._glTextures[renderer.CONTEXT_UID];
+      let newTex = false;
+      let width = slice.shape[1];
+      let height = slice.shape[0];


I suppose, shape[1] is width and shape[0] and height because of the transpose(1, 0) call in line 193?

tomka · 2019-01-11T16:59:01Z

Thanks Andrew! I read through the changes and everything looks good, LGTM! The block changes are well structured and provide a reasonable interface for the stack viewer's tiling perspective. Also, numjs seems to provide a nice API.

With respect to WebGL2, I think we should also test for availability initially and show the "Modern browser" message if unavailable. It might be nice to be able to use WebGL2 features in other parts of CATMAID and rely on it being there. As far as I understand, Safari doesn't support WebGL2, but I believe there are only a few people using Safari for CATMAID and I think their user experience with CATMAID is likely better with Chromium or Firefox anyway. Alternatively, we can provide WebGL1 as fallback when constructing the Pixi layer.

It's good that there are checks for BigInt and dynamic imports. Unfortunately, FireFox doesn't seem to be close to BigInt support yet. Also nice to see all the tile source related things being brought together into the TileSource namespace.

This is necessary for stores that need to take action on eviction, e.g., to write mutated values to a backend.

Construct an explicit WebGL2 context and pass it to Pixi. This is necessary to use WebGL2 features elsewhere, e.g., integer textures and sized textures. Pixi seems to work correctly when using this WebGL2 context.

Fix a bug where navigating to the next section would end up returning to the same section because of epsilon and flooring.

Add a tile source type for N5 files served directly over HTTP. Note that this tile source is experimental and the URL format is subject to breaking changes. Since this is the first image block source type, add an abstract base class to the frontend for those types. For loading N5, use n5-wasm at tag 0.1.0. n5-wasm is built release, packaged with wasm-bindgen, then updated to use browser-builting WASM module loading with: https://github.com/FreeMasen/wasm-chrome-hack then edited to use appropriate CATMAID paths. Also includes numjs 0.16.0.

Add a prototype stack layer for displaying image block sources via Pixi.

Allow stacks that have image block sources to be viewed from any orientation by wrapping the stack in the front end. The image block layer permutes dimensions of the loaded blocks appropriate to the orientation, which allows for all views of the stack to use the same cache.

Throw an error if either dynamic import or BigInt are not supported by the browser. Move the dynamic import inside an evaluated string so that it does not prevent Firefox loading the file.

Handle the case at stack boundaries where a block may have fewer Z slices than the block size.

Use this modified time as the state identifier for now. Mtime has the following drawbacks: - Only millisecond resolution in javascript - Not content-verifiable And these advantages: - Free (from request headers) - Standards compliant for other features like cache invalidation

This also converts existing layers' returns to Promises.

This prevents conflict with the event source clear method.

Use new method in N5-wasm to consume rather than copy the data vector, which greatly reduces GC and improves performance when browsing these stacks.

This can occur when the layer is rendered before the image block source has fully loaded.

This matches the behavior of other stack layers.

Fix a case where in particular timing conditions with the image block layer, smooth scrolling would not draw section changes because the microtask queue was not empty, but section changes where slightly slower than the max FPS interval. This one-line change courtesy 5 hours of debugging.

aschampion · 2019-01-16T18:37:50Z

This will merge after CI passes. The version of n5-wasm used is tagged as 0.1.0 in the repo and published as a crate.

aschampion · 2019-01-16T18:52:53Z

Build failure looks like a Travis/sauce/pypy problem, not relevant.

tomka · 2019-01-16T18:58:47Z

🎉

aschampion · 2019-01-16T19:20:14Z

Forgot to do the WebGL2 check/fallback. Will create an issue.

unidesigner · 2019-01-23T17:21:02Z

Fantastic to see this functionality in CATMAID!

I'd be eager to test this on a small scale sample (<1GB) and provide feedback if that's useful. Is there some brief documentation on how to set up the N5/Data Layout and the CATMAID stack? Are uint32 and/or uint64 label stacks also supported?

aschampion · 2019-01-23T18:00:02Z

It's still considered a prototype feature, some details will change, and a lot of functionality is still limited to the console, so there's not much documentation yet. Basic steps are:

Host the N5 over HTTP (will need to support CORS)
Create a CATMAID stack of N5 type, with URL like: https://example.org/path/to/n5/and/dataset/%SCALE_DATASET%/0_1_2, where 0_1_2 are the slicing dimensions for the stack's default orientation.

%SCALE_DATASET% will be substituted to the conventional s0, s1, etc. If you don't have scale levels, you can leave that substitution string out of the URL.
If you have scale levels, also set the CATMAID stack scale level downsample factors appropriately. This data is dynamically read by the client from the N5 attributes using the paintera/BigCat conventions, but because of CATMAID's stack/mirror data model must be duplicated in the stack properties.
Tile width/height...mostly don't matter, eventually won't at all, but for now do create some edge cases on first load. Just set them to the block X Y dimension.
File extension is ignored.

There's no UI for this yet, but once you create an N5 stack you can open it in arbitrary orthviews. For example, if you have a deep link URL with a parameter like sid0=9, you can change it to sid0=9_zy and it will open a ZY view into the stack.

Are uint32 and/or uint64 label stacks also supported?

See this table for what's been tested. All will supported eventually, but each requires solving a few issues, mostly around shader generality and javascript numeric issues.

There's a known bug with blocks at the stack boundary that I fixed previously but must've been lost to rebasing.

unidesigner · 2019-01-23T22:10:01Z

Great, I'll try that and report back.

unidesigner · 2019-01-29T10:47:01Z

Awesome - it works! I used a dataset without scale levels.

A few comments/questions:

When I create the stack in the admin interface, the "Custom downsampling" value seems to automatically populate, but with wrong values so I can not save the stack. Selecting the radio button for "XY downsampling only" and selecting the value to 0 and saving the stack does not work. ( Please correct the errors below. ) What I needed to do is to explicitly delete the content of the "Custom downsampling" field, then it saves the stack.
The trick with setting sid0=9_zy works. Is there a way to manually configure the ortho stacks (e.g. by adding stacks with a particular image base url)? Or should I wait until you implement UI for it.
When creating the n5 file, I used the z5py library. The axis ordering is a bit confusing to me. If I save a numpy array (xyz) in n5, it gets stored internally as (zyx). So I'd need to set /2_1_0 for the CATMAID url. Otherwise, I'd have to transpose the numpy array before writing to use /0_1_2. What would be the recommendation if the access pattern in mostly browsing in z in the XY view?
Is there documentation about the proper syntax for the "Custom downsampling" field? An example would help a lot right after "Downsampling factors along each dimensions for each zoom level." in case this is recommended for N5 types.
For converting a larger stack available as a series of large 2d image tiles to n5: Is there a recommendation for a converter script / binary to do it efficiently?
Possible interpolation issue/artefacts, comparing Neuroglancer (N5 loaded via LocalVolume) and CATMAID (using the sid0=9_zy trick). I set the voxel resolution to 10,10,25 nm (xyz):

(should be exact the same location)

Thanks!

aschampion · 2019-01-29T17:18:15Z

Yes, the downsampling fields are largely broken because it was a fight with django to get them working at all. They annoy me, too, so I'll take another look at them.
You could artificially create N5 stacks with any orientation you want, by changing the slicing dims (as you did by using 2_1_0), but the plan is to create UI for opening the dynamically reoriented stacks so that there's no reason to create ortho stacks in the first place.
Numpy is c/row-major, n5 is fortran/column-major. For best performance you want it to end up so that X is dimension zero such that you end up using 0_1_2. See this discussion.
No, I'll add documentation.
For something truly big you'll want to look at the n5-spark or hot-knife Saalfeldlab repos and adapt that to your purposes. For things in the single terabyte scale you can get it done in reasonable time with n5gest import command.
CATMAID is doing linear interpolation; the vertical seams are the tile boundaries, which OpenGL can't interpolate across. Switch to nearest neighbor interpolation in the layer properties. One of the sets of changes I'm working on will allow layer settings/filters to be saved and loaded by default for stacks.

aschampion · 2019-01-29T20:00:54Z

Note to self that it looks that this may work in Firefox >= 65 since BigInt is now available behind feature flag javascript.options.bigint.

aschampion · 2019-01-29T21:38:26Z

The problem with the downsample factors field seems to be that Django is no longer calling DownsampleFactorsFormField's compress method, despite the fact that it used to, it should, and the docs still indicate it does (in fact it's required!). So the hacky list intermediate representation DownsampleFactorsFormField and DownsampleFactorsWidget pass around to keep the broken MultiValueField validation from going nuts decompresses the decompressed intermediate representation again and ends up with the 1|0|(1,1,1) mess.

I've wasted too much time on this today, but will fix it at some point.

unidesigner · 2019-02-01T22:28:00Z

Thanks for having a look. The problem sounds complicated for such a simple field. One perhaps (?) simple fix from the UI perspective is just to make sure that nothing ends up in the Custom Field when it is not selected, and the selected field remains the selected field (e.g. XY downsampling only) without switching from e.g. 0 to 2 automatically. But yes, it's not of high priority and shouldn't waste more time.
I tried, e.g. by adding a ZY and XZ stack, setting this orientation in the project stack, and indexing with 2_1_0 and 0_2_1 respectively. For some reason, the views ZY and XZ are not pointing to the correct location, but I can't tell where the issue is (the dataset props are "dimensions": [712, 712, 336], "blockSize": [178, 178, 6]). I'll try to import with n5gest and see if it produces the same N5 data layout. See pictures below for the same location in CATMAID and Neuroglancer. Also, a number of requests are made to the N5 file that do not exists (I can't tell if this is due to a boundary access).
Thanks for the pointer. Good to know that it's not only me that got confused.
I have n5gest compiled but I have some difficulty to guess how to call it properly. I do have:
./target/release/n5gest import test.n5 data gzip 178,178,10 /to/folder/*.png
What are the options for compression? Is this the correct way to specify block size? How to pass the files (0000.png, 0001.png, ...)?
They are gone with nearest. Will be nice to have the persistence for the settings/filters, especially to keep the label color maps.

unidesigner · 2019-02-05T21:10:56Z

Found https://asciinema.org/a/208278 - import now works.

aschampion · 2019-02-06T16:11:55Z

Yeah, the documentation for import needs to be improved, but I'd like to find a way to make the compression types generate their own documentation first.

Re: 2. If your dimensions are still such that your "natural" XYZ axes are 210, having a stack with an ZY orientation and 210 slicing dims doesn't make much sense to me -- it's not going to change any of the performance issues with the transpose slicing, and the project space is even more permuted from the "natural" axes. Note also that the reorientation of the "_zy" suffixes in the URL is applied on top of the stack-project orientation.

aschampion · 2019-02-06T16:15:11Z

Also in those screenshots it looks like the resolution in the stack's Z is squashed (while in neuroglancer it looks correct). Is it set wrong in the stack properties, or could this be a bug with the anisotropic rendering?

unidesigner · 2019-02-06T16:45:55Z

I confirmed now that my conversion of my numpy array with shape [712,712,336] to n5 using a) transpose first and then save with z5py, and b) using n5gest on an image series of 712x712 files matches.

So, the following are my two stack configurations for a XY and a XZ stack:

XY:

XZ:

Am I missing a suffix?

Re: anisotropy: see stack config above. I do not have any additional attributes in the n5 file. It is squashed in CATMAID, but not in NG.

aschampion · 2019-02-06T16:51:57Z

If you create an explicit XZ stack like that (as opposed to the implicit reoriented stacks created by the "_xz" URL suffixes), you need to manually permute the properties. So the dimension should be [712, 336, 712] and the resolution should be [10, 25, 10]. Otherwise they look fine -- I didn't realize you had already fixed your z5py transposed stack.

Getting rid of this manual duplication/permutation tedium is the point of the front-end reorientation. I'll add UI to make that easier to use soon.

unidesigner · 2019-02-06T17:17:23Z

With permuting the dim/res, it works now as expected, including the anisotropy, great!

Re the compression parameter: Would be good if they can generate their documentation themselves. Right now, I don't know how to find out what is available. Btw, with the github.com/seung-lab/compressedseg method, I get quite good compression ratios.

Two more question regarding n5gest:

How do you use it for label volumes? I tried to import an uint32 from a tiff series but it failed ("32 bits per channel not supported"). I guess it's the same for uint64.
Is generating multi-scale volumes out of scope for n5gest or are you planning to implement this? Otherwise, I'll have to look into the Saalfeld n5-Spark toolchain.

aschampion · 2019-02-06T21:24:49Z

n5gest import is a quick hack to avoid having to do something via n5-spark for small everyday workloads. Unfortunately, adding 32bpp import support would be a bit of work since it would require adding it to the upstream rust tiff and image libraries. (Though I have done that for other features I needed.) Alternatively, writing some sort of ImgLib2-like more image processing focused rust image library if I could find a conspirator. Image stack import/export is one of the least developed/slowest parts of n5-spark, however.
I hadn't planned on it, since n5-spark does this very well and (at least on the Janelia cluster) is simple to get running.

aschampion · 2019-02-06T21:31:27Z

Almost forgot: related to label import/downsampling, you should also be aware of Philipp's great paintera-conversion-helper. The winner-take-all (i.e., unilabel) mode of this is what we're targeting for CATMAID to be able to support in the near term.

For reorientable stacks, add buttons to the layer controls to open reoriented stack views. Correctly generate deep link URLs for stack viewers whose primary stacks are reoriented. Post-polish on #1827.

This UI has always been cumbersome, but recently broke entirely for custom factors do to Django changes. Fix this and simplify the interaction between field and widget here. This is deceptively difficult because: - MultiValueField and MultiWidget assume list (and *sometimes* tuple) types are already decompressed. Downsample factors are a list of Integer3D, and have to be so elsewhere for use as an ArrayField. - SimpleArrayField also makes assumptions about coupling with its default widget. - The SimpleArrayFields used are of Integer3DFormField, which is another MultiValueField/MultiWidget of list type. See discussion in #1827.

unidesigner · 2019-03-28T06:51:39Z

You probably saw this already, but Neuroglancer now supports N5 natively. Although with some specific expectations for the file layout.

@aschampion Could be nice to revisit the synchronized navigation between CATMAID and Neuroglancer you implemented at some point. Is this code in some branch?

aschampion mentioned this pull request Dec 17, 2018

Label volume visualization and painting #1801

Open

43 tasks

aschampion requested a review from tomka January 8, 2019 19:31

tomka reviewed Jan 11, 2019

View reviewed changes

django/applications/catmaid/static/js/image-block.js Show resolved Hide resolved

tomka approved these changes Jan 11, 2019

View reviewed changes

django/applications/catmaid/static/js/layers/pixi-image-block-layer.js Outdated Show resolved Hide resolved

django/applications/catmaid/static/js/layers/pixi-image-block-layer.js Outdated Show resolved Hide resolved

tomka reviewed Jan 11, 2019

View reviewed changes

aschampion added 22 commits January 16, 2019 13:13

Abstract error for unimplemented abstract methods

5e105f2

LRU cache: emit events on eviction

7e9e868

This is necessary for stores that need to take action on eviction, e.g., to write mutated values to a backend.

Pixi context: use WebGL2

85540d1

Construct an explicit WebGL2 context and pass it to Pixi. This is necessary to use WebGL2 features elsewhere, e.g., integer textures and sized textures. Pixi seems to work correctly when using this WebGL2 context.

Stack: account for epsilon in z transform

f1f2bc7

Fix a bug where navigating to the next section would end up returning to the same section because of epsilon and flooring.

Add prototype PixiImageBlockLayer

0f2ae1f

Add a prototype stack layer for displaying image block sources via Pixi.

N5 image block source: do not break firefox

960117c

Throw an error if either dynamic import or BigInt are not supported by the browser. Move the dynamic import inside an evaluated string so that it does not prevent Firefox loading the file.

Image block stacks: handle out-of-bounds and empty blocks

6fde3b6

Image block layer: handle Z undersized blocks

bf18d0a

Handle the case at stack boundaries where a block may have fewer Z slices than the block size.

Image block layer: use own width/height (minor)

1846b09

Pixi layer: fix interpolation mode setting

c54f280

Image block layer: implement pixel reading

cedea35

This also converts existing layers' returns to Promises.

LRU cache: rename clear method to evictAll

21a569c

This prevents conflict with the event source clear method.

N5 tile source: consume block into data

85f0536

Use new method in N5-wasm to consume rather than copy the data vector, which greatly reduces GC and improves performance when browsing these stacks.

Image block layer: handle unknown dtype

0d5cfaa

This can occur when the layer is rendered before the image block source has fully loaded.

Image block layer: ignore old buffer swaps

6d6defd

This matches the behavior of other stack layers.

N5 image block source: switch from mtime to etag

fbdd482

N5 image block source: fix canary check

724f8f6

Image blocks: deduplicate simultaneous requests

a768707

Image block layer: reduce allocations and GC

d3c411e

aschampion added 2 commits January 16, 2019 13:26

Image block layer: fix several rendering problems

0d5f22f

aschampion force-pushed the features/image-block-layer branch from 1c13ea9 to 0d5f22f Compare January 16, 2019 18:32

aschampion merged commit 027c17a into dev Jan 16, 2019

aschampion mentioned this pull request Jan 16, 2019

PixiContext should fall back to WebGL1 if WebGL2 is not available #1837

Closed

aschampion deleted the features/image-block-layer branch January 18, 2019 19:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Image block layer #1827

Image block layer #1827

aschampion commented Dec 17, 2018 •

edited

Loading

aschampion commented Jan 8, 2019

tomka commented Jan 9, 2019

tomka Jan 11, 2019

tomka commented Jan 11, 2019

aschampion commented Jan 16, 2019

aschampion commented Jan 16, 2019

tomka commented Jan 16, 2019

aschampion commented Jan 16, 2019

unidesigner commented Jan 23, 2019

aschampion commented Jan 23, 2019 •

edited

Loading

unidesigner commented Jan 23, 2019

unidesigner commented Jan 29, 2019

aschampion commented Jan 29, 2019

aschampion commented Jan 29, 2019

aschampion commented Jan 29, 2019

unidesigner commented Feb 1, 2019

unidesigner commented Feb 5, 2019

aschampion commented Feb 6, 2019

aschampion commented Feb 6, 2019

unidesigner commented Feb 6, 2019 •

edited

Loading

aschampion commented Feb 6, 2019

unidesigner commented Feb 6, 2019

aschampion commented Feb 6, 2019

aschampion commented Feb 6, 2019

unidesigner commented Mar 28, 2019

Image block layer #1827

Image block layer #1827

Conversation

aschampion commented Dec 17, 2018 • edited Loading

aschampion commented Jan 8, 2019

tomka commented Jan 9, 2019

tomka Jan 11, 2019

Choose a reason for hiding this comment

tomka commented Jan 11, 2019

aschampion commented Jan 16, 2019

aschampion commented Jan 16, 2019

tomka commented Jan 16, 2019

aschampion commented Jan 16, 2019

unidesigner commented Jan 23, 2019

aschampion commented Jan 23, 2019 • edited Loading

unidesigner commented Jan 23, 2019

unidesigner commented Jan 29, 2019

aschampion commented Jan 29, 2019

aschampion commented Jan 29, 2019

aschampion commented Jan 29, 2019

unidesigner commented Feb 1, 2019

unidesigner commented Feb 5, 2019

aschampion commented Feb 6, 2019

aschampion commented Feb 6, 2019

unidesigner commented Feb 6, 2019 • edited Loading

aschampion commented Feb 6, 2019

unidesigner commented Feb 6, 2019

aschampion commented Feb 6, 2019

aschampion commented Feb 6, 2019

unidesigner commented Mar 28, 2019

aschampion commented Dec 17, 2018 •

edited

Loading

aschampion commented Jan 23, 2019 •

edited

Loading

unidesigner commented Feb 6, 2019 •

edited

Loading