Label volume visualization and painting

[aschampion]

This is a tracking ticket for MVP visualization and painting of label volumes via CATMAID.

Goals

• Visualize label (that is, integer global component ID) volumes as arbitrary label-to-color mapped layers in the CATMAID stack viewer.
• Create, paint, and merge labels into label volumes in the CATMAID frontend.
• Persist mutated label volumes to the CATMAID backend, preventing conflicting concurrent edits and race conditions.

Non-goals

Our vision for CATMAID does not include a rich, general-purpose painting tool or segmentation proofreading tool, encompassing features like:

• Multiscale painting
• Single label volume multilabel painting
• Online meshing and skeletonization
• BigDataViewer- / BigVolumeViewer- like rendering
• Context-aware painting (adaptive filling, etc.)
• Intelligent label splitting
• Semi-automation

Instead, we believe these are the purview of other tools like Paintera / BigCat / neuroglancer and that CATMAID should interoperate with these tools. Our vision for interoperation is incremental, progressive enhancement through three general stages of:

1. Data exchange via import / export
2. Navigation and selection synchronization
3. Versioned state synchronization via common backends / data providers

The plan below builds towards this vision. Relatedly, features that would be handled by such a state synchronization backend are also non-goals, such as the N-dimensional persistence itself, efficient delta exchange and conflict resolution.

Prior work

A painting layer was previously prototyped by Larry in CATSOP (#533, catsop feature branch). However, this was never merged to mainline CATSOP (which in turn was never merged into mainline CATMAID). Painting was never a development focus of CATSOP; the majority of CATSOP development was dealing either with SOPNET-specific issues, WebGL support (merged into CATMAID), database scaling optimization (partially merged into CATMAID), or the more general problem of dependent dataflow synchronization in a distributed, collaborative setting. None of these are goals here.

Draft plan

General prerequisite infrastructure

• Managed a workaround to avoid this: Allow webpack builds in CATMAID frontend. This is necessary so that CATMAID can use the n5-wasm NPM package. It also enables many new features for CATMAID frontend development, including factorization of core CATMAID modules into reusable libraries and use of TypeScript. For now the webpack bundle will coexist with django-pipeline bundled CATMAID.
  • feature branch
  • Related issues: Infrastructure tracking: compilation, source mapping, and TypeScript #1338
  • npm package.json and webpack bundle for frontend
  • Use django-webpack-loader
  • Include bundle in frontend index.html
  • Separate dev and production bundle configurations
  • (future) Migrate existing dependencies out-of-tree to webpack
  • (future) Migrate existing CATMAID modules to webpack
• n5-wasm
  • Bundle via webpack
  • (future) Publish to npm
• Include javascript ndarray or numjs via webpack for ergonomic access to the data returned by n5-wasm.
  • Apparently numjs is built on top of ndarray and has the most useful functionality, slicing and view.

Volumetric stack access and rendering

• Create an ImageBlockLayer, comparable to TileLayer, for rendering N5 image volumes in the StackViewer. Note that unlike tile layers this will be WebGL-only (via Pixi for now) like PixiTileLayer, without a legacy DOM fallback.
  • Factor non-tile-specific functionality into StackLayer abstract base class which both TileLayer and ImageBlockLayer can extend (or into mixin/component if appropriate)
  • XOR
    • Option A
      • Implement a new layer from scratch.
    • Option B
      • Extend ImageBlockLayer from PixiTileLayer, keeping the majority of tile creation logic.
  • ImageBlockLayer should refer to an ImageBlockCache (or ImageBlockStore or ImageBlockManager), similar to the global texture cache.
    • For arcane reasons I can explain offline, it's not currently possible for n5-wasm itself to implement caching beyond the browser request cache.
    • Having this cache be explicit also allows for a kludge for cache sharing between orthoviews into the same N5 stack, which otherwise would not share any caches because they refer to stacks CATMAID believes are different.
    • Either ImageBlockCache should have a CATMAID event bus for mutation and eviction, to which ImageBlockLayer subscribes, or ImageBlockLayer must have explicit notification methods for block-specific redraws. This is necessary for block mutation during painting.
    • ImageBlockCache should also track a state identifier for the last version of each block in the cache received from n5-wasm. Initially this will be either an mtime or content hash.
• Add an imageValueAtLocation or similar method to TileLayer that looks up the image intensity value for a location inside the current view boundary. This would already be useful for querying intensity in raw stacks, but will be necessary for label picking in label stacks.
• N5 tile source type (separate from H2N5 type)
  • For now, one stack per orientation even though this is needless duplication.
    • Otherwise, would need some perspective orientation on the IBL that would break other layers like overlay
  • Each N5 stack needs: dims to slice X, Y, Z, channels
    • Format for now is like http://example.org/path/to/folder.n5/group/dataset/%SCALE_DATASET%/0_1_3
    • (future) Remaining color channels, like H2N5
  • (future) Resolve tile source / volumetric source ambiguity in backend

Label stack specification and visualization

• feature branch (volatile)
• Backend specification of stack type, either Raw or Label. @tomka and I have discussed alternatives (e.g., property of ProjectStack, specification via relation like stack groups and channels) and are fine with this being a new column directly on Stack.
  • Additional metadata for label stacks (e.g., special label IDs like background) could either be stored via StackClassInstance related properties or in the N5 attributes.json. For now this is stored in a magic Stack.metadata property, catmaidLabelMeta.specialLabels.
• Label stacks can only be rendered if WebGL layers are available in the frontend.
• Create a frontend model for label stacks that retrieves label stack metadata and handles requests to the backend to create new label stacks.
• When loading label stacks, in addition to the stack layer the front end creates a label stack manager.
  • Label stack managers are analogous to SkeletonAnnotations in that they coordinate the selection state, display and mutation of label annotations in tools, but rather than having one per project, there is one per open label stack.
  • Label stacks managers include:
    • A stack layer, for now only ImageBlockLayer is supported
    • A label map provider, which converts partition ID, local label ID tuples into a global label ID. For now this is identical to the local label ID (because this iteration only supports labels as global component IDs without any agglomeration graph), but for the sake of future evolution it is crucial this mapping step is possible. Presumably this happens between the ImageBlockCache and the ImageBlockLayer, possibly through a wrapper. In this design label mapping occurs in Javascript and the texture uploaded to the GPU would be the global label IDs.
    • The control of LabelFilter filters that are applied to the ImageBlockLayer through the existing PixiLayer filter system. This is already partially implemented (cc @clbarnes).
      • This filter has a shader that converts the label ID to a linear index, initially via a hash (like neuroglancer or the existing RandomizedLayerColorMap or ObjectLabelColorMap filters).
      • The hashed value is then used as an index into a color texture, e.g., some 2^20 x 1 texture. Initially this can only randomize colors like ObjectLabelColorMap, but can later be adapted to assign particular colors via a label selection source model just as skeleton sources do now.

Painting

• Related issues: paintable areas (as in arealist and areatree types in TrakEM2) #687
• Create a label painting tool.
  • Related issues: Tool refactor #1051, Tracing tool without tracing overlay #1760
  • This tool has an active label stack, for which it owns a mutable block cache that is used by the ImageBlockLayer.
  • The tool has the following action events. This is an attempt to balance CATMAID, TrakEM2, and Paintera interface idioms, but 🚨🚴commence the bikeshedding🚴🚨:
    • Left click paints with the active label in unlabeled voxels, similar to exclusive mode in TrakEM2
      • If no label is active, a new one is created (via request to the backend, see below) and is made active
    • Ctrl-left click paints with the active label, overwriting any previous labeling
    • Alt-left click erases the active label
    • Alt-Ctrl-left click erases any label (paints with the background label)
    • Right click / A selects the label under the cursor as active
    • Shift-left click merges the active label with the label under the cursor
    • D deselects any active label
  • Painting can be implemented in one of two ways:
    • Option A
      • A canvas or separate Pixi texture overlays the label stack. Strokes are painted into this layer while being drawn, then periodically or on stroke release, the buffer is pushed into the ImageBlockCache backing the label layer.
    • Option B
      • Strokes are painted directly into the ImageBlockCache.
  • (future) Other painting enhancements:
    • Adaptive filling to some raw image stack, like TrakEM2 (see Snap 2D painting to gradient of raw data saalfeldlab/paintera#103)
    • Path simplification, e.g., this example
• Mutated blocks are serialized by n5-wasm and POSTed along with their state identifier to the backend (via write-back from frontend cache). For now, the CATMAID backend proxies these requests to a small persistence webservice that writes them to the HTTP served N5 filesystem via rust-n5. Requests either succeed, returning a new state identifier for that block, or fail if the state identifier is stale. More details about the service can be hammered out offline with me. Obviously it's my trojan horse for sneaking versioned backends into CATMAID (or really, getting CATMAID to interoperate with a backend that isn't specific to it).
  • For now, blocks can either be posted individually or as sets that are expected to succeed or fail as an atomic unit. N5 doesn't support multi-block locking, but rust-n5 could be trivially extended to do so. The webservice itself will handle multi-block locking, as it would also be responsible for any future, more elaborate, conflict resolution mechanism.
  • The backend also has an endpoint to reserve a label ID. For now, this is a global label ID, and reservations are non-revokable. The CATMAID proxy creates a ClassInstance representing the CATMAID ontology object for this label.
    • The persistence webservice will initially internally implement this through the usual global max label ID scheme.

Unresolved questions

• Provenance tracking (we are deferring any discussion of this as it is a more general problem)
• Label stack creation
• Label annotation
• Label review
• Pen / tablet support (likely requires pointer events refactoring from Boss tile source #1656)
• Command / undo system integration

Alternatives

• Rust ndarray seems to be wasm-compatible, so it may be possible and have better performance to use that rather than javascript ndarray, and have more of the ImageBlockCache functionality in wasm. This functionality will likely be provided by an external library in future iterations anyway.
• n5-wasm bypassing CATMAID frontend, although this may be undesirable as we want to go through CATMAID's access control and session management infrastructure.
  • n5-wasm could implement a N5Writer directly over WebDAV and bypass the CATMAID backend for block writing, but WebDAV is a mess and few implementations provide locking.
  • n5-wasm could implement a N5Writer for a generic RESTful HTTP backend.

Known limitations

• This feature is not mergeable to master because:
  • It relies on an external webservice
  • It relies on n5-wasm, which requires BigInt, which is currently Chrome-only
  • It relies on WebGL 2.0 for features such as integer textures and integer texel lookup, which is currently only supported in Chrome and Firefox
    • For more explanation of why this is a good tradeoff versus WebGL 1.0 compatibility, see neuroglancer's explanation
• Although the Chrome-only and WebGL 2.0 requirements should allow support for UINT64 volumes, the initial implementation will likely only support UINT32 to postpone working around Javascript numeric issues.
• Conflicts are detected for concurrent edits of the same block, but no conflict resolution mechanism is specified. That is, conflicting concurrent edits other than the first submitted fail and are reset.
• Merging labels is an explicit relabeling mutation on the label volume, rather than a hierarchical agglomeration graph. For manual-scale data and spatially compact structures like nuclei, etc., this should be fine for a first attempt.
• No derived data artifacts are created like server-cached label meshes or skeletonization.
• No 3D viewer visualization.

Misc. notes

• Equivocal terms: CATMAID already has "labels" (text associated with a treenode), which are sometimes referred to as "tags", and "textlabels" (text associated with a location and display properties). The latter are what most other reconstruction tools refer to as "annotations".

cc @tomka @bellonet

[aschampion]

Day 1 meeting outcomes

Strategy

• We'll build towards the draft plan for now, but may not implement painting/mutation in CATMAID initially if non-concurrent editing through Paintera is sufficient.

Design

• Multi-block locking is the responsibility of the application layer (in this case, the webservice) rather than of any N5 implementation.
• ImageBlockCache will eventually be replaced by an external library that can be shared with the webservice, so should be written such that it can be compatible with future features like multiscale and multilabel, even if it does not initially implement them.

Tasks

• @bellonet and @axtimwalde will generate an example label N5 volumes from existing contour data.
• @hanslovsky will demo Paintera with this dataset once it is ready.
• @aschampion will make the dataset browsable in CATMAID via H2N5 once it is ready.
• @aschampion will work on n5-wasm bundling and then the ImageBlockLayer prototype.

[aschampion]

Day 3 group hacking

Tasks

• @aschampion will continue working on label stack manager
  • Global registry for stack -> manager/annotations
  • Per-label-stack manager that
    • Tracks active label
    • Manages special labels
    • Manages appropriate filters on stack layers
• @bellonet will work on painting tool with @tomka and @aschampion
  • Scaffolding
    • Create PaintingTool
    • Create PaintingLayer or PaintingOverlay
    • Create canvas and capture painting strokes

Later Cleanup

• Ice age tool issues
  • Rename EditTool to something more clear like MetaTool
  • Refactor Navigator to use correct prototype implementation
  • Fix TracingTool to prototype inheritance
• Basic requirements should not include pymagick (long, system-lib conflict prone build that's only used for crops and ROI snaps)
• StackLayer interpolation made should be an enum-like of linear, nearest, inherit, rather than a boolean
• Changing the global interpolation mode should only change it for 'inherit' stacks

[aschampion]

Day 5 meeting outcomes

End of day goals were set to show independent progress on the three main components of this solution:

Volumetric stack access and rendering

• Have ImageBlockLayer display a layer from a (raw) stack

Label stack specification and visualization

• Load the dauer raw and label data in three orthoviews with synced label coloring
  • TODO (@aschampion): have LabelAnnotations understand stack group orthoviews, so that the controllers will share settings for all layers of the same underlying stack data.

Painting

• Mouse strokes in the painting tool paint onto a JS canvas

[aschampion]

Some cleanup tasks and code merging plans after the week is up:

N5

• Release v0.3
  • VecDataBlock bound changes
  • Display for DataType
  • Unify modification time changes from n5-wasm into block_metadata by making fields optional

H2N5

• Merge channel packing feature branch
• Block caching feature branch
  • Depend on N5 0.3
  • If a workaround to avoid the unstable feature is found, merge the branch
• Release

n5-wasm

• Depend on N5 0.3
• Modification time ETag
  • First pass: ModifiedReader traits and impls
  • Switch to ETag
  • Second pass: unify reading mtime only with N5AsyncReader via block_metadata changes upstream
• Release v0.0.1 on cargo
• Release v0.0.1 on npm

CATMAID

Merge order/branch dependencies for features:

• Label stack support #1816 Label stacks
• Image block layer #1827 Image block layer
• --EVERYTHING ABOVE IS MERGEABLE. EVERYTHING BELOW IS A BLOCKER--
• Block storage service
• Painting tool
• webpack

[aschampion]

Backend specification of stack type, either Raw or Label.

@tomka Regarding this, now that the frontend LabelAnnotations transitively infers the equivalence of label stacks via groups they are related to as views, I'm wondering if a different approach to specifying stack type should be taken. The motivation is this: if we want to start adding annotations to label IDs, that label ID needs to be scoped to a label stack. However, this may not be one stack, but many sharing the same label set/semantics. Hence we want to be able to create something in the Class system that is a label space, to which either stacks or stack groups can be related. In that design, the most meaningful specification of whether a stack is a label stack is not a property on individual stacks, but whether a stack (via StackClassInstance) or any of its view-related stack groups (via StackGroupClassInstance) is related to a label space class instance.

But maybe we should go with the simpler, Stack field solution first. I just hate making migrations likely to be superceded soon after.

[aschampion]

Summary of changes (mostly from gitter) and additional tasks:

• n5-wasm is being used without webpack initially. This means N5 loading, and most likely painting, will be mergeable.
• Attempting to load N5 stacks in browsers without BigInt (i.e., everything except Chrome for now) support should raise an error.
• To have shared caches for orthoviews into N5 stacks (important both for painting and efficient data loading), the frontend will introduce ReorientedStacks, that wrap existing stacks into new orientations. This will only be available for stacks with image block source mirrors.
  • Deep link URLs, etc.
• This also somewhat depends on fixes described in comments to When updating tiles, check column and row against firstCol and firstR… #1753.

[aschampion]

Progress tracking for tested label bit depth support

bits	vis (shader hash)	shader ids	selection	issues
8	✅			shader generality
16				shader generality
24/32	✅	✅	✅
32	✅			premultiplied alpha
48/64				16bpp, bigint, shader p
53/64				premultiplied alpha, 16bpp, bigint, shader p
64				premultiplied alpha, 16bpp, bigint, shader p, js numerics

[aschampion]

Minor update: bit depths > 32 are waiting on scijs/ndarray#43 to be merged (which will then need a release and an update in numjs).