Skip to content

Latest commit

 

History

History
924 lines (750 loc) · 30.3 KB

render-ws-java-client.md

File metadata and controls

924 lines (750 loc) · 30.3 KB

Java Clients for Render Web Services

The following Java client applications/tools utilize the render web services to retrieve and/or store persisted specifications. Clients work with retrieved data locally enabling large scale parallel processing of data.

Name Description
Box Client Renders uniform arbitrarily sized boxes to disk for one or more layers of a stack.
Coordinate Client Translates coordinates between stacks with different alignments of common source tiles.
Copy Stack Client Copies tiles from one stack to another.
Import JSON Client Imports JSON tile and transform specifications into a stack.
Import Match Client Imports JSON point match data into a match collection.
Import Transform Changes Client Imports JSON transform changes for existing stack tiles into a new stack.
Mipmap Client Renders down-sampled mipmap images for a stack's source tiles to disk.
Point Match Client Generates and stores SIFT point matches for one or more canvas (e.g. tile) pairs.
Render Section Client Renders a composite image of all tiles in a layer for one or more stack layers.
Section Update Client Updates the z value for all tiles in a stack section.
Stack Client Supports stack management functions like creation, state changing, and deletion.
Tile Pair Client Calculates potential neighbor pairs for all tiles in a range of stack layers.
Tile Removal Client Supports various mechanisms for identifying and removing sets of stack tiles.
Transform Section Client Adds a common shared transform to all tiles in one or more stack layers.
Validate Tiles Client Validates all tiles in one or more stack layers removing any invalid tiles.
Warp Transform Client Generates warp (Thin Plate Spline or Moving Least Squares) transforms for one or more stack layers.

Box Client

Renders uniform arbitrarily sized boxes to disk for one or more layers of a stack. All generated images have the same dimensions and pixel count and are stored within a CATMAID LargeDataTileSource directory structure that looks like this:

[root directory]/[project]/[stack]/[tile width]x[tile height]/[mipmap level]/[z]/[row]/[col].[format]

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.BoxClient [options] Z values for layers to render

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Stack name
  * --rootDirectory
       Root directory for rendered tiles (e.g. /tier2/flyTEM/nobackup/rendered_boxes)
  * --height
       Height of each box
  * --width
       Width of each box
       
    --binaryMask
       use binary mask (e.g. for DMG data)
       Default: false
    --createIGrid
       create an IGrid file
       Default: false
    --label
       Generate single color tile labels instead of actual tile images
       Default: false
    --skipInterpolation
       skip interpolation (e.g. for DMG data)
       Default: false

    --forceGeneration
       Regenerate boxes even if they already exist
       Default: false
    --format
       Format for rendered boxes
       Default: png
    --maxLevel
       Maximum mipmap level to generate
       Default: 0
    --maxOverviewWidthAndHeight
       Max width and height of layer overview image (omit or set to zero to disable overview generation)
       
    --numberOfRenderGroups
       Total number of parallel jobs being used to render this layer (omit if only one job is being used)
    --renderGroup
       Index (1-n) that identifies portion of layer to render (omit if only one job is being used)

Source Code:

Coordinate Client

Translates coordinates between stacks with different alignments of common source tiles.

Although the render web services provide APIs for mapping coordinates directly on the render server, those APIs should only be used for small scale mapping efforts. The coordinate client is designed to minimize remote service requests and maximize local CPU usage for mapping large batches of coordinates.

This tool was originally developed to support batch mapping of JSON coordinates within a specific layer. More recently, support for mapping coordinates across layers and for mapping coordinates in SWC format files was added.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.CoordinateClient [options]

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Stack name
       
    --toOwner
       Name of target stack owner (for round trip mapping, default is source owner)
    --toProject
       Name of target stack project (for round trip mapping, default is source owner)
    --toStack
       Name of target stack (for round trip mapping, omit for one way mapping)
       
    --fromJson
       JSON file containing coordinates to be mapped (.json, .gz, or .zip)
    --toJson
       JSON file where mapped coordinates are to be stored (.json, .gz, or .zip)
       
    --fromSwcDirectory
       directory containing source .swc files
    --toSwcDirectory
       directory to write target .swc files with mapped coordinates

    --numberOfThreads
       Number of threads to use for conversion
       Default: 1
    --localToWorld
       Convert from local to world coordinates (default is to convert from world to local)
       Default: false
    --z
       Z value for all source coordinates

Cluster Usage:

The tool's default setup uses 1 thread for processing allowing it to be run on a single core. Alternatively, you can request more cores and then increase the --numberOfThreads option accordingly. At Janelia, we've mapped 1.8 million points in an hour using ten 32 core nodes with 30 threads per node. This works out to a mapping rate of roughly 100 points per minute per core.

Source Code:

Examples:

JSON World To Local Example

  • Given world-v4.json:
[
  {"world": [230000.0, 79000.0]}
]
  • Map world to local for v4 stack:
map_coordinates.sh --baseDataUrl http://tem-services:8080/render-ws/v1 --owner flyTEM --project FAFB00 --stack v4_align_tps --z 4186 --fromJson world-v4.json --toJson local-v4.json
  • Creates result file local-v4.json:
[
  [
    { "tileId": "140828171446014019.4186.0", "local": [ 2041.0829262039915, 1373.9360369769856, 4186.0 ] },
    { "tileId": "140828171446015019.4186.0", "visible": true, "local": [ 46.64010738010984, 1406.3592571603513, 4186.0 ] }
  ]
]

JSON Local To World Example

  • Map local-v4.json to v5 stack:
map_coordinates.sh --baseDataUrl http://tem-services:8080/render-ws/v1 --owner flyTEM --project FAFB00 --stack v5_align_tps --z 4186 --fromJson local-v4.json --toJson world-v5.json --localToWorld
  • Creates result file world-v5.json:
[
  { "tileId": "140828171446015019.4186.0", "world": [ 28469.717357674046, 36159.01272794745, 4186.0 ] }
]

JSON Round Trip Example

  • Given world-v4.json:
[
  {"world": [230000.0, 79000.0]}
]
  • Map v4 world to v5 world:
map_coordinates.sh --baseDataUrl http://tem-services:8080/render-ws/v1 --owner flyTEM --project FAFB00 --stack v4_align_tps --toStack v5_align_tps --z 4186 --fromJson world-v4.json --toJson world-v5.json
  • Creates result file world-v5.json:
[
  { "tileId": "140828171446015019.4186.0", "world": [ 28469.717357674046, 36159.01272794745, 4186.0 ] }
]

SWC Round Trip Example

  • Given v4/skeleton.swc:
# SWC format file
# based on specifications at http://research.mssm.edu/cnic/swc.html
# PointNo Label X Y Z Radius Parent 
1 0 920000 316000 146510 2 -1
  • Map v4 swc files to v5:
map_coordinates.sh --baseDataUrl http://tem-services:8080/render-ws/v1 --owner flyTEM --project FAFB00 --stack v4_align_tps --toStack v5_align_tps --fromSwcDirectory v4 --toSwcDirectory v5
  • Creates result file v5/skeleton.swc:
# SWC format file
# based on specifications at http://research.mssm.edu/cnic/swc.html
# PointNo Label X Y Z Radius Parent 
1 0 113878.869430696184 144636.0509117898 146510 2 -1

Note that if the v4 directory contained multiple swc files, each one would get mapped and saved to the v5 directory.

Copy Stack Client

Copies tiles from one stack to another.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.CopyStackClient [options]

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --fromStack
       Name of source stack
  * --toStack
       Name of target stack
  * --z
       Z value of section to be copied

    --toOwner
       Name of target stack owner (default is same as source stack owner)
    --toProject
       Name of target stack project (default is same as source stack project)
    --keepExisting
       Keep any existing target stack tiles with the specified z (default is to remove them)
       Default: false
    --maxX
       Maximum X value for all tiles
    --maxY
       Maximum Y value for all tiles
    --minX
       Minimum X value for all tiles
    --minY
       Minimum Y value for all tiles
    --completeToStackAfterCopy
       Complete the to stack after copying all layers
       Default: false
    --replaceLastTransformWithStage
       Replace the last transform in each tile space with a 'stage identity' transform
       Default: false
    --splitMergedSections
       Reset z values for tiles so that original sections are separated
       Default: false

Source Code:

Import JSON Client

Imports JSON tile and transform specifications into a stack.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.ImportJsonClient [options] list of tile spec files (.json, .gz, or .zip)

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Name of stack for imported data
       
    --transformFile
       file containing shared JSON transform specs (.json, .gz, or .zip)
    --validatorClass
       Name of validator class (e.g. org.janelia.alignment.spec.validator.TemTileSpecValidator).  Exclude to skip validation.
    --validatorData
       Initialization data for validator instance.

Source Code:

Import Match Client

Imports JSON point match data into a match collection.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.ImportMatchClient [options] list of canvas match data files, each file (.json, .gz, or .zip) can contain an arbitrary set of matches

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Match collection owner
  * --collection
       Match collection name
       
    --batchSize
       maximum number of matches to batch in a single request
       Default: 10000

Source Code:

Import Transform Changes Client

Imports JSON transform changes for existing stack tiles into a new stack.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.ImportTransformChangesClient [options]

  Options:

  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Name of source stack containing base tile specifications
  * --targetStack
       Name of target (align, montage, etc.) stack that will contain imported transforms
  * --transformFile
       File containing list of transform changes (.json, .gz, or .zip).  
       For best performance, changes for all tiles with the same z should be grouped into the same file.

    --changeMode
       Specifies how the transforms should be applied to existing data
       Default: REPLACE_LAST
       Possible Values: [APPEND, REPLACE_LAST, REPLACE_ALL]
    --targetOwner
       Name of owner for target stack that will contain imported transforms (default is to reuse source owner)
    --targetProject
       Name of project for target stack that will contain imported transforms (default is to reuse source project)
    --validatorClass
       Name of validator class (e.g. org.janelia.alignment.spec.validator.TemTileSpecValidator).  Exclude to skip validation.
    --validatorData
       Initialization data for validator instance.

Source Code:

Mipmap Client

Renders down-sampled mipmap images for a stack's source tiles to disk.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.MipmapClient [options] Z values for layers to render

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Stack name
  * --rootDirectory
       Root directory for mipmaps (e.g. /tier2/flyTEM/nobackup/rendered_mipmaps/FAFB00)

    --forceGeneration
       Regenerate mipmaps even if they already exist
       Default: false
    --format
       Format for mipmaps (tiff, jpg, png)
       Default: tiff
    --minLevel
       Minimum mipmap level to generate
       Default: 1
    --maxLevel
       Maximum mipmap level to generate
       Default: 6

    --numberOfRenderGroups
       Total number of parallel jobs being used to render this layer (omit if only one job is being used)
       Default: 1
    --renderGroup
       Index (1-n) that identifies portion of layer to render (omit if only one job is being used)
       Default: 1

Source Code:

Point Match Client

Generates and stores SIFT point matches for one or more canvas (e.g. tile) pairs.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.PointMatchClient [options] 
         canvas_1_URL canvas_2_URL [canvas_p_URL canvas_q_URL] ... 
         (each URL pair identifies render parameters for canvas pairs)

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Match collection owner
  * --collection
      Match collection name

    --debugDirectory
      Directory to save rendered canvases for debugging (omit to keep rendered data in memory only)
    --matchStorageFile
      File to store matches (omit if matches should be stored through web service) 
       
    --SIFTfdSize
      SIFT feature descriptor size: how many samples per row and column
      Default: 8
    --SIFTminScale
      SIFT minimum scale: minSize * minScale < size < maxSize * maxScale
      Default: 0.5
    --SIFTmaxScale
      SIFT maximum scale: minSize * minScale < size < maxSize * maxScale
      Default: 0.85
    --SIFTsteps
      SIFT steps per scale octave
      Default: 3

    --matchFilter
      Identifies if and how matches should be filtered
      Default: SINGLE_SET
      Possible Values: [NONE, SINGLE_SET, CONSENSUS_SETS, AGGREGATED_CONSENSUS_SETS]
    --matchIterations
      Match filter iterations
      Default: 1000
    --matchMaxEpsilon
      Minimal allowed transfer error for match filtering
      Default: 20.0
    --matchMaxNumInliers
      Maximum number of inliers for match filtering
    --matchMaxTrust
      Reject match candidates with a cost larger than maxTrust * median cost
      Default: 3.0
    --matchMinInlierRatio
      Minimal ratio of inliers to candidates for match filtering
      Default: 0.0
    --matchMinNumInliers
      Minimal absolute number of inliers for match filtering
      Default: 4
    --matchModelType
      Type of model for match filtering
      Default: AFFINE
      Possible Values: [TRANSLATION, RIGID, SIMILARITY, AFFINE]
    --matchRod
      Ratio of distances for matches
      Default: 0.92

    --renderScale
      Render canvases at this scale
      Default: 1.0
    --renderFileFormat
      Format for saved canvases (only relevant if debugDirectory is specified)
      Default: JPG
      Possible Values: [JPG, PNG, TIF]
    --fillWithNoise
      Fill each canvas image with noise before rendering to improve point match derivation
      Default: true
    --clipHeight
      Number of full scale pixels to include in rendered clips of TOP/BOTTOM oriented montage tiles
    --clipWidth
      Number of full scale pixels to include in rendered clips of LEFT/RIGHT oriented montage tiles
      
    --canvasGroupIdAlgorithm
      Algorithm for deriving canvas group ids
      Default: FIRST_TILE_SECTION_ID
      Possible Values: [FIRST_TILE_SECTION_ID, FIRST_TILE_Z, COLLECTION]
    --canvasIdAlgorithm
      Algorithm for deriving canvas ids
      Default: FIRST_TILE_ID
      Possible Values: [FIRST_TILE_ID, FIRST_TILE_Z, CANVAS_NAME]
    --firstCanvasPosition
      When clipping, identifies the relative position of the first canvas to 
      the second canvas
      Possible Values: [TOP, BOTTOM, LEFT, RIGHT]
      
    --numberOfThreads
      Number of threads to use for processing
      Default: 1

Source Code:

Render Section Client

Renders a composite image of all tiles in a layer for one or more stack layers.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.RenderSectionClient [options] Z values for sections to render

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Stack name
  * --rootDirectory
       Root directory for rendered layers (e.g. /tier2/flyTEM/nobackup/rendered_boxes)

    --doFilter
       Use ad hoc filter to support alignment
       Default: true
    --fillWithNoise
       Fill image with noise before rendering to improve point match derivation
       Default: true
    --format
       Format for rendered boxes (png, jpg, tiff)
       Default: png
    --scale
       Scale for each rendered layer
       Default: 0.02

Source Code:

Section Update Client

Updates the z value for all tiles in a stack section.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.SectionUpdateClient [options]

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Stack name
  * --sectionId
       Section ID
  * --z
       Z value

Source Code:

Stack Client

Supports stack management functions like creation, state changing, and deletion.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.StackClient [options]

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Stack name
  * --action
       Management action to perform
       Possible Values: [CREATE, CLONE, SET_STATE, DELETE]

    --cycleNumber
       Processing cycle number
    --cycleStepNumber
       Processing cycle step number
    --materializedBoxRootPath
       Root path for materialized boxes
    --stackResolutionX
       X resoution (in nanometers) for the stack
    --stackResolutionY
       Y resoution (in nanometers) for the stack
    --stackResolutionZ
       Z resoution (in nanometers) for the stack
    --versionNotes
       Notes about the version being created

    --stackState
       New state for stack
       Possible Values: [LOADING, COMPLETE, READ_ONLY, OFFLINE]

    --cloneResultProject
       Name of project for stack created by clone operation (default is to use
       source project)
    --cloneResultStack
       Name of stack created by clone operation
    --skipSharedTransformClone
       Only clone tiles, skipping clone of shared transforms (default is false)

    --sectionId
       The sectionId to delete
    --zValues
       Z values for filtering

Source Code:

Tile Pair Client

Calculates potential neighbor pairs for all tiles in a range of stack layers.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.TilePairClient [options]

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Stack name
  * --minZ
       Minimum Z value for all tiles
  * --maxZ
       Maximum Z value for all tiles
  * --toJson
       JSON file where tile pairs are to be stored (.json, .gz, or .zip)

    --xyNeighborFactor
       Multiply this by max(width, height) of each tile to determine radius for locating neighbor tiles
       Default: 0.9
    --explicitRadius
      Explit radius in full scale pixels for locating neighbor tiles (if set, will override --xyNeighborFactor)
    --zNeighborDistance
       Look for neighbor tiles with z values less than or equal to this distance from the current tile's z value
       Default: 2

    --maxPairsPerFile
      Maximum number of pairs to include in each file.
      Default: 100000

    --baseOwner
       Name of base/parent owner from which the render stack was derived (default assumes same owner as render stack)
    --baseProject
       Name of base/parent project from which the render stack was derived (default assumes same project as render stack)
    --baseStack
       Name of base/parent stack from which the render stack was derived (default assumes same as render stack)

    --excludeCompletelyObscuredTiles
       Exclude tiles that are completely obscured by reacquired tiles
       Default: true
    --excludeCornerNeighbors
       Exclude neighbor tiles whose center x and y is outside the source tile's x and y range respectively
       Default: true
    --excludePairsInMatchCollection
       Name of match collection whose existing pairs should be excluded from the generated list (default is to include all pairs)
    --existingMatchOwner
      Owner of match collection whose existing pairs should be excluded from the generated list (default is owner)
    --minExistingMatchCount
      Minimum number of existing matches to trigger pair exclusion
      Default: 0
    --excludeSameLayerNeighbors
       Exclude neighbor tiles in the same layer (z) as the source tile
       Default: false
    --excludeSameSectionNeighbors
       Exclude neighbor tiles with the same sectionId as the source tile
       Default: false
    --onlyIncludeTilesFromStack
      If specified, this stack is used to whitelist which tile pairs get included from the location stack

    --minX
       Minimum X value for all tiles
    --maxX
       Maximum X value for all tiles
    --minY
       Minimum Y value for all tiles
    --maxY
       Maximum Y value for all tiles

Source Code:

Tile Removal Client

Supports various mechanisms for identifying and removing sets of stack tiles.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.TileRemovalClient [options] tileIds_to_remove

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Stack name

    --tileIdJson
       JSON file containing array of tileIds to be removed (.json, .gz, or .zip)

    --hiddenTilesWithZ
       Z value for all hidden tiles to be removed

    --keepZ
       Z value for all tiles to be kept
    --keepMaxX
       Maximum X value for all tiles to be kept
    --keepMaxY
       Maximum Y value for all tiles to be kept
    --keepMinX
       Minimum X value for all tiles to be kept
    --keepMinY
       Minimum Y value for all tiles to be kept

Source Code:

Transform Section Client

Adds a common shared transform to all tiles in one or more stack layers.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.TransformSectionClient [options] Z values

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Stack name
  * --transformId
       Identifier for transformation
  * --transformClass
       Name of transformation implementation (java) class
  * --transformData
       Data with which transformation implementation should be initialized (expects values to be separated by ',' instead of ' ')
       
    --replaceLast
       Replace each tile's last transform with this one (default is to append new transform)
       Default: false

    --targetProject
       Name of target project that will contain transformed tiles (default is to reuse source project)
    --targetStack
       Name of target stack that will contain transformed tiles (default is to reuse source stack)

    --validatorClass
       Name of validator class (e.g. org.janelia.alignment.spec.validator.TemTileSpecValidator).  Exclude to skip validation.
    --validatorData
       Initialization data for validator instance.

Source Code:

Validate Tiles Client

Validates all tiles in one or more stack layers removing any invalid tiles.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.ValidateTilesClient [options] Z values

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Stack owner
  * --project
       Stack project
  * --stack
       Stack name
  * --validatorClass
       Name of validator class (e.g. org.janelia.alignment.spec.validator.TemTileSpecValidator).
  * --validatorData
       Initialization data for validator instance.

Source Code:

Warp Transform Client

Generates warp (Thin Plate Spline or Moving Least Squares) transforms for one or more stack layers.

Usage:

java -cp current-ws-standalone.jar org.janelia.render.client.WarpTransformClient [options] Z values

  Options:

  * --baseDataUrl
       Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
  * --owner
       Owner for all stacks
  * --project
       Project for all stacks
  * --alignStack
       Align stack name
  * --montageStack
       Montage stack name
  * --targetStack
       Target stack (tps or mls) name

    --alpha
       Alpha value for MLS transform
    --deriveMLS
       Derive moving least squares transforms instead of thin plate spline transforms
       Default: false

    --validatorClass
       Name of validator class (e.g. org.janelia.alignment.spec.validator.TemTileSpecValidator).  Exclude to skip validation.
    --validatorData
       Initialization data for validator instance.

Source Code: