Geometry / Point selection

[kevinsweeney84]

It would be advantageous to be able to select points within itkwidgets. For example, while viewing a collection of 3D glyphs, to be able to select one and gather some information etc. This is currently supported in vtk.js using vtkInteractorStyleTrackballCamera.

[kevinsweeney84]

@PaulHax @thewtex :
Do we have any idea as to how hard this may be to introduce?
On the project I am currently working on, the inability to do point selection is turning into a blocker.

• E.g. View of projected embedding space with 10k+ data points. Select data (ideally through click) and view corresponding input data.

Might try give it a go myself ... if I knew where to start.

[PaulHax]

I don't think it would be too hard, but not trivial.

Here is the place where the image voxel picker updates.

itk-vtk-viewer/src/Rendering/VTKJS/vtk/ItkVtkViewProxy.js

Line 214 in 6ebfe1c

function updateAnnotations(callData) {

I would start there.

[kevinsweeney84]

@PaulHax
I started to look at these changes and have a demo working away.

However, I am noticing that the ID's returned by model.annotationPicker.getPointId() are off by a factor of about 50 (i.e. ID 1 would be returned as 50, 2 as 100 etc) when using geometries in the viewer (i.e. view(geometries=glyphs.GetOutput()))

Note: (Points from getPointId are shown in Orig Point ID in annotation and actual point ID's are calculated by Math.floor(ID / 50))

I have ran the same code using point_sets instead (i.e. view(point_sets=[point_set_1])) and this returns the correct values. (Points from getPointId shown in Orig Point ID in annotation)

Do we know of any reason why the point IDs returned on glyphs might be off like this?
Can show changes to ItkVtkViewProxy.js file if deemed necessary.

Code from notebook (stitched together)

import numpy as np
from itkwidgets import view
import vtk

# Point View
gaussian_1_mean = [0.0, 0.0, 0.0]
gaussian_1_cov = [[1.0, 0.0, 0.0], [0.0, 2.0, 0.0], [0.0, 0.0, 0.5]]
point_set_1 = np.random.multivariate_normal(gaussian_1_mean, gaussian_1_cov, 3000)

view(point_sets=[point_set_1])

# Glyph View
point_data = vtk.vtkPoints()
polydata = vtk.vtkPolyData()
for ptId in range(point_set_1.shape[0]):
    pt = point_set_1[ptId]

    point_data.InsertNextPoint(pt[0], pt[1], pt[2])
    polydata.SetPoints(point_data)

sphere_src = vtk.vtkSphereSource()
sphere_src.SetRadius(0.08)

glyphs = vtk.vtkGlyph3D()
glyphs.ScalingOff()
glyphs.SetSourceConnection(sphere_src.GetOutputPort())
glyphs.SetInputData(polydata)
glyphs.Update()

view(geometries=[glyphs.GetOutput()])

[PaulHax]

Wow, you figured things out! 👏

Off by constant factor eh? My understanding of the basic VTK datatypes are limited. Maybe when viewing geometries, this does it?
model.annotationPicker.setUseCells(true)

Good source for VTK.js info: https://discourse.vtk.org/c/web/9

[sweenke4]

I have this working for my own usage where I am providing glyphs as input.

However, due to my inability to be able to determine the data type being presented (vtk discource quesion) it would not be viable for me to create a pull request for this work into the master itk-vtk-viewer code.
Further, some additional testing would be required, for example, what if there were both volume and non-volume representations together. Currently the annotations would overlap.

I have attached the changes I have made below in the hope that this could aid the incorporation of this ability to select data points in future revisions of this module.

Bases of code changes made to v10.7.4 of itk-vtk-viewer

ItkVtkViewProxy.js

• Added secondary annotation for use with point ID's
  • Can see current hover point and last selected (clicked) data point

const CursorCornerAnnotationPointID =
  '<table class="corner-annotation" style="margin-left: 0;"><tr><td style="margin-left: auto; margin-right: 0;"">Point ID:&nbsp;&nbsp;</td><td style="text-align:center;" colspan="3">${pointID}</td></tr><tr><td style="margin-left: auto; margin-right: 0;"">(Selected):&nbsp;&nbsp;</td><td style="text-align:center;" colspan="3">${selected}</td></tr></table>'

• Within updateAnnotations() added additional else if to update annotation (if applicable)

...
} else if (model.pointRepresentation) {
  publicAPI.setCornerAnnotation('se', CursorCornerAnnotationPointID)
      
  // KTS: 20th March 2023: Glyphs are represented by 50 data points so to get the point ID for the selected
  // data point we need to divide the pointID value by 50
  publicAPI.updateCornerAnnotation({
    pointID: Math.floor(model.annotationPicker.getPointId()/50),
    selected: model.selectedId,
  })

} else {
  model.lastPickedValues = null
}

• Within model.interactor.onLeftButtonPress() saved the point ID for the clicked point

model.interactor.onLeftButtonPress(event => {

  // KTS: 20th March 2023: Save the ID of the selected point
  if (model.annotationPicker.getPointId() != -1) {
    model.selectedId = Math.floor(model.annotationPicker.getPointId()/50)
  }

  if (model.clickCallback && model.lastPickedValues) {
    model.clickCallback(model.lastPickedValues)
  }
})

• Within publicAPI.addRepresentation filtered the representations to non-volume representations and determined their number

// KTS: 20th March 2023: Filter out the representations to only select the non-volume
const pointRepresentations = model.representations.filter(rep => {
  const isNotVolumeRepresentation = !!!rep.getVolumes().length
  return isNotVolumeRepresentation
})
// KTS: 20th March 2023: Determine the number of non-volume representations
var pointRepresentationsLength = pointRepresentations.length

• Also within publicAPI.addRepresentation I then added the appropriate actor to the annotatioPicker pick list

...
} else if (!!pointRepresentationsLength) {
  // KTS 16 Mar 2023: We want the glyph actor, which will be the last geometry added within the latent explorer tool
  model.pointRepresentation = pointRepresentations[pointRepresentationsLength -1]
  model.pointRepresentation
    .getActors()
    .forEach(model.annotationPicker.addPickList)

  publicAPI.setAnnotationOpacity(1.0)
}

• Finally, added required variable to the DEFAULT_VALUES

const DEFAULT_VALUES = {
  ...
  selectedId: null,   // KTS: 20th March 2023: Added to store the selected (clicked) Point ID information
}

[PaulHax]

Thank your for pioneering this and sharing your results!