Crash Report

[Erotemic]

Not sure exactly what happend, but I was attempting to explore a small dataset and I got this traceback:

Note that for multi-users you need to use and configure a launcher.
And to prevent your browser from opening, add '--server' to your command line.
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Some weights of the model checkpoint at facebook/detr-resnet-50 were not used when initializing DetrForObjectDetection: ['model.backbone.conv_encoder.model.layer1.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer2.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer3.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer4.0.downsample.1.num_batches_tracked']
- This IS expected if you are initializing DetrForObjectDetection from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DetrForObjectDetection from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py:1123: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
  warnings.warn(
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/umap/umap_.py:1945: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/numba/np/ufunc/parallel.py:371: NumbaWarning: The TBB threading layer requires TBB version 2021 update 6 or later i.e., TBB_INTERFACE_VERSION >= 12060. Found TBB_INTERFACE_VERSION = 12050. The TBB threading layer is disabled.
  warnings.warn(problem)
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/umap/umap_.py:2437: UserWarning: n_neighbors is larger than the dataset size; truncating to X.shape[0] - 1
  warn(
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py:1123: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
  warnings.warn(
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/umap/umap_.py:1945: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py:1123: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
  warnings.warn(
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py:1123: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
  warnings.warn(
ERROR:root:Exception raised by task = <Task finished name='Task-180' coro=<TransformsApp._update_images() done, defined at /home/joncrall/code/nrtk-explorer/src/nrtk_explorer/app/transforms.py:290> exception=RuntimeError('The expanded size of the tensor (1066) must match the existing size (800) at non-singleton dimension 1.  Target sizes: [1066, 1066].  Tensor sizes: [1066, 800]')>
Traceback (most recent call last):
  File "/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/trame_server/utils/asynchronous.py", line 19, in handle_task_result
    task.result()
  File "/home/joncrall/code/nrtk-explorer/src/nrtk_explorer/app/transforms.py", line 316, in _update_images
    self.compute_predictions_source_images(self.state.source_image_ids)
  File "/home/joncrall/code/nrtk-explorer/src/nrtk_explorer/app/transforms.py", line 267, in compute_predictions_source_images
    annotations = self.compute_annotations(ids)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/code/nrtk-explorer/src/nrtk_explorer/app/transforms.py", line 213, in compute_annotations
    predictions = self.detector.eval(
                  ^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/code/nrtk-explorer/src/nrtk_explorer/library/object_detector.py", line 84, in eval
    predictions_in_baches = [
                            ^
  File "/home/joncrall/code/nrtk-explorer/src/nrtk_explorer/library/object_detector.py", line 87, in <listcomp>
    self.pipeline(images, batch_size=adjusted_batch_size),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/object_detection.py", line 104, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py", line 1143, in __call__
    outputs = list(final_iterator)
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 634, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 678, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    return self.collate_fn(data)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py", line 189, in inner
    padded[key] = _pad(items, key, _padding_value, padding_side)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py", line 120, in _pad
    tensor[i, : len(item[key][0]), :] = item[key][0].clone()
    ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The expanded size of the tensor (1066) must match the existing size (800) at non-singleton dimension 1.  Target sizes: [1066, 1066].  Tensor sizes: [1066, 800]

On a clean run with a 3D UMAP embedding I was able to click around and look at several images, but some gave me an error saying JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex'). With 2D UMap it didn't seem to cause a problem. Traceback is:

(pyenv3.11.9) joncrall@toothbrush:~/code/nrtk-explorer$ nrtk-explorer --dataset ~/code/shitspotter/shitspotter_dvc/test_imgs30_d8988f8c.kwcoco.zip

App running at:
 - Local:   http://localhost:8080/
 - Network: http://127.0.0.1:8080/

Note that for multi-users you need to use and configure a launcher.
And to prevent your browser from opening, add '--server' to your command line.
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Some weights of the model checkpoint at facebook/detr-resnet-50 were not used when initializing DetrForObjectDetection: ['model.backbone.conv_encoder.model.layer1.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer2.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer3.0.downsample.1.num_batches_tracked', 'model.backbone.conv_encoder.model.layer4.0.downsample.1.num_batches_tracked']
- This IS expected if you are initializing DetrForObjectDetection from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DetrForObjectDetection from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/umap/umap_.py:1945: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/numba/np/ufunc/parallel.py:371: NumbaWarning: The TBB threading layer requires TBB version 2021 update 6 or later i.e., TBB_INTERFACE_VERSION >= 12060. Found TBB_INTERFACE_VERSION = 12050. The TBB threading layer is disabled.
  warnings.warn(problem)
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/umap/umap_.py:2437: UserWarning: n_neighbors is larger than the dataset size; truncating to X.shape[0] - 1
  warn(
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py:1123: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
  warnings.warn(
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/umap/umap_.py:1945: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(f"n_jobs value {self.n_jobs} overridden to 1 by setting random_state. Use no seed for parallelism.")
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/umap/umap_.py:2437: UserWarning: n_neighbors is larger than the dataset size; truncating to X.shape[0] - 1
  warn(
 JS Error => Error: Points must be an array of either 2 or 3 dimensional number arrays
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py:1123: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
  warnings.warn(
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py:1123: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
  warnings.warn(
 JS Error => Error: Points must be an array of either 2 or 3 dimensional number arrays
 JS Error => Error: Points must be an array of either 2 or 3 dimensional number arrays
 JS Error => Error: Points must be an array of either 2 or 3 dimensional number arrays
/home/joncrall/.pyenv/versions/3.11.9/envs/pyenv3.11.9/lib/python3.11/site-packages/transformers/pipelines/base.py:1123: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
  warnings.warn(
 JS Error => Error: Points must be an array of either 2 or 3 dimensional number arrays
 JS Error => Error: Points must be an array of either 2 or 3 dimensional number arrays
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')
 JS Error => error: Uncaught TypeError: Cannot read properties of undefined (reading 'getPointSequenceIndex')

[PaulHax]

Thanks the the details!

I have not reproduced that first one yet. Vicente conjectures it may be different sized images sneaking into the same batch.

For the second JS Error I'm not seeing it with the change to ScatterPlot.vue in ##88

[PaulHax]

@Erotemic Can you share the dataset that triggered the first runtime error RuntimeError: The expanded size of the tensor?

[Erotemic]

Yes I can 😉 💩

Torrent: https://academictorrents.com/details/ee8d2c87a39ea9bfe48bef7eb4ca12eb68852c49

IPFS: https://ipfs.io/ipfs/QmNj2MbeL183GtPoGkFv569vMY8nupUVGEVvvvqhjoAATG

Girder: https://data.kitware.com/#user/598a19658d777f7d33e9c18b/folder/66b6bc34f87a980650f41f90

Info: https://github.com/Erotemic/shitspotter

[vicentebolea]

As for the second error this seems to be about our Vue component passing the wrong dimension to the umap property.

For the first error I am still investigating this

[vicentebolea]

@jourdain

[vicentebolea]

@Erotemic I spent sometime trying to reproduce this but I could not, could you share the steps and dataset used to reproduce this?

On another topic we do group images by its pixel size, this is important for batching (an considerable performance improvement technique). However, we do not consider the image color bits meaning that if you have images with different color density this could occur, is this is the case with the dataset. If that is the case the solution in the code will be trivial (convert all images to RGB before passing to the model).