Limit number of FileAnnotations retrieved for OMERO.web UI

[knabar]

Addresses issue #517

The current "Choose attachment" dialog retrieves a list of every FileAnnotation not currently linked to the selected object(s) for linking, causing poor performance or failures on systems with a large number of FileAnnotations.

The goal of this PR is to address the potential performance impact only; any other improvements e.g. to the UI should be addressed in separate PRs.

Main discussion points:

API changes

1. getFilesByObject now returns a tuple of the number of total files and a list of FileAnnotationShims; previously the method returned a single list of FileAnnotations.
2. In case of no objects passed to the controller, getFilesByObject previously returned all FileAnnotations, and will now raise a ValueError. This also means listFileAnnotations is no longer used and could be deprecated.

Testing

There are currently no stated expectations to the exact behavior of the "Choose attachment" dialog or the getFilesByObject method. The UI also does not expose all of the available options of the method. With the large number of permutations of options and object types (and multiple object types, which are not handled), exhaustive testing will be difficult. Current testing is likely minimal.

• We should document the expected behavior of the "Choose attachment" dialog in the possible situations.
• A testing plan is needed.

Potential pitfalls:

• It is currently possible for multiple users to link the same annotation to the same parent object in a read-annotate group

Discussion

• What should be the limit on number of FileAnnotations returned? Currently set to 100, but suggested to increase to several hundred.
• How should FileAnnotations be sorted? Currently by descending updateEvent.time, but suggested to use name as previously.
• Indication in the dialog that the results are truncated; space is currently limited, but some indication that not all available annotations are shown is likely needed.

[will-moore]

With a bunch of FileAnnotations created with https://github.com/will-moore/python-scripts/blob/main/create_file_annotations.py this looks like:

If I add a bunch of FileAnnotations, then these are correctly excluded from the dialog next time I open it and the count goes down accordingly.

Do we want to boost the page size and sort alphabetically as discussed?

[knabar]

Do we want to boost the page size and sort alphabetically as discussed?

@will-moore I'm working on that today.

[will-moore]

I think this PR is failing tests at https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/lastCompletedBuild/testReport/OmeroWeb.test.integration.test_annotate/TestFileAnnotations/test_batch_add_fileannotations_2_/

The test is failing to exclude an annotation previously linked to the selected Projects.
I tested in merge-ci webclient and found the same thing with FileAnnotations previously created by upload. This may be because they don't have a namespace??

[pwalczysko]

the feature works, but I am being offered in the dialog attachments which I already attached. Basically concurring with @will-moore ' testing results

[knabar]

@will-moore @pwalczysko I added the NULL check for the namespace to bring the new query more in line with the old one

[will-moore]

Minor point: hard-coding 500 here and in container.py feels a bit brittle.
I guess you could make a MAX_FILES variable and add it into the context, or add 'file_count': len(files) to the page context and then do {% if total_files > file_count %}, showing {{ file_count }} {% endif %}

[knabar]

Thanks @will-moore, I wanted to do exactly that and then promptly forgot about it. Fixed now.

[will-moore]

Working fine:

[pwalczysko]

Works as expected on merge-ci. lgtm

[will-moore]

Changes look good - tested locally (merge-ci unavailable just now, but had tested there earlier) - all working.

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/unable-to-add-attachment-using-omero-web/84146/22