Iqr web demo #10

joshanderson-kw · 2021-11-03T16:25:22Z

No description provided.

Also added async LSH code generation compute function and CLI interface.

Made work function the first argument in order to be more similar in argument format to other map functions.

Plus added loop logging to compute classifications script.

And: - added a descriptor output validation step to update script. - tweaked release note - added image-only note to compute_many_descriptors function due to current content validation code (opens as PIL.Image).

MEMEX specific scripts

Like: - Lazy connection during postgres descriptor index batch operations - remove int cast for parameters in get_cdr_images - tweaked some logging messages - added progress reporting to many-descriptor computation script - added release note

Various updates to minor things

Index label classifier

Before, performed explicitly image content specific actions on input data, which is not generically guarenteed. Added CLI option to optionally enable those checks when user knows input data is going to be imagery. Moved byte loading check from base compute function into the utility script.

…image-dependence Removed explicit image data dependence from util

…tion

Fixed bug with element iteration in ``compute_many_descriptors`` function

Of note: - Changed ``compute_many_descriptors.py`` script default batch size to 0, meaning no high level batches. - Removed old and added new configuration section to docker IQR service config due to the recent addition of IQR controller session timeout/expiration feature. - Caffe descriptor generator can now take a file-like object for its image mean parameter. - Added logging message when running ``parallel_map`` with ordered turned on. - Changed default value for IQR web service session expiration enabled to false.

fix/update configs and comments

script to cluster via MiniBatchKMeans (scikit-learn)

…r index In order to take advantage of any batch-query functionality in descriptor index backend used.

memex hackathon updates -- deep learning classifier configs

Standardized cli_parsing function definition and filled in missing utility hooks in utility documentation file.

Add setup.py installation support

add-deprecation-notice

codecov · 2021-11-03T16:32:51Z

Codecov Report

Attention: Patch coverage is 40.20270% with 177 lines in your changes missing coverage. Please review.

Project coverage is 67.58%. Comparing base (358323d) to head (53d3e86).
Report is 319 commits behind head on master.

Files	Patch %	Lines
smqtk_iqr/utils/compute_many_descriptors.py	29.16%	68 Missing ⚠️
smqtk_iqr/utils/generate_image_transform.py	28.57%	50 Missing ⚠️
smqtk_iqr/utils/compute_functions.py	14.89%	40 Missing ⚠️
smqtk_iqr/utils/nn_index_tool.py	56.41%	17 Missing ⚠️
smqtk_iqr/utils/runApplication.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #10      +/-   ##
==========================================
+ Coverage   64.09%   67.58%   +3.48%     
==========================================
  Files          28       34       +6     
  Lines        3250     3424     +174     
==========================================
+ Hits         2083     2314     +231     
+ Misses       1167     1110      -57

Flag	Coverage Δ
unittests	`67.58% <40.20%> (+3.48%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

docker/smqtk_iqr_playground/Dockerfile

Purg · 2021-11-08T21:44:30Z

While we're not going to get to >=80% coverage, let's add additional unit tests to cover the lines you've modified here (in reference to the "codecov/patch" check).

docker/docker-compose.build.yml

docker/smqtk_iqr_playground/Dockerfile

docker/tpl/caffe1/Dockerfile

docker/tpl/faiss/gpu/Dockerfile

smqtk_iqr/web/search_app/modules/file_upload/FileUploadMod.py

smqtk_iqr/web/search_app/sample_configs/runApp.IqrSearchDispatcher.json

docker/smqtk_iqr_playground/entrypoint.sh

docker/docker-compose.build.yml

docker/smqtk_iqr_playground/default_confs/runApp.IqrClassifier.json

smqtk_iqr/utils/bits.py

smqtk_iqr/utils/combinatorics.py

smqtk_iqr/utils/compute_functions.py

smqtk_iqr/web/search_app/modules/iqr/iqr_search.py

tests/utils/test_bit_utils.py

tests/utils/test_combinatorics.py

docker/smqtk_iqr_playground/default_confs/cpu/compute_many_descriptors.json

* add-missing-files: (64 commits) Fix IQR Playground docker image build issues Fix issues with flake8 and mypy due to rebase on CI update Add some mypy settings to make testing non-trivial Fix various Flake8 exposed issues Update FAISS TPL docker image build to compile actual bytecode Update ABC references to use collections.abc instead of collections Fixup/Update iqr images to use FAISS indexing Update classifier service docker builds IQR image update part 2 - Update/Collapse dockerfile Caffe update part 2 - New caffe TPL setup Update FAISS docker image Fix iqr-playground image build and helper run scripts Renamed DescriptorIndex to DescriptorSet Fix compute_functions.compute_many_descriptors + unit tests Added FAISS docker image build to inherit/copy from Update docstrings Update variable name for clarity Renamed various utility modules with "_utils" in their names PEP8 cleanup of compute_functions.py Avoid loading all data elements for transformation ...

Purg

99% there! Just two minor tweaks before we merge this.

docs/webservices/iqrdemonstration.rst

docker/docker-compose.build.yml

Purg and others added 30 commits January 29, 2016 16:38

Moved compute functions to a sub-module of SMQTK

061bef0

Also added async LSH code generation compute function and CLI interface.

finished hash compute function + CLI frontend script

55d2b90

fixed some documentation

cdcc9b1

moved NoClassificationError home, updated some documentation

34c066b

reordered some arguments to parallel_map

7a26e62

Made work function the first argument in order to be more similar in argument format to other map functions.

report_progress takes logging function, not class

c876a9f

Plus added loop logging to compute classifications script.

Added number processed logging to compute_many_descriptors function

8d20293

add missing logging to compute_many_descriptors

69d967f

added more logging

3600e65

Added more documentation to update script about accounting for results

d9a9c9a

And: - added a descriptor output validation step to update script. - tweaked release note - added image-only note to compute_many_descriptors function due to current content validation code (opens as PIL.Image).

Merge pull request #204 from Purg/dev/memex-scripts

a83f012

MEMEX specific scripts

Various updates to minor things

64bc538

Like: - Lazy connection during postgres descriptor index batch operations - remove int cast for parameters in get_cdr_images - tweaked some logging messages - added progress reporting to many-descriptor computation script - added release note

Merge pull request #214 from Purg/dev/small-fixes

b87410b

Various updates to minor things

Added a final report, fixed typo, added import fail warning

fbbfe1d

Merge pull request #219 from Purg/dev/index-label-classifier

93563d4

Index label classifier

Merge pull request #227 from Purg/dev/descriptor-compute-util-remove-…

6a68bde

…image-dependence Removed explicit image data dependence from util

Fixed bug with element iteration in compute_many_descriptors func…

5787483

…tion

Merge pull request #231 from Purg/dev/fix-descriptor-compute-function

d2b6a5a

Fixed bug with element iteration in ``compute_many_descriptors`` function

Merge pull request #257 from Purg/dev/fix-configs

4486247

fix/update configs and comments

Separated out clustering function + added simple test

03c5a9e

Added note/test about when given an empty index

95eaa2e

Merge pull request #258 from Purg/dev/minibatchkmeans-clustering

0fa94a5

script to cluster via MiniBatchKMeans (scikit-learn)

Reworked clustering compute function to use multi-getter on descripto…

0c6fcfa

…r index In order to take advantage of any batch-query functionality in descriptor index backend used.

Merge pull request #270 from Purg/dev/memex-hackathon-updates

283d8d3

memex hackathon updates -- deep learning classifier configs

moved bin scripts into module, added entry-point hooks in setup.py

b6b5101

Update utilities and documentation

eaf23b7

Standardized cli_parsing function definition and filled in missing utility hooks in utility documentation file.

Merge pull request #271 from Purg/dev/add-setup-py

027b0c0

Add setup.py installation support

Add optional ability to build dataset from files input to cmd

0dfa4b7

bardkw and others added 2 commits July 13, 2021 14:35

Merge branch 'master' into deprecate-mono-repo

f85708e

Merge pull request #424 from bardkw/deprecate-mono-repo

4beffc2

add-deprecation-notice

joshanderson-kw requested a review from Purg November 3, 2021 16:25

joshanderson-kw force-pushed the iqr_demo branch from 09a0951 to b01932e Compare November 3, 2021 16:40

Purg marked this pull request as draft November 4, 2021 14:12

joshanderson-kw marked this pull request as ready for review November 4, 2021 21:47

joshanderson-kw commented Nov 4, 2021

View reviewed changes

docker/smqtk_iqr_playground/Dockerfile Show resolved Hide resolved

Purg reviewed Nov 9, 2021

View reviewed changes