Float funque plus features #9

MallikarjunKamble · 2023-10-20T11:56:44Z

Floating Point Implementation for new features:

Added support for 5-Tap nadeanu spatial csf
Added support for nadenau_weight, li, hill, watson and mannos_weight wavelet csf
Added MS_ESSIM feature
Added ST-RRED feature

Fixed:

Repeated aligned_malloc memory allocation for dwt buffers has been removed
Added json file for float_funque
Added features list in integer_funque

Co-authored-by: Ajayshyam <[email protected]> Co-authored-by: Rakshith G B <[email protected]> Co-authored-by: RakshithGB <[email protected]> Co-authored-by: Neha Mary Thomas <[email protected]> Co-authored-by: Jeeva Raj A <[email protected]>

Added integer implementation for ADM, VIF, Motion score, SSIM, Spatial filter, DWT Added new feature extractor for integer funque as vmaf_fex_integer_funque. Added new json file funque_integer.json Optimisations to VIF, ADM, spatial filter, SSIM, DWT modules Cleaned up unused memory Some other updates: VIF: Merged computation of var_x, var_y, cov_xy into a single loop. Memory optimization by removing intermediate buffers Merged the last stage(numerator & denominator) score computations to the integral image function ADM: Avoided computations for pixels outside the border, that are not used for score computation Optimised integral image (added data across 3 bands & used for integral computation) Memory optimizations by removing intermediate buffers Merged the last stage (numerator & denominator) score computations to the integral image & decouple functions Resizer: Optimised resizer when resizing width & height by 2.

SIMD optimisations for 32bit & 64bit ARM Memory optimizations for integer FUNQUE (ADM & VIF) Configurable reflect padding in integer VIF & ADM Higher bit-depth support in resizer & spatial filter. All other modules are bit-depth independent. Resolved all the build warnings in FUNQUE modules Resolved memory leak in libvmaf code (when the reference and distorted videos have an unequal number of frames) Mink3 pool SSIM (configurable using a macro) Stability fixes for VIF (configurable using a macro)

Code clean-up and removal of warnings Resolved build error reported on the last pull request Build system updates to support ARM 32-bit compilation Bug fix in ADM (handling overflow during accumulation)

Added README.md for funque, located at libvmaf/src/feature/third_party/funque/README.md. It contains the following details: Compile & build steps Funque model & feature name configurations The build issue for hbd_resizer in float has been resolved.

* Add AVX2 version of CSF * Add AVX2 version of DWT * Add avx2 version of compute_vif and vif_horz_integralsum * Add AVX2 version of compute_SSIM * Use ARCH_X86 flag for including avx2 files and VMAF_X86_CPU_FLAG_AVX2 to map the avx2 function Separate the AVX2 and C code in a x86 folder * Add AVX2 version of ADM * Add SSE computation in integer_adm_declouple_avx2 and integer_compute_ssim_funque_avx2 Co-authored-by: PierreIntel <[email protected]>

* Add AVX2 version of integer_funque_image_mad_avx2 * Add AVX2 version of SAST (h/vresize, 8 & 10 bits) Co-authored-by: PierreIntel <[email protected]>

…nfigurable

…ith it, marking it temporal kills parallelism, best to keep motion in separate feature extractor we can leverage from vmaf

…gn the DWT decomposition to allow ssim and ADM calculation on all needed levels

…eights

- Add structure for ST-RRED levels - Add function caller - Add functions in the feature collector - Add ST-RRED functions in the provided features - Add ST-RRED features in json files - Add files required for build

- Add leaf level function used to compute entropy and scale - Add the MACROS and functions definitions - Borrow reflect padding and integral image code from VIF feature

…ions

…ikarjunKamble/vmaf into WAVELET_DEBUG_MSSSIM_STRRED

…BUG_MSSSIM_STRRED

abhinaukumar · 2023-12-06T04:36:34Z

libvmaf/src/feature/third_party/funque/funque_filters.c

+
+void spatial_csfs(float *src, float *dst, int width, int height, float *tmp_buf, int num_taps)
+{
+    int fwidth;


fwidth appears to be a duplicate of num_taps.

We will address num_taps redundancy in the next commit

abhinaukumar · 2023-12-06T04:37:28Z

libvmaf/src/feature/third_party/funque/integer_funque_filters.c

+void integer_spatial_filter(void *src, spat_fil_output_dtype *dst, int dst_stride, int width,
+                            int height, int bitdepth, spat_fil_inter_dtype *tmp, int num_taps)
+{
+    int fwidth;


fwidth appears to be a duplicate of num_taps.

We will address num_taps redundancy in the next commit

abhinaukumar · 2023-12-06T04:43:04Z

libvmaf/src/feature/third_party/funque/funque_ssim.c

+            cs_sum += cs;
+            ssim_sq_sum += (l * cs) * (l * cs);
+            l_sq_sum += l * l;
+            cs_sq_sum += cs * cs;


l and cs need to be mink3 pooled for use in MS-ESSIM.

Addressed. Please review and close.

abhinaukumar · 2023-12-06T04:45:58Z

libvmaf/src/feature/third_party/funque/funque_ssim.c

+    return ret;
+}
+
+int compute_ms_ssim_mean_scales(MsSsimScore* score, int n_levels)


This function doesn't appear to be handled appropriately under ENABLE_MINK3POOL. Flagging to double-check.

This is ssim. We will correct the code in ssim and add support for mink3 pooling and update in next commit.

abhinaukumar · 2023-12-06T04:48:36Z

libvmaf/src/output.c

@@ -99,7 +99,7 @@ int vmaf_write_output_xml(VmafContext *vmaf, VmafFeatureCollector *fc,
            int err = vmaf_feature_score_pooled(vmaf, feature_name, j, &score,
                                                0, pic_cnt - 1);
            if (!err)
-                fprintf(outfile, "%s=\"%.6f\" ", pool_method_name[j], score);
+                fprintf(outfile, "%s=\"%.30f\" ", pool_method_name[j], score);


Are these .30f formattings deliberately left in or are they holdovers from debugging?

This is to take care of ST-RRED accuracy. ST-RRED values go upto E-30

…amble/vmaf into WAVELET_DEBUG_MSSSIM_STRRED

MallikarjunKamble · 2023-12-11T15:19:33Z

Fixed Point Implementation:

Addition of Multi-Scale feature computation (All 4 scales)
Addition of support for crop width and crop height
Addition of MINK3 pool in MS_SSIM
Bug fixes for accuracy improvements wrt python

Enhancements to Floating point:

Addition of support for crop width and crop height
Addition of MINK3 pool in MS_SSIM
Addition of filter coeffs for Li and Hill

ajayshyam and others added 30 commits January 3, 2023 09:42

More fixes

6e5c39d

Code clean-up and removal of warnings Resolved build error reported on the last pull request Build system updates to support ARM 32-bit compilation Bug fix in ADM (handling overflow during accumulation)

Funque documentation (#5)

a8a1447

Added README.md for funque, located at libvmaf/src/feature/third_party/funque/README.md. It contains the following details: Compile & build steps Funque model & feature name configurations The build issue for hbd_resizer in float has been resolved.

AVX2 Second release (SAST and MAD) (#7)

6bf8376

* Add AVX2 version of integer_funque_image_mad_avx2 * Add AVX2 version of SAST (h/vresize, 8 & 10 bits) Co-authored-by: PierreIntel <[email protected]>

add missing newlines at the end of some files

1940a9a

fix arm build by removing x86 bits out of shared code

7c318f1

add an option for disabling the spatial csf

4fab11a

compute multiple ADM levels, add missing DWT csf, make spatial csf co…

34e342c

…nfigurable

use mean pooling for ssim if minkowski is turned off

403f924

remove motion as feature extractor can't work with multiple threads w…

099fa31

…ith it, marking it temporal kills parallelism, best to keep motion in separate feature extractor we can leverage from vmaf

compute ssim at multiple levels, provide a single aggregated adm score

6b72aca

some formatting changes

c0a90dd

start to align integer code with the float version, still need to ali…

4ed4786

…gn the DWT decomposition to allow ssim and ADM calculation on all needed levels

only allocate spat_filter if spatial_csf is on, pre-compute DWT csf w…

5641a29

…eights

fix names for integer funque features after the alignment with float

21ac467

Update feature names

f3082fd

Revert last two commits

7b53972

Add ST-RRED code structure.

b7875cd

- Add structure for ST-RRED levels - Add function caller - Add functions in the feature collector - Add ST-RRED functions in the provided features - Add ST-RRED features in json files - Add files required for build

Implement the ST-RRED functionality

561c99f

- Add leaf level function used to compute entropy and scale - Add the MACROS and functions definitions - Borrow reflect padding and integral image code from VIF feature

Implement code to compute temporal ST-RRED values

030cbc1

Add 5-Tap nadeanu spatial filter

330d77f

Code cleanup for spatial filter

4da1471

Add function argument to pass frame index

e22313d

Bug Fix: Pass correct parameter values for entropy and scale calculat…

00f5253

…ions

Add aggregation function to compute spatial values

cbe88d6

Add code structure to compute temporal values

5d8783f

Copy previous frame values

3e402f8

niranjankumar-ittiam and others added 17 commits November 28, 2023 18:19

Fix MS_SSIM integer code for levels 2 and 3

2df28f2

Fix for handling sign in mean scale computation

5170aa2

Fixed all the warnings

5a5cde4

Fixed ms_ssim struct pointer

a2675ac

Fix precision loss in MS_SSIM co variance

08852bc

Added fixes to match float wavelet scores

a61cddc

Added changes to process width and height based on spat csf

203ea73

Integer: Added changes to process width and height based on spat csf

3d9abc3

Fix precision loss in MS_SSIM covariance

101a3d1

Fixes for feature not appending scores

54d27df

Merge branch 'WAVELET_DEBUG_MSSSIM_STRRED' of https://github.com/Mall…

eb74fa4

…ikarjunKamble/vmaf into WAVELET_DEBUG_MSSSIM_STRRED

Added 22-bit lookup table support for wavelet function

0e5beb1

Added fixes for ms_ssim negative values

d0b6f93

Add Mink3 integer code

0cf86d5

Add Mink3 int values to dict

3e29581

Fix the dwt caller to send correct level

f70f13c

Merge remote-tracking branch 'origin/Wavelet_dwt_fix' into WAVELET_DE…

0f96840

…BUG_MSSSIM_STRRED

abhinaukumar reviewed Dec 6, 2023

View reviewed changes

niranjankumar-ittiam and others added 6 commits December 7, 2023 14:23

Fix wavelet MS-SSIM mismatch in scales 1, 2 and 3 integer code

f5e6de7

STRRED: Added fixes for variance overflow

8eb23a9

Fixes for variance overflow

e7f4cfc

Added strred to calculate max levels

ca53eb2

Implement clang-format for consistent code styling

27c6124

Merge branch 'WAVELET_DEBUG_MSSSIM_STRRED' of github.com:MallikarjunK…

ff4392e

…amble/vmaf into WAVELET_DEBUG_MSSSIM_STRRED

MallikarjunKamble closed this by deleting the head repository Dec 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Float funque plus features #9

Float funque plus features #9

MallikarjunKamble commented Oct 20, 2023

abhinaukumar Dec 6, 2023

MallikarjunKamble Dec 11, 2023

abhinaukumar Dec 6, 2023

MallikarjunKamble Dec 11, 2023

abhinaukumar Dec 6, 2023

MallikarjunKamble Dec 11, 2023

abhinaukumar Dec 6, 2023

MallikarjunKamble Dec 11, 2023

abhinaukumar Dec 6, 2023

MallikarjunKamble Dec 11, 2023

MallikarjunKamble commented Dec 11, 2023

Float funque plus features #9

Float funque plus features #9

Conversation

MallikarjunKamble commented Oct 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MallikarjunKamble commented Dec 11, 2023