Skip to content

Latest commit

 

History

History
458 lines (347 loc) · 23.6 KB

Appendix-Restoration-Filter.md

File metadata and controls

458 lines (347 loc) · 23.6 KB
Raw

Top level

Restoration Filter

The restoration filter is applied after the constrained directional enhancement filter (CDEF) and aims at improving the reconstructed picture by recovering some of the information lost during the compression process. The restoration filter involves two main filters, namely a Wiener filter and a self-guided restoration filter with projection (SGRPROJ).

The processing of a frame by the restoration filter proceeds by splitting the picture into restoration units. The size of the restoration units is set in the function set_restoration_unit_size and is set to the maximum restoration unit size (RESTORATION_UNITSIZE_MAX) of 256 for both luma and chroma when picture_width*picture_height>352*288, otherwise it is set to 128.

At the frame level, the restoration filter operates with one of the following modes: OFF, Wiener filter, SGRPROJ filter for the whole frame, or switching between those three modes at the restoration unit level.

I. Wiener Filter

1. Description of the algorithm [To be completed]

The Wiener filter is a separable symmetric filter (7/5/3-tap filters), where only three, two or one coefficient(s) for the horizontal and vertical filters are included in the bit stream due to symmetry. The constraints on the Wiener filter to reduce complexity are as follows:

  • The filter is separable. Let c_v and c_h be the wx1 vertical and horizontal filter kernels.

  • The filter kernels c_v and c_h are symmetric:
    $c_v(i)=c_v(w-1-i)$, $c_h(i)=c_h(w-1-i)$, where i=0,1...,r-1.

  • The sum of the coefficients is 1: $\sum c_v(i) = \sum c_h(i) = 1$.

The design of the Wiener filter proceeds in an iterative manner:

  • Starts with initial guess for both the vertical and horizontal filters.
  • Design one of the two filters while holding the second filter fixed, then repeat the process

The filter is designed over windows of size 64x64. The filter taps are either 7, 5, or 3 for luma and 5 or 3 for chroma.

2. Implementation

Inputs to rest_kernel: Output frame of the CDEF filter.

Outputs of rest_kernel: Restored picture, filter parameters.

Control flags:

Table 1. Control flags for the Wiener filter.
Flag Level Description
enable_restoration Sequence Indicates whether to use restoration filters for the whole sequence.
wn_filter_mode Picture Controls the quality-complexity tradeoff of the filter as a function of the encoder mode.
allow_intrabc Picture Indicates whether to enable intra block copy. The restoration filter is not active when allow_intrabc is set to 1.

Step 1 - Splitting the frame into segments

The frame to be filtered is divided into segments to allow for parallel filtering operations on different parts of the frame. Each segment could involve more than one restoration unit. The sizes and number of segments are set as follows (see EbEncHandle.c):

unit_size = 256;
rest_seg_w = MAX((max_input_luma_width /2) / unit_size, 1);
rest_seg_h = MAX((max_input_luma_height/2 ) / unit_size, 1);
rest_segment_column_count = MIN(rest_seg_w,6);
rest_segment_row_count = MIN(rest_seg_h,4);

Each segment may consist of several restoration units. Each restoration unit is split into restoration processing units of size 64x64 for luma (#define RESTORATION_PROC_UNIT_SIZE 64(in EbRestoration.h))

Step 2 – Restoration filter search. Each segment goes through a search to identify the best Wiener filter parameters for each restoration unit in the segment (restoration_seg_search).

Loop over all restoration units in each tile segment (foreach_rest_unit_in_tile_seg)

  • Determine the best filtering parameters for the restoration unit (search_wiener_seg)
    • The initial Wiener filter coeff are computed. See functions svt_av1_compute_stats, wiener_decompose_sep_sym, finalize_sym_filter. (This step may be skipped, see optimization section). Check that the new filter params are an improvement over the identity filter (i.e. no filtering). If not, exit Wiener filter search and do not use Wiener filter. See function compute_score.
    • Refine the initially computed Wiener filter coeffs (see function finer_tile_search_wiener_seg). Up to three refinement steps are performed (with step sizes 4,2,1). In each step, the filter coeffs are shifted according to the step size. Filtering is applied to the filter block with the new coeff values and the SSE is computed (try_restoration_unit_seg). If the refined coeff values are better than the original, the coeff values are updated.
    • The best filter coeffs are returned, along with the corresponding SSE of the filtered block.

Step 3 – Identify the best filtering mode for the whole frame

  • Loop over the picture planes to identify the best restoration option for each of the picture planes
    • Loop over all available filtering options (RESTORE_NONE, RESTORE_WIENER, RESTORE_SGRPROJ, RESTORE_SWITCHABLE), compute the resulting cost for using each of the options over the whole frame, and choose the best option for the whole frame. The selection is based on the rate-distortion cost of the different options. (rest_finish_search)

Step 4 – Filter each restoration unit in the frame using the identified best option from step 3 above. (av1_loop_restoration_filter_frame)

More details on try_restoration_unit_seg

  • Filter stripes of height 64 (try_restoration_unit_seg)
    • Loop over stripes (av1_loop_restoration_filter_unit)
      • Loop over the restoration processing units in a stripe and apply filtering to each restoration processing unit using the already identified best filtering parameters for each restoration processing unit. (wiener_filter_stripe)
  • Compute the sse for the filtered restoration unit (sse_restoration_unit)

II. Self-Guided Restoration Filter with Subspace Projection (SGRPROJ)

1. Description of the algorithm

The main objective behind using the SGRPROJ filter is to smooth the reconstructed image while preserving edges. The filter consists of two main components, namely a self-guided filter and a subspace projection of the reconstructed image.

1.1 Self-guided filter

The self-guided filter feature makes use of a guide image where the objective is to transfer features in the guide image to the reconstructed picture. When the guide image is the same as the reconstructed image, the filter is called a self-guided filter. The objective in this case is to apply filtering that is a function of the spatial characteristics (variance) of the immediate neighborhood of the pixel to be filtered. The main idea behind the filter is outlined below.

A filtered value $\mathbf{p_f}$ of a sample value $\mathbf{p_r}$ in the reconstructed image is generated as follows:

$\mathbf{p_f=fp_r+(1-f)\mu}$

where $\mu$ is the average of a small window w around $\mathbf{p_r}$ in the reconstructed picture and $0<= \mathbf{f} < 1$ is a function of the variance of the samples in the window w.

  • When the variance of w is large, then $\mathbf{f}$ is close to 1, and the filtered sample value is very close to $\mathbf{p_r}$ i.e. very little filtering takes place, and high frequency features (edges) in w are preserved.
  • When the variance of w is very small, then $\mathbf{f}$ is close to 0, and the filtered sample value is very close to $\mu$, i.e. $\mathbf{p_r}$ is replaced by a value close to $\mu$ and smoothing takes place.

The figure below illustrates the main idea behind the filter.

restoration_filter_fig1

Figure 1. Example of how the variance of the area around the sample to filtered affects the selection of the self-guided restoration filter parameters.

The derivation of the filter parameters is outlined below.

  • Compute the mean $\mu$ and the square of the variance $\sigma^2$ of a $(2r+1)x(2r+1)$ window w around the sample $\mathbf{p_r}$ in the reconstructed image.
  • Define $f=\frac{\sigma^2}{\sigma^2+\varepsilon}$, $\mathbf{g=(1-f)\mu}$. The parameter $\varepsilon$ is used to tune the filter.
  • Repeat the same computations above for every sample in the window w (or for a subset of those samples). Define F and G to be the averages of $\mathbf{f}$ and $\mathbf{g}$ computed for all samples in the window w (or for a subset of those samples), respectively.
  • Filtering: $\mathbf{p_f=Fp_r+G\mu}$

The performance of the self-guided filter is generally not sufficient to produce good quality reconstructed images. As a result, a further restoration step is considered and involves the use of subspace projection.

1.2. Subspace Projection

The main idea behind subspace projection is as follows:

  • Construct two restored versions of the reference picture generated using the self-guided filter using two different $(r, \varepsilon)$ parameter pairs.
  • Consider the difference between each of the two restored versions and the reference picture and consider the subspace generated by those two differences.
  • Project the difference between the source image and the reconstructed image into the constructed subspace.

To illustrate the idea of subspace projection, consider the following column vectorized version of the corresponding pictures:

  • $X_s$: Source.
  • $X_r$: Reconstructed.
  • $X_1$ and $X_2$ : Filtered (i.e. restored) versions of $X_r$ using self-guided filter based on parameters ( $r_1,\varepsilon_1$) and ( $r_2,\varepsilon_2$) respectively.
  • $X_f$ : Final restored version of $X_r$. ( $X_f-X_r$ ) is obtained by projecting ( $X_s-X_r$) onto the subspace generated by ( $X_1-X_r$ ) and ($X_2-X_r$ )

restoration_filter_fig2

Figure 2. Illustration of the idea of subspace projection in the SGRPROJ filter.
  • $(X_s-X_r)=\alpha(X_1-X_r)+\beta(X_2-X_r)$
  • $\begin{bmatrix}\alpha \\ \beta\end{bmatrix} = (A^TA)^{-1}A^Tb$
  • $A=[(X_1-X_r)(X_2-X_r)],b=[(X_s-X_r)]$
  • $X_f=(1-\alpha-\beta)X_r + \alpha X_1+ \beta X_2$

2. Implementation

Inputs to rest_kernel: Output frame of the CDEF filter.

Outputs of rest_kernel: Restored picture, filter parameters.

Control flags:

Table 2. Control flags for the SGRPROJ filter.
Flag Level Description
enable_restoration Sequence Indicates whether to use restoration filters for the whole sequence.
sg_filter_mode Picture Controls the quality-complexity tradeoff of the filter as a function of the encoder mode.
allow_intrabc Picture Indicates whether to enable intra block copy. The restoration filter is not active when allow_intrabc is set to 1.

Details of the implementation

The main function calls associated with the SGRPROJ filter as indicated in Figure 3 below.

restoration_filter_fig3

Figure 3. Main function calls associated with the SGRPROJ filter.

The main steps involved in the implementation of the algorithm are outlined below, followed by more details on some of the important functions.

Step 1 - Splitting the frame into segments

The frame to be filtered is divided into segments to allow for parallel filtering operations on different parts of the frame. Each segment could involve more than one restoration unit. The sizes and number of segments are set as follows (see EbEncHandle.c):

unit_size = 256;
rest_seg_w = MAX((max_input_luma_width /2) / unit_size, 1);
rest_seg_h = MAX((max_input_luma_height/2 ) / unit_size, 1);
rest_segment_column_count = MIN(rest_seg_w,6);
rest_segment_row_count = MIN(rest_seg_h,4);

Each segment may consist of a number of restoration units. Each restoration unit is split into restoration processing units of size 64x64 for luma (#define RESTORATION_PROC_UNIT_SIZE 64(in EbRestoration.h))

Step 2 – Restoration filter search. Each segment goes through a search to identify the best SGRPROJ filter parameters for each restoration unit in the segment (restoration_seg_search).

Loop over all restoration units in a given tile segment (foreach_rest_unit_in_tile_seg)

  • Determine the best filtering parameters for the restoration unit (search_selfguided_restoration)

    • Determine the search range for epsilon values [start_ep, end_ep] for epsilon values to use in optimizing the filter parameters, where epsilon is indicated in the description of the algorithm presented above.
    • Loop over the epsilon values in the range [start_ep, end_ep]
      • Loop over 64x64 restoration processing units in the restoration unit (apply_sgr)
        • Filter all samples in the 64x64 restoration processing unit (av1_selfguided_restoration(_avx2 or _c). More details on av1_selfguided_restoration(_avx2 or _c) are included below.
      • Generate the projection of the (Source -Reconstructed) onto the subspace generated by (Filtered_recon_1 - Reconstructed) and (Filtered_recon_2 - Reconstructed), where Filtered_recon_1 and Filtered_recon_2 are two restored version of thereconstructed picture, and generate the corresponding projection coordinates xq[0] and xq[1], which correspond to (α) and (β) in the description of the algorithm outlined above. (get_proj_subspace)
      • Compute the following parameters in (encode_xq)
        • xqd[0]: Clamped value of xq[0]
        • xqd[1]: Clamped value of (128 - xqd[0] - xq[1])
      • Perform a refinement search around the identified parameters xq[0] and xq[1] to see if any other nearby parameters provide a better filtering error. (finer_search_pixel_proj_error)
      • Keep track of the best filtering error for the restoration unit, the corresponding epsilon and (xqd[0], xqd[1]) parameters.
    • Increment a counter sg_frame_ep_cnt[bestep] for the identified best epsilon value from the steps above.
    • Return the best epsilon and (xqd[0], xqd[1]) parameters for the current restoration unit.

  • Filter stripes of height 64 (try_restoration_unit_seg)

    • Loop over stripes (av1_loop_restoration_filter_unit)
      • Loop over the restoration processing units in a stripe and apply filtering to each restoration processing unit using the already identified best filtering parameters for each restoration processing unit. (sgrproj_filter_stripe)

  • Compute the sse for the filtered restoration unit (sse_restoration_unit)

The function calls associated with step 2 above are summarized in the diagram shown in Figure 4 below.

restoration_filter_fig4

Figure 4. Continuation of Figure 3 showing the function calls associated with SGRPROJ filter parameter search.

Step 3 – Identify the best filtering mode for the whole frame

  • Loop over the picture planes to identify the best restoration option for each of the picture planes
    • Loop over all available filtering options (RESTORE_NONE, RESTORE_WIENER, RESTORE_SGRPROJ, RESTORE_SWITCHABLE), compute the resulting cost for using each of the options over the whole frame, and choose the best option for the whole frame. The selection is based on the rate-distortion cost of the different options. (rest_finish_search)

Step 4 – Filter each restoration unit in the frame using the identified best option from step 3 above. (av1_loop_restoration_filter_frame)

More details on av1_selfguided_restoration(_avx2 or _c).

For a given 64x64 block in a restoration unit, integral images D and C corresponding to the sum of elements in the 64x64 block and to the sum of their squares, respectively, are generated. These two integral images make is very easy to compute the mean and variance of any block with arbitrary location and size in the 64x64 block.

The integral images D and C are used to compute the filter parameters for each sample in the 64x64 block. The filter parameters are stored in arrays A and B.

To filter a given sample, the filter parameters for neighboring samples are averaged. The filter parameters are obtained from the arrays A and B. The neighboring samples are taken from a window of size (2r+1)*(2r+1) around the sample to be filtered, where r could be 1 or 2 (r=0 implied SGRPROJ filter is OFF). A weighted average of the neighboring filtering parameters for the neighboring samples is used in filtering the current sample, as outlined above in the description of the filter algorithm. See av1_selfguided_restoration_c, selfguided_restoration_fast_internal and selfguided_restoration_internal for the C implementation, av1_selfguided_restoration_avx2, integral_images, calc_ab_fast, final_filter_fast, calc_ab, final_filter for the avx2 implementation.

3. Optimization of the algorithm

Both the Wiener filter and the SGRPROJ filters involve, at the restoration unit level, a search procedure for the best Wiener filter parameters and for the best SGRPROJ filter parameters. The tradeoff between complexity and quality is achieved by limiting the extent of the filter parameter search.

3.1 Wiener filter search

Wiener filter optimization controls are set in set_wn_filter_ctrls ().

Filter Tap Level

For the wiener filter, the search could be performed using either 3, 5 or 7 tap filters for luma, or 3 or 5 tap filters for chroma. The parameter cm->wn_filter_mode is used to specify the level of filter complexity, where increasing levels of filter search complexity are defined by considering an increasing number of filter taps for both luma and chroma, as given in the table below.

Table 3. Description of the Wiener filter settings for the different wn_filter_mode values.
wn_filter_mode Settings
0 OFF
1 3-Tap luma/ 3-Tap chroma
2 5-Tap luma/ 5-Tap chroma
3 7-Tap luma/ 5-Tap chroma

Filter Coeff Selection

Generally, the Wiener filter coeffs for each restoration unit are computed; however, if the Wiener filter coeff values of ref frames are available, they can be used instead (and the computation can be skipped). When enabled, cm->wn_filter_ctrls.use_prev_frame_coeffs will set the initial coeff values to those chosen by the nearest list 0 reference frame for each corresponding restoration unit. Refinement (if enabled – see next section) will then be performed.

Filter Coeff Refinement

After the initial filter coeff values are selected, a refinement search can be performed to improve the coeff values. The refinement is performed iteratively, with 3 step sizes: 4, 2, 1. By enabling cm->wn_filter_ctrls.max_one_refinement_step only a step size of 4 is used in the refinement (smaller step sizes, which improve granularity of the coeff, and therefore accuracy, will be skipped). To disable the refinement and automatically use the computed coeffs without refinement, set cm->wn_filter_ctrls.use_refinement to 0.

3.2 SGRPROJ filter search

The search for the best SGRPROJ filter is normally performed by evaluating the filter performance for each of the sixteen different $\varepsilon$ values in the interval $[0,15]$, where $\varepsilon$ is used in the outline of SGRPROJ algorithm presented above. The algorithmic optimization of the filter search involves restricting the range of $\varepsilon$ values in the search operation. The parameter cm->sg_filter_mode is used to specify different level of search complexity, where a higher value of cm->sg_filter_mode would correspond to a wider interval of $\varepsilon$ values and a more costly search. The parameter step is used to control the width of the search interval, and is given in the following table.

Table 5. Step parameter as a function of the sg_filter_mode.
sg_filter_mode step
0 OFF
1 0
2 1
3 4
4 16

The optimization proceeds as follows:

  1. The encoder mode enc_mode specifies the SGRPROJ filter mode sg_filter_mode.

  2. The sg_filter_mode specifies the parameter step through the function get_sg_step.

  3. The interval [start_ep, end_ep] of $\varepsilon$ values to search is specified as follows (search_selfguided_restoration):

  • The $\varepsilon$ values sg_ref_frame_ep[0] and sg_ref_frame_ep[1] of the reference pictures are used to define the center of the interval mid_ep as follows: 
    if (sg_ref_frame_ep[0] < 0 && sg_ref_frame_ep[1] < 0) then mid_ep = 0
else if (sg_ref_frame_ep[1] < 0) then mid_ep = sg_ref_frame_ep[0]
else if (sg_ref_frame_ep[0] < 0( then mid_ep = sg_ref_frame_ep[1]else mid_ep = (sg_ref_frame_ep[0] + sg_ref_frame_ep[1]) / 2
  • The interval limits are given by: 
    start_ep = 0 if (sg_ref_frame_ep[0] < 0 && sg_ref_frame_ep[1] < 0), else start_ep = max(0, mid_ep - step)
end_ep = 8 if (sg_ref_frame_ep[0] < 0 && sg_ref_frame_ep[1] < 0), else end_ep = min(8, mid_ep + step)

4. Signaling

Table 7. Restoration filter signals.
Signal Description
At the frame level
frame_restoration_type RESTORE_NONE, RESTORE_WIENER, RESTORE_SGRPROJ, RESTORE_SWITCHABLE
restoration_unit_size Size of restoration unit. For luma: 128x128 or 256x256
At the restoration unit level
restoration_type RESTORE_NONE, RESTORE_WIENER, RESTORE_SGRPROJ
wiener_info Vertical and horizontal filter coefficient array vfilter and hfilter.
sgrproj_info epsilon, projection parameters xqd[0] and xqd[1]

Notes

The feature settings that are described in this document were compiled at v1.4.0 of the code and may not reflect the current status of the code. The description in this document represents an example showing how features would interact with the SVT architecture. For the most up-to-date settings, it's recommended to review the section of the code implementing this feature.

References

[1] Debargha Mukherjee, Shunyao Li, Yue Chen, Aamir Anis, Sarah Parker and James Bankoski, “A Switchable Loop-restoration with Side-information Framework for the Emerging AV1 Video Coding,” International Conference on Image Processing, pp. 265-269, 2017.