Forward vs. Backward Flow Field Estimation

[cverrier]

Hi, I just found your repository and I tried to dig deeper into the details of the model you built to estimate optical flow.

I have a question regarding the direction of the flow field that the network outputs: is it a forward flow field or a backward flow field?

Some precisions:

• By 'forward flow field', I mean "start with pixels in the first frame and try to find where they move to in the second frame", or in other words, "where does each pixel in the first frame end up in the second frame?", which also means that "the flow vectors point from the first frame to the second frame".
• By 'backward flow field', I mean the opposite of the forward flow field, i.e. "start with pixels in the second frame and try to find where they came from in the first frame", i.e. "where did each pixel in the second frame come from in the first frame?", i.e. "the flow vectors point from the second frame back to the first frame".

Based on the code that I read, the flow estimation starts with the computation of the correlations between the features associated with the two input frames:

NeuFlow_v2/NeuFlow/neuflow.py

Line 92 in 204b5e3

flow0 = self.matching_s16.global_correlation_softmax(feature0_s16, feature1_s16)

NeuFlow_v2/NeuFlow/matching.py

Line 19 in 204b5e3

correspondence = F.scaled_dot_product_attention(feature0, feature1, self.flatten_grid)

Then the flow is calculated as

NeuFlow_v2/NeuFlow/matching.py

Line 23 in 204b5e3

flow = correspondence - self.grid

Though I am not entirely sure, I think this represents the displacement from the source frame (i.e. the first frame) to the target frame (i.e. the second frame). Therefore, I would assume that the forward flow field is what the network estimates.

Is this correct?