Decoder inputs

[LIIHWF]

Hi,

I have a question regarding the inputs for the Decoder in InterFuser.

InterFuser/interfuser/timm/models/interfuser.py

Line 1025 in e4f0314

hs = self.decoder(self.query_embed.repeat(1, bs, 1), memory, query_pos=tgt)[0]

I noticed that the decoder has three inputs. However, the first input self.query_embed and the third input tgt seem to be irrelevant to the sensor data or other contextual parameters. Do they act as placeholders whose values do not play any role?

I have this question because I found that self.query_embed is the created in __init__ method and assigned randomly by calling reset_parameter. tgt is computed according to self.query_pos_embed, and self.query_pos_embed is created and assigned similarly to self.query_embed.

After self.query_embed and self.query_pos_embed are first randomly assigned, their values do not change during the execution of the Interfuser model.

Did I miss something in the code that is actually changing the values of these inputs?
Otherwise, the randomly assigned values can affect the reproducibility of the autopilot's behavior, which is not desired in many cases.

[deepcs233]

Hi!
Sorry for the late reply. The query_embed is learnable and it can be updated during training. The tgt variable is the cosine positional encoding and is not learnable. Both two variables act as the positional encoding.