List of all possible "tags"?

[Linardos]

In the notebook it is specified that:

There are examples of what a tag might be here but I couldn't find a comprehensive list of all the possible tags. You mention that the same 'tensor_name' and 'round' are used in such a case with the tag 'trained' to specify the weights after training. What would I use instead of "trained" to obtain the initial weights for example? Is there a list detailing all possible tags that I missed somewhere?

[sarthakpati]

Pinging @msheller @alexey-gruzdev @psfoley

[alexey-gruzdev]

@Linardos I am sorry again, looks like in original answer for some reason attachment didn't work. I would put here a screen of dataframe that I got from MNIST. Hope it helps!

[psfoley]

@Linardos to follow onto what Alexey provided, there is a flow that the tags follow in the normal training sequence. At the highest level, the

('model',) -> These are the tags given to the aggregated tensor that will be downloaded from the aggregator to the collaborator for the purpose of training or aggregated_model_validation.

('trained',) -> As you correctly pointed out, the 'trained' tag is used to denote the trained model weights coming from each collaborator. There is a transformation that happens to the tensorkey as it is sent from the collaborator to the aggregator, namely that the 'origin' of the tensorkey changes from the common name of the collaborator (let's use as example collaborator name one). The reason for this is that the tensor is decompressed once it moves to the aggregator, and then because that tensor can only be found on the aggregator, it's origin becomes aggregator. The collaborator name is then moved into the tag, which is where you see ('trained','one',) in Alexey's image above.

('aggregated',) -> This tag is the result of applying a built-in or custom aggregation scheme to the collaborator model weights. It should be noted that this is not the final version that will be downloaded by the collaborator at the start of the next round. This ('aggregated',) tensor then undergoes compression and decompression, at which point the tensor is then relabeled ('model,'). The reason for this is to guarantee that the version of the model downloaded by the collaborator is the same as what is stored on the aggregator (i.e. for lossy compression of model weights, which is not used in this competition)

(*,'delta',) -> The delta` tag is an option provided within OpenFL to only send the difference between model weights between the aggregator and collaborator. This can be very useful for lossy compression. This flag is not enabled for the competition, so you don't need to worry about this tag.

[GorkemP]

Dear organizers,

I am still confused about tags and it prevents us implementing our method. When I inspect the details of db_iterator, I see that for any layer, value of (trained,) at round x-1 equals to value of (aggregated,) at round x. So, it seems to me that after aggragating the models coming from collaborators, resulting aggregated model layer/weights are tagged as (trained,) (for that round x-1) and (aggregated,) is the model distributed to collaborators at the start of the next round. For example, at round 3, if I want to get model weights that are distributed to collaborators, I think I should look for (aggregated,) at round 3 not round 2. But, example codes contradict with this, as @Linardos shared above (as screen-shot), previous round is fl_round-1 not fl_round. What am I missing?

[psfoley]

@GorkemP, I'm sorry for the confusion. Your understanding is correct. The example @Linardos posted above is the way that you could resolve the prior round's aggregated tensor.

At the point when you have access to the db_iterator, the aggregated tensors have not been generated for the current round. If it is round 1 for example, you would be able to see each of the ('trained',*) tensors produced by each of the collaborators performing the training task for round1 , but the output of your custom aggregation function will produce the ('aggregated',) tensor for round 2. Using the ('aggregated',) tensor for round 1 would be the layer value the collaborator would have started with for the current round prior to training. Does this make sense?

[GorkemP]

@psfoley Thank you for the answer. It totally makes sense, we can only get aggregated result in the next round (with (aggregated,) tag for round x or (trained,) tag for round x-1 ). I still think (or not understand) that this example is confusing:

Since it uses (aggregated,) tag for (fl_round-1), previos_tensor_value refers to two step before. Yet, we normally refer gradients as single steps, this is the confusing part for me. In our aggregation method, we should calculate gradients for each collaborator with respect to the last aggregated model (we should measure how much each collaborator training changed the weights), in our case I believe that I should just change the code from fl_round-1 to fl_round in the above example.