Storing of models or architectures in optimizers

[jr2021]

In the get_candidates function of the zerocost branch optimizers Bananas and Npenas there is a discrepancy between how candidates in the next_batch are stored.

If the acquisition function is being optimized via "random_sampling", then model is being stored:

...
candidates.append(model)

Otherwise, if it is being optimized via "mutation", then model.arch is being stored:

...
    candidate = arch
candidates.append(candidate)

However, the function get_best_candidates (which is called directly after get_new_candidates) treats candidates as a list of models:

values = [acq_fn(model.arch, [{'zero_cost_scores' : model.zc_scores}]) for model in candidates]

Does this imply that the optimization of the acquisition function via "mutation" is not used in the main loop of either Bananas or Npenas? If so, how and when should this option be used?

[Neonkraft]

We used only random sampling for the optimization in the paper. You can change the mutation code to comply with the RS acquisition function.