Plotting convergence of alpha weights in One-Shot Optimizers #154

jr2021 · 2023-01-30T08:40:42Z

Finalized Lukas' idea for heat map to visualize the convergence of alpha weights for One-Shot optimizers, via two new configuration parameters config.save_arch_weights and config.plot_arch_weights

Plotting was made to be extensible to larger search spaces by limiting the number of edges to 4, but this could also be parameterized if the user wants to be able to control this too.

…lpha_plotting

Neonkraft · 2023-02-01T10:19:41Z

naslib/defaults/trainer.py

-        # logger.info("param size = %fMB", n_parameters)
-        self.search_trajectory = utils.AttrDict(
+        logger.info("param size = %fMB", n_parameters)
+        self.errors_dict = utils.AttrDict(


I would keep the name as "search_trajectory", since it also contains non-error metrics such as accuracies, train_time and params

Neonkraft · 2023-02-01T10:21:44Z

naslib/defaults/trainer.py

@@ -453,8 +488,6 @@ def evaluate(
                )
            )

-            return top1.avg


Why has this line been removed?

Neonkraft · 2023-02-01T10:21:54Z

naslib/defaults/trainer.py

@@ -293,7 +330,6 @@ def evaluate(
                metric=metric, dataset=self.config.dataset, dataset_api=dataset_api
            )
            logger.info("Queried results ({}): {}".format(metric, result))
-            return result


Why has this line been removed?

Neonkraft · 2023-02-01T10:22:50Z

naslib/defaults/trainer.py

                epoch,
                self.train_top1.avg,
+                self.train_top5.avg,


This is useful, but it doesn't belong with the viz PR

Neonkraft · 2023-02-01T10:25:29Z

naslib/defaults/trainer.py

@@ -83,7 +88,7 @@ def search(self, resume_from="", summary_writer=None, after_epoch: Callable[[int
            resume_from (str): Checkpoint file to resume from. If not given then
                train from scratch.
        """
-        logger.info("Beginning search")
+        logger.info("Start training")


Try and keep the changes in a PR relevant to the functionality it addresses. Reversing the commits in a PR should only affect the functionality it introduces.

Neonkraft · 2023-02-01T10:29:50Z

naslib/defaults/trainer.py

            start_time = time.time()
            self.optimizer.new_epoch(e)

+            arch_weights_lst = []


Generally, as a programming practice, don't mention the data-structure in the name of the variable. arch_weights.append(...) makes it clear that it is a list, not a dict.

Neonkraft · 2023-02-01T10:31:24Z

naslib/defaults/trainer.py

@@ -108,11 +113,26 @@ def search(self, resume_from="", summary_writer=None, after_epoch: Callable[[int

        for e in range(start_epoch, self.epochs):

+            # create the arch directory (without overwriting)
+            if self.config.save_arch_weights: 


It is better for readability to write this as if self.config.save_arch_weights is True

Neonkraft · 2023-02-01T10:32:58Z

naslib/defaults/trainer.py

@@ -284,7 +321,7 @@ def evaluate(
            self._setup_checkpointers(search_model)  # required to load the architecture

            best_arch = self.optimizer.get_final_architecture()
-        logger.info(f"Final architecture hash: {best_arch.get_hash()}")
+        logger.info("Final architecture:\n" + best_arch.modules_str())


Does not belong in this PR

Neonkraft · 2023-02-01T10:39:16Z

naslib/utils/utils.py

@@ -322,8 +322,8 @@ def get_train_val_loaders(config, mode="train"):
    data = config.data
    dataset = config.dataset
    seed = config.search.seed
-    batch_size = config.batch_size
-    train_portion = config.train_portion
+    batch_size = config.batch_size if hasattr(config, "batch_size") else config.search.batch_size


This looks like a bug fix independent of the visualization code. If so, create a new PR, or push the fix directly to Develop.

Neonkraft · 2023-02-01T10:58:40Z

naslib/defaults/trainer.py

@@ -8,12 +8,17 @@
 import torch
 import numpy as np

+import matplotlib.pyplot as plt
+import seaborn as sns


Neither matplotlib.pyplot nor seaborn are used in this file. Remove.

Seaborn is missing in requirements.txt.

Neonkraft · 2023-02-01T11:03:02Z

examples/plot_save_arch_weights.py

+from naslib.optimizers import DARTSOptimizer, GDASOptimizer, DrNASOptimizer
+from naslib.search_spaces import NasBench101SearchSpace, NasBench201SearchSpace, NasBench301SearchSpace
+
+from naslib.utils import set_seed, setup_logger, get_config_from_args, create_exp_dir


I tried running this file and it crashed because create_exp_dir is not imported in the __init__.py of utils

Neonkraft · 2023-02-01T11:09:27Z

naslib/defaults/trainer.py

@@ -198,6 +215,13 @@ def search(self, resume_from="", summary_writer=None, after_epoch: Callable[[int
            if after_epoch is not None:
                after_epoch(e)

+        # save and possibly plot architectural weights
+        logger.info(f"Saving architectural weight tensors: {self.config.save}/arch_weights.pt")
+        if hasattr(self.config, "save_arch_weights") and self.config.save_arch_weights:


self.config.save_arch_weights is True for better readability

Neonkraft · 2023-02-01T11:10:42Z

naslib/defaults/trainer.py

+        logger.info(f"Saving architectural weight tensors: {self.config.save}/arch_weights.pt")
+        if hasattr(self.config, "save_arch_weights") and self.config.save_arch_weights:
+            torch.save(arch_weights, f'{self.config.save}/arch_weights.pt')
+            if hasattr(self.config, "plot_arch_weights") and self.config.plot_arch_weights:


self.config.plot_arch_weights is True

Neonkraft · 2023-02-01T11:12:05Z

naslib/utils/vis/utils.py

+    all_weights = torch.load(f'{config.save}/arch_weights.pt') # load alphas
+
+    # unpack search space information
+    alpha_dict = {}


Avoid data-structure name in var name

Neonkraft · 2023-02-01T11:29:30Z

naslib/utils/vis/utils.py

+import numpy as np
+
+import matplotlib.pyplot as plt
+from matplotlib.cm import ScalarMappable


Unused import

Neonkraft · 2023-02-01T14:51:17Z

naslib/utils/vis/utils.py

+    alpha_dict = {}
+    min_soft, max_soft = np.inf, -np.inf
+    for graph in optimizer.graph._get_child_graphs(single_instances=True):
+        for edge_weights, (u, v, edge_data) in zip(all_weights, graph.edges.data()):


There's a bug here. all_weights is a list of size 28 (in case of nb301), each a tensor of size (n_steps, n_operations). The first 14 are from the normal cells, while the next 14 are from the reduction cells. The loop assigns the same alphas for both normal and reduction cells in line 29.

Neonkraft

Please address the comments :)

There seems to be a small bug in the plotting code.

John Robertson added 5 commits December 13, 2022 14:45

Merge branch 'Develop' of https://github.com/automl/NASLib into Dev_A…

77e6191

…lpha_plotting

modifed Lukas' code for alpha convergence plotting

d6fe57a

initial changes for alpha plotting procedure

a7fba22

one shot model alpha weights visualization

6ca93d9

divided plots into sets of 4 heatmaps

d3d8300

jr2021 requested review from arberzela, yashsmehta and crwhite14 as code owners January 30, 2023 08:40

jr2021 changed the title ~~Dev alpha plotting~~ Plotting convergence of alpha weights in One-Shot Optimizers Jan 30, 2023

Neonkraft reviewed Feb 1, 2023

View reviewed changes

naslib/defaults/trainer.py

@@ -453,8 +488,6 @@ def evaluate(

)

)

return top1.avg

Copy link

Collaborator

Neonkraft Feb 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why has this line been removed?

Neonkraft reviewed Feb 1, 2023

View reviewed changes

John Robertson added 2 commits February 2, 2023 11:14

resolved discrepencies unrelated to pull request and added trainer.py

0082349

added utils change and vis file to commit

3d689cf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plotting convergence of alpha weights in One-Shot Optimizers #154

Plotting convergence of alpha weights in One-Shot Optimizers #154

jr2021 commented Jan 30, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft Feb 1, 2023

Neonkraft left a comment

Plotting convergence of alpha weights in One-Shot Optimizers #154

Are you sure you want to change the base?

Plotting convergence of alpha weights in One-Shot Optimizers #154

Conversation

jr2021 commented Jan 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Neonkraft left a comment

Choose a reason for hiding this comment