Allow region_layer_map json to be loaded into BrainModel during commitment

[KartikP]

This allows a region_layer_map dictionary to be loaded during ModelCommitment.

Region layer maps need to be in a folder in the model's directory called region_layer_map and in a file called ${MODEL_IDENTIFIER}.json.

For example, if you want to add the layer map for alexnet.

You would create a /alexnet/region_layer_map/alexnet.json file that contains the region_layer_map dictionary.

{ "V1": "features.2", "V2": "features.7", "V4": "features.7", "IT": "features.12" }

Not particularly happy with the function, but i think it gets the job done. Open to feedback.

[KartikP]

resnext101_32x8d_wsl_ still being run on helper folder changes and causing OOM. Need to remove that 😂

[deirdre-k]

Looks good! Just the one comment

[mschrimpf]

I don't like that the ModelCommitment has to know about plugins. Can we not load the specification inside the plugin and pass it to the ModelCommitment as a parameter?

[KartikP]

Modifying the __init__.py was the original approach. However, after talking with the others, it seems modifying the init file, especially when there can be multiple model identifiers listed in a single model directory could be messy. It would also make for a messier PR/commit during the model submission.

By adding a check in ModelCommitment, all that would be needed would be a single region_layer_map json per model_identifier in the model directory. But I do agree that it disrupts the current flow of the plugin model.

If we do not want to modify the __init__.py file of each model (and therefore cannot pass the map to ModelCommitment as a parameter), what would you recommend?

[mschrimpf]

just thinking out loud, but what about an intermediate approach where the ModelCommitment expects the full layer mapping, but submissions can specify this to be automatically discovered? Something like

model_registry['mymodel'] = lambda: ModelCommitment(
    identifier='mymodel',
    activations_model=get_model(),
    region_layer_map=discover_region_layer_mapping('mymodel'))

The discover_region_layer_mapping would then first look for a mapping file in the plugin, and if that doesn't exist make one by running the layer selection. This way the plugin __init__ would not have to be modified but we still maintain the separation between plugins and model helpers.

[the--lynx]

Martin - Kartik and I just discussed and I would like to propose something a bit more direct that what you suggested. We create a means to determine if we are running locally (any place but AWS) or in production (AWS). For local runs (again, anything but AWS) if no json file exists, we perform layer mapping and then write the json file.
For AWS runs we assume that region_layer_map json file exists and throw an exception if it does not, terminating the scoring run.

This is changing the assumptions of modelCommitment, which hopefully we don't do frequently, but not significantly (and not even as significantly as the solution you propose above). Additionally, as a benefit, local users, at the time of their PR already have the layer map and have thought about it and checked it against reality. They can regenerate as many times as desired, simply by removing the json file.

[mschrimpf]

yes I think we agree on those points.

What I'm not sure about is if you want this detection to happen inside ModelCommitment or in the plugin. I am arguing for the latter so that we keep the clean decoupling between the plugin system and model helpers. The only advantage I see for putting it inside the ModelCommitment is to avoid the model plugins having to specify layer discovery, but I like that it makes this explicit to users. And the downside of having it inside of ModelCommitment is that it makes adapting this mapping more difficult for models such as CORnet and VOneNet.

[KartikP]

Write local JSON file to the model directory. This can be PR'd when they choose to submit.

[mschrimpf]

Proposal in order to move this forward: you go ahead with this plan, we merge, and I submit another PR that implements what I have in mind.