Inference and loading pretrained model problems

[DaddyWesker]

Hello.

Thanks for your code. I've wanted to make py-script to infer your model instead of prlab.cli launches you've provided. So, I've simply wanted to load pre-trained weights to the model. If i got it right, I'm need to use model PyramidSRShare, but the model consists of several sub-networks like stn, classifier, pyramid_groups. You've provided edsr_baseline_x2-1bc95232.pt pre-trained file, which i've tried to load like in load_weights

for idx in range(len(self.multiples)):
     xn_name = 'weight_path_x{}'.format(self.multiples[idx])
     if self.config.get(xn_name, None) is not None:
         out = self.pyramid_group[idx].load_state_dict(torch.load(self.config[xn_name]), strict=True)
         print('load weights for {}'.format(xn_name), out)

As i understand, this x2 file should be loaded to pyramid_group[1] which was created
pyramid_group.append(self.make_layer_pyramid(mul, input_spec)) using mul = 2. But when i'm trying to load the weights, i'm getting "Unexpected keys" and "Missing keys" error. So, probably, i'\m doing something wrong.

Moreover, even if I'll be able to load properly weights to pyramid_group[1] there are still stn, classifier and other pyramid_group[0,2] which won't be initialized with pretrained weights. So, should there be some other pre-trained weights too?

[thanhhungqb]

Thank you for your interest in our publication.
The simplest way to infer from your code without prlab.cli (with weights of the final training process available for resume_learner):

from prlab.common.dl import pipeline_control_multi

conf = json.load('config/raf-db.json')
# remove all pipe that for training and report process_pipeline_10+

conf = pipeline_control_multi(**conf)
learn = config['learn']
# do whatever with learn/model (fastai), e.g. predict for one ...

There are two ways to load pre-trained weights for our model:

1. Load each sub-network separated. It seems that you were tried the first way.
   The first approach ONLY using during the training process, and then, there are only some pre-trained sub-networks provided, such as EDSR (you tried, edsr_baseline_x2-1bc95232.pt provided by EDSR authors, not our), base network (VGG16). However, some sub-networks are not provided and should be learning during the training process.

2. Load the whole network in one. This approach can be used during the training process and also in inferent times (deploy). After trained, the weights of the network are saved and can be load later.
   Yes, we have the weights for the whole network for the final results (that we reported), but we did not upload because most researchers want to train and get it by themself.
   If you want to get it (saved model network weights), please let me know.
   Please refer prlab.fastai.pipeline.resume_learner for load the whole network weights (process_pipeline_5 of config/raf-db.json).

prlab.cli is an entry point, you can easily write a py-script yourself by following the step by step of processing pipeline we defined in the JSON configure. It defines the data and process to load data, network, do the training and predict, and report. Below are steps to make py code yourself:

1. First create a DICT named $CONF with all necessary configure, the easy way is to clone from configure file, e.g. fold, path, csv_path, etc. (except process_pipeline* is for the process)
2. For each function/class call in process_pipeline_* from 0-..., you can make a call from your py-script like that:
   $CONF = func(**$CONF) # where $CONF is a dict variable at step 1

E.g., from config/raf-db.json we can make a py script to run like that, (one fold):
(need import modules, and using, bellow I were the full module path)

conf = json.load('config/raf-db.json') # using data only, process_pipeline* can be ommit

# run processes
# 0
conf = prlab.fastai.pipeline.device_setup(**conf)
conf = prlab.fastai.utils.general_configure(**conf)

# 1
conf = prlab.fastai.pipeline.exchange_fc(**conf)

# 5
conf = prlab.fastai.pipeline.data_load_folder_df(**conf)
conf = prlab.fastai.pipeline.create_obj_model(**conf)
...

In each step, e.g. in # 5 (process_pipeline_5), if you want to be more controllable for create_obj_model(...), you could follow the implement to see the params in the conf variable are using.
Note that, process_pipeline_* is just a sequence of function/object calls you need to call in your code, (keep their order).

[DaddyWesker]

Thanks for the explanations about prlab.cli since i haven't met before such library and haven't worked with that either. But first time for everything =)

About full pretrained model - it will be really great to get this model. Could you upload it?