[ RMSProp ] add rms prop optimizer #2587
Conversation
This PR implement rms prop optimizer. unittest for rms prop is required. . emplementation of RMSProp Class . Seperate the optimizer properties into optimizer common properties. and some fixes. . add enum for rms props **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: jijoong.moon <[email protected]>
jijoongmoon
requested review from
myungjoo,
again4you,
jaeyun-jung,
leemgs,
wooksong,
helloahn,
kparichay,
gichan-jang,
anyj0527,
zhoonit,
lhs8928,
songgot,
jihochu,
DonghakPark,
SeoHyungjun,
baek2sm,
skykongkong8,
djeong20,
EunjuYang and
a team
as code owners
|
📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2587. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/. |
|
|
cibot: 
| Tensor denom = wv.apply<float>(sqrtFloat<float>); | ||
| denom.add_i(epsilon); | ||
|
|
||
| wv.divide(denom, x_grad); |
There was a problem hiding this comment.
It seems to compute
W = W - alpha * wv / (sqrt(wv) + epsilon), which should be
W = W - alpha * x_grad / (sqrt(wv) + epsilon) so as to work properly.
| wv.divide(denom, x_grad); | |
| x_grad.divide_i(denom); |
There was a problem hiding this comment.
It seems eunju is right.
Further more unlike the adam optimizer torch seems to work in the same way as other frameworks do.
Please refer https://pytorch.org/docs/stable/generated/torch.optim.RMSprop.html
| if (opt && istrequal(opt->getType(), "rmsprop")) { | ||
| std::string rmsprop = "rmsprop"; | ||
| model_file.write(rmsprop.c_str(), 7); | ||
| for (auto iter = model_graph.cbegin(); iter != model_graph.cend(); |
There was a problem hiding this comment.
- Saving rms prop optimizer variable code seems to be integrated with that of adam optimizer
- Needs to handle the load rms prop optimizer variable
| @@ -85,6 +85,7 @@ typedef enum { | |||
| typedef enum { | |||
| ML_TRAIN_OPTIMIZER_TYPE_ADAM = 0, /**< Adam Optimizer */ | |||
| ML_TRAIN_OPTIMIZER_TYPE_SGD = 1, /**< Stochastic Gradient Descent Optimizer */ | |||
| ML_TRAIN_OPTIMIZER_TYPE_RMSPROP = 2, /** rmsprop */ | |||
There was a problem hiding this comment.
Please match the format of other optimizer description.
rmsprop -> RMSProps Optimizer
| class RMSProp : public Optimizer { | ||
| public: | ||
| /** | ||
| * @brief Construct a new Adam object |
| ? context.getGradient() | ||
| : empty_tensor; | ||
|
|
||
| if (x_grad.empty()) |
There was a problem hiding this comment.
It seems that it can be merged with ternary operator
| }; | ||
|
|
||
| x_grad = wv.apply<float>(sqrtEps, x_grad); | ||
| context.applyGradient(context.getLearningRate(), x_grad); |
There was a problem hiding this comment.
RumOptimizerContext does not have a function which has a prototype of applyGradient(double, tensor) as the CI issued.
| Tensor denom = wv.apply<float>(sqrtFloat<float>); | ||
| denom.add_i(epsilon); | ||
|
|
||
| wv.divide(denom, x_grad); |
There was a problem hiding this comment.
It seems eunju is right.
Further more unlike the adam optimizer torch seems to work in the same way as other frameworks do.
Please refer https://pytorch.org/docs/stable/generated/torch.optim.RMSprop.html
In this PR
This PR implements rms prop optimizer.
unittest for rms prop is required.
. implementation of RMSProp Class
. Separate the optimizer properties into optimizer common properties. and some fixes.
. add enum for rms props
Self evaluation: