Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[onert-micro] Introduce TrainingConfigureTool #13684

Open
BalyshevArtem opened this issue Aug 16, 2024 · 2 comments
Open

[onert-micro] Introduce TrainingConfigureTool #13684

BalyshevArtem opened this issue Aug 16, 2024 · 2 comments

Comments

@BalyshevArtem
Copy link
Contributor

Let's introduce training configure tool. This tool should find a configuration for network training with some optimal memory costs.
The Training Tool will be able to perform actions to improve the training process:

  • based on a given memory budget, select only a part of the layers for training, or even a part of the weights of a certain layer - this process called sparse backpropagation.
  • applications of mixed precision quantization, search for parts of the network for which the materialization technique will be performed (when intermediate results are not saved for some part of the network, but they are recalculated in the backpropagation process)
  • optimizations on the graph itself
  • and so on

Related issue: #12873

@BalyshevArtem
Copy link
Contributor Author

draft - #13479

@BalyshevArtem
Copy link
Contributor Author

Result of the tool from #13479 for our target model:
Default training - training all network, all operations, and all tensors:

CROSS_ENTROPY_METRIC_RESULT = 0.187580
PEAK_MEMORY_ESTIMATION_RESULT = 148 624 bytes

Training with training config file, produced by training configure tool:

CROSS_ENTROPY_METRIC_RESULT = 0.187278
PEAK_MEMORY_ESTIMATION_RESULT = 62 160 bytes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant