This repository has been archived by the owner on Sep 18, 2024. It is now read-only.
NNI v2.9 Release
Neural Architecture Search
- New tutorial of model space hub and one-shot strategy. (tutorial)
- Add pretrained checkpoints to AutoFormer. (doc)
- Support loading checkpoint of a trained supernet in a subnet. (doc)
- Support view and resume of NAS experiment. (doc)
Enhancements
- Support
fit_kwargs
in lightning evaluator. (doc) - Support
drop_path
andauxiliary_loss
in NASNet. (doc) - Support gradient clipping in DARTS. (doc)
- Add
export_probs
to monitor the architecture weights. - Rewrite configure_optimizers, functions to step optimizers / schedulers, along with other hooks for simplicity, and to be compatible with latest lightning (v1.7).
- Align implementation of DifferentiableCell with DARTS official repo.
- Re-implementation of ProxylessNAS.
- Move
nni.retiarii
code-base tonni.nas
.
Bug fixes
- Fix a performance issue caused by tensor formatting in
weighted_sum
. - Fix a misuse of lambda expression in NAS-Bench-201 search space.
- Fix the gumbel temperature schedule in Gumbel DARTS.
- Fix the architecture weight sharing when sharing labels in differentiable strategies.
- Fix the memo reusing in exporting differentiable cell.
Compression
- New tutorial of pruning transformer model. (tutorial)
- Add
TorchEvaluator
,LightningEvaluator
,TransformersEvaluator
to ease the expression of training logic in pruner. (doc, API)
Enhancements
- Promote all pruner API using
Evaluator
, the old API is deprecated and will be removed in v3.0. (doc) - Greatly enlarge the set of supported operators in pruning speedup via automatic operator conversion.
- Support
lr_scheduler
in pruning by usingEvaluator
. - Support pruning NLP task in
ActivationAPoZRankPruner
andActivationMeanRankPruner
. - Add
training_steps
,regular_scale
,movement_mode
,sparse_granularity
forMovementPruner
. (doc) - Add
GroupNorm
replacement in pruning speedup. Thanks external contributor @cin-xing . - Optimize
balance
mode performance inLevelPruner
.
Bug fixes
- Fix the invalid
dependency_aware
mode in scheduled pruners. - Fix the bug where
bias
mask cannot be generated. - Fix the bug where
max_sparsity_per_layer
has no effect. - Fix
Linear
andLayerNorm
speedup replacement in NLP task. - Fix tracing
LightningModule
failed inpytorch_lightning >= 1.7.0
.
Hyper-parameter optimization
- Fix the bug that weights are not defined correctly in
adaptive_parzen_normal
of TPE.
Training service
- Fix trialConcurrency bug in K8S training service: use
${envId}_run.sh
to replacerun.sh
. - Fix upload dir bug in K8S training service: use a separate working directory for each experiment. Thanks external contributor @amznero .
Web portal
- Support dict keys in Default metric chart in the detail page.
- Show experiment error message with small popup windows in the bottom right of the page.
- Upgrade React router to v6 to fix index router issue.
- Fix the issue of details page crashing due to choices containing
None
. - Fix the issue of missing dict intermediate dropdown in comparing trials dialog.
Known issues
- Activation based pruner can not support
[batch, seq, hidden]
. - Failed trials are NOT auto-submitted when experiment is resumed (#4931 is reverted due to its pitfalls).