Usage report of the efficient_conv_bn_eval feature #1252
Replies: 6 comments 21 replies
-
|
I used the maskrcnn model as a baseline to test the difference of turning on this function. When the model first started training, it could indeed save 13% of the memory. But as the training continued, I used nvidia-smi to check the memory. I found that the memory continued to increase from 7GB. It soared until it occupied 24GB, but the memory usage in the training log record remained as low as it was at the beginning |
Beta Was this translation helpful? Give feedback.
-
|
Hi, numbers shown in |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your prompt reply, I'll check again! |
Beta Was this translation helpful? Give feedback.
-
|
@buxuewushu1314 for example, please check the |
Beta Was this translation helpful? Give feedback.
-
|
Train resnet18 on mmpretrain-v1.0.2 using '-- cfg options effective'_ Conv_ Bn_ After eval="[backbone]" ', there is almost no savings memory. Do I need to specify a version for mmretrain, or do I need any other settings?
|
Beta Was this translation helpful? Give feedback.
-
|
Can you show your full training config? e.g. which task you are training on, and the config file you are using. In general, pre-training uses BN in train mode, where |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your reply，yeah，it should be train mode，so does mmpretrain provide an interface for setting BN in eval mode, how should to do? |
Beta Was this translation helpful? Give feedback.
-
|
Generally we don't pretrain in eval mode. Pretraining requires train mode. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you again. classification model was still effective during fine-tuning by setting 'norm_eval', testing shows that it can save 33.8% memory,good job! |
Beta Was this translation helpful? Give feedback.
-
|
Glad it helps :) |
Beta Was this translation helpful? Give feedback.
-
Hello, I have set "norm_val", but there is an error：torch.fx.proxy.TraceError: symbolically traced variables cannot be used as inputs to control flow,how should to do? |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/Jano-rm-rf/Test/blob/main/DINO-Test6-Config.py |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
This is not a problem caused by version incompatibility. This is due to the current implementation of efficient_conv_ bn_eval feature is based on the symbolic trace in torch.fx. Due to the limited ability of symbolic trace, it is not possible to handle expressions that are only known at runtime or depend on parameters such as values or shapes of input. In the future, we will use torch.compile to implement this feature, and there will be no such issue at that time. If you want to learn more about symbolic trace, you can check this link https://pytorch.org/docs/stable/fx.html#limitations-of-symbolic-tracing or https://zhuanlan.zhihu.com/p/644590863. |
Beta Was this translation helpful? Give feedback.
-
|
The specific reason for this issue is that your config uses eficientnetv2, and its initialization specifies the conv type as Conv2dAdaptivePadding (https://github.com/open-mmlab/mmpretrain/blob/main/mmpretrain/models/backbones/efficientnet_v2.py#L186). The forward function of Conv2dAdaptivePadding contains condition judgment of pad_h calculated based on x.size() (https://github.com/open-mmlab/mmcv/blob/main/mmcv/cnn/bricks/conv2d_adaptive_padding.py#L58). |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your reply, I will modify conv type and test it |
Beta Was this translation helpful? Give feedback.
-
|
Can this method still work if the batch_size is set to 1 during training? |
Beta Was this translation helpful? Give feedback.
-
|
As long as your BN is in eval mode during training, this method should work. Nevertheless, smaller batchsize may lead to less benefit in memory reduction. |
Beta Was this translation helpful? Give feedback.
-
|
I understand, thanks for your reply! |
Beta Was this translation helpful? Give feedback.
-
|
Tested on the SlowOnly model (
|
Beta Was this translation helpful? Give feedback.
-
|
Glad it helps :) |
Beta Was this translation helpful? Give feedback.
-
|
I have encoutered a question, please help. ./tools/dist_train.sh bisai.py 1 --cfg-options efficient_conv_bn_eval="[backbone]"my dist_train.sh #!/usr/bin/env bash
CONFIG=$1
GPUS=$2
NNODES=${NNODES:-1}
NODE_RANK=${NODE_RANK:-0}
PORT=${PORT:-29568}
MASTER_ADDR=${MASTER_ADDR:-"127.0.0.1"}
PYTHONPATH="$(dirname $0)/..":$PYTHONPATH \
CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch \
--nnodes=$NNODES \
--node_rank=$NODE_RANK \
--master_addr=$MASTER_ADDR \
--nproc_per_node=$GPUS \
--master_port=$PORT \
$(dirname "$0")/train.py \
$CONFIG \
--launcher pytorch ${@:3}
Then I have the following exception: I checked a lot of material and ask GPT, but got no answer. Please do help and thx in advance. |
Beta Was this translation helpful? Give feedback.
-
this model might contain certain conv that cannot be traced by |
Beta Was this translation helpful? Give feedback.
-
|
if your code supports from torch._inductor import config as inductor_config
inductor_config.efficient_conv_bn_eval_fx_passes = True |
Beta Was this translation helpful? Give feedback.
-
Please report here how enabling
efficient_conv_bn_evalfeature helps your training. We expect the feature to seamlessly reduce memory footprint without affecting the performance. Currently it is an optional feature, but if the feedback is good enough, we can turn this feature into default :)Beta Was this translation helpful? Give feedback.
All reactions