How did you set the random seed of Table 1.

[KyominHwang]

Hi,

I'm reproducing your result in Table 1 with the hyper-parameter reported in the shared CSV file in google drive.
Now I train all models in A100 GPU, but there exists a small gap between the reported performance.

1. So, I'm curious about how you set the random seed in Table 1.

And also, when I ran vtab-svhn dataset with random seed [42, 44, 82, 100, 800], I got unstable top1 results as follows:

seed42 : 79.78
seed44 : 65.91
seed82 : 80.37
seed100 : 81.75
seed800 : 81.47

and the command is as follows:

11 for seed in "42" "44" "82" "100" "800"; do
12 CUDA_VISIBLE_DEVICES=3 python3 train.py
13 --config-file configs/prompt/cub.yaml
14 MODEL.TYPE "vit"
15 DATA.BATCH_SIZE "128"
16 MODEL.PROMPT.NUM_TOKENS "50"
17 MODEL.PROMPT.DEEP "True"
18 MODEL.PROMPT.DROPOUT "0.1"
19 DATA.FEATURE "sup_vitb16_imagenet21k"
20 DATA.NAME 'vtab-svhn'
21 DATA.NUMBER_CLASSES "10"
22 SOLVER.BASE_LR "1.25"
23 SOLVER.WEIGHT_DECAY "0.0"
24 SEED ${seed}
25 MODEL.MODEL_ROOT "./weights"
26 DATA.DATAPATH "./dataset"
27 OUTPUT_DIR "output/seed${seed}"
28 done

2. So, I'm curious about does the visual prompt tuning is sensitive to the hyper-parameter(e.g. random seed). Or, does the command wrong?

[KyominHwang]

Or, can you share the per-task results of the Fig. 15.-Average(left most figure)?