[E2E] Yolo3 will failed with pytorch pinned torchbench for XPU lost

[mengfei25]

🐛 Describe the bug

python benchmarks/dynamo/torchbench.py --accuracy --bfloat16 -d xpu -n10 --training --only yolov3 --backend=inductor

Traceback (most recent call last):
File "/home/sdp/actions-runner/_work/torch-xpu-ops/pytorch/benchmarks/dynamo/common.py", line 4886, in run
) = runner.load_model(
File "/home/sdp/actions-runner/_work/torch-xpu-ops/pytorch/benchmarks/dynamo/torchbench.py", line 312, in load_model
benchmark = benchmark_cls(
File "/home/sdp/actions-runner/_work/torch-xpu-ops/benchmark/torchbenchmark/util/model.py", line 24, in call
obj = type.call(cls, *args, **kwargs)
File "/home/sdp/actions-runner/_work/torch-xpu-ops/benchmark/torchbenchmark/models/yolov3/init.py", line 58, in init
self.training_loop, self.model, self.example_inputs = prepare_training_loop(
File "/home/sdp/actions-runner/_work/torch-xpu-ops/benchmark/torchbenchmark/models/yolov3/yolo_train.py", line 630, in prepare_training_loop
device = torch_utils.select_device(
File "/home/sdp/actions-runner/_work/torch-xpu-ops/benchmark/torchbenchmark/models/yolov3/yolo_utils/torch_utils.py", line 31, in select_device
assert torch.cuda.is_available(), 'CUDA unavailable, invalid device %s requested' % device # check availablity
AssertionError: CUDA unavailable, invalid device xpu requested

https://github.com/pytorch/benchmark/blob/766a5e3a189384659fd35a68c3b17b88c761aaac/torchbenchmark/models/yolov3/yolo_utils/torch_utils.py#L28-L31

def select_device(device='', apex=False, batch_size=None):
    # device = 'cpu' or '0' or '0,1,2,3'
    cpu_request = device.lower() == 'cpu'
    if device and not cpu_request:  # if device requested other than 'cpu'
        os.environ['CUDA_VISIBLE_DEVICES'] = device  # set environment variable
        assert torch.cuda.is_available(), 'CUDA unavailable, invalid device %s requested' % device  # check availablity

Versions

Torchbench: 766a5e3a189384659fd35a68c3b17b88c761aaac

[mengfei25]

Skipped in CI & Nightly test #1230