Differences between torch.compile(model) and torch.compile(model.train_step)

[gaopinghai]

I am trying the torch.compile() feature. I notice that the mmengine use this function as below:

func = getattr(self.model, target)
compiled_func = torch.compile(func, **compile_cfg)
setattr(self.model, target, compiled_func)

where traget = "train_step". But this way I got the error below.

However, no error or warning comes out if I use self.model = torch.compile(self.model, **compile_cfg). But the running speed has no difference. What is wrong?

The error I got with the original code is:

[2023-04-11 11:24:46,368] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,369] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,370] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,371] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,371] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,372] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,372] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,373] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,373] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,374] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,374] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,375] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,375] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,376] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,376] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,377] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,378] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,378] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,379] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,379] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,380] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,380] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,381] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,381] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,382] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,382] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,383] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,383] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,384] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,384] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,385] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,385] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,386] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,387] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,387] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,388] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,388] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,389] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,389] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,390] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,390] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,391] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,391] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,392] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,392] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,393] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,393] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,394] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,394] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,395] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,395] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,396] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,397] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,397] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,398] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,398] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,399] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,399] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,400] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,400] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,401] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,401] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,402] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,402] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,403] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,403] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,404] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,404] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,405] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,406] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,406] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,407] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,407] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,408] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,408] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,409] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,409] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,410] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,410] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,411] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,411] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,412] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,412] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,413] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,413] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,414] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,415] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,415] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,416] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,416] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,417] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,417] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,418] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,418] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,419] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,419] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,420] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,420] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,421] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,421] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,422] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,422] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,423] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,423] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,424] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,425] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,425] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,426] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,426] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,427] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,427] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,428] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,428] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,429] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,429] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,430] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,430] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,431] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,431] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,432] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,432] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,433] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,433] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,434] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,434] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,436] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,436] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,437] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,437] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,438] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,438] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,439] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,439] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,440] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,440] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,441] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,441] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,442] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,442] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,443] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,443] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,444] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,445] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,445] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,446] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,446] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,447] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,447] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,448] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,448] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,449] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,449] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,450] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,450] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,451] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,451] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,452] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,452] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,453] torch._inductor.utils: [WARNING] DeviceCopy in input program
[2023-04-11 11:24:46,453] torch._inductor.utils: [WARNING] DeviceCopy in input program
Traceback (most recent call last):
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 358, in _compile
    out_code = transform_code_object(code, transform)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 683, in transform_code_object
    transformations(instructions, code_options)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 345, in transform
    tracer.run()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1890, in run
    super().run()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 609, in run
    and self.step()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 569, in step
    getattr(self, inst.opname)(inst)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 351, in wrapper
    return inner_fn(self, inst)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1020, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 500, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 141, in call_function
    return var.add_options(var.call_method(tx, "__init__", args, kwargs))
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 711, in call_method
    return super().call_method(tx, name, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 262, in call_method
    ).call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 297, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 265, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 98, in call_function
    return tx.inline_user_function_return(
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 536, in inline_user_function_return
    result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1994, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2060, in inline_call_
    tracer.run()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 609, in run
    and self.step()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 569, in step
    getattr(self, inst.opname)(inst)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 150, in impl
    self.push(fn_var.call_function(self, self.popn(nargs), {}))
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 550, in call_function
    res = binop_handler(tx, args[0], args[1], options)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 236, in user_defined_inplace_handler
    return a.call_method(tx, forward_name, [b], {})
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 711, in call_method
    return super().call_method(tx, name, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 262, in call_method
    ).call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 297, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 265, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 98, in call_function
    return tx.inline_user_function_return(
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 536, in inline_user_function_return
    result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1994, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2060, in inline_call_
    tracer.run()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 609, in run
    and self.step()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 569, in step
    getattr(self, inst.opname)(inst)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 351, in wrapper
    return inner_fn(self, inst)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1020, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 500, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 389, in call_function
    return self.obj.call_method(tx, self.name, args, kwargs).add_options(self)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 711, in call_method
    return super().call_method(tx, name, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 262, in call_method
    ).call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 297, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 265, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 98, in call_function
    return tx.inline_user_function_return(
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 536, in inline_user_function_return
    result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1994, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2060, in inline_call_
    tracer.run()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 609, in run
    and self.step()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 569, in step
    getattr(self, inst.opname)(inst)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 351, in wrapper
    return inner_fn(self, inst)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1020, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 500, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 567, in call_function
    result = handler(tx, *args, **kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 848, in call_len
    return args[0].call_method(tx, "__len__", args[1:], kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 711, in call_method
    return super().call_method(tx, name, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 262, in call_method
    ).call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 297, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 265, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 98, in call_function
    return tx.inline_user_function_return(
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 536, in inline_user_function_return
    result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1994, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2060, in inline_call_
    tracer.run()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 609, in run
    and self.step()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 569, in step
    getattr(self, inst.opname)(inst)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 351, in wrapper
    return inner_fn(self, inst)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1020, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 500, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 567, in call_function
    result = handler(tx, *args, **kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 848, in call_len
    return args[0].call_method(tx, "__len__", args[1:], kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/variables/constant.py", line 129, in call_method
    return ConstantVariable(len(self.value), **options)
TypeError: object of type 'NoneType' has no len()

from user code:
   File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/nn/modules/container.py", line 400, in extend
    offset = len(self)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/nn/modules/container.py", line 313, in __len__
    return len(self._modules)

Set torch._dynamo.config.verbose=True or TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gph/Desktop/e2vid_seq/train_e2vid_seq.py", line 131, in <module>
    main()
  File "/home/gph/Desktop/e2vid_seq/train_e2vid_seq.py", line 127, in main
    runner.train()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1706, in train
    model = self.train_loop.run()  # type: ignore
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/mmengine/runner/loops.py", line 96, in run
    self.run_epoch()
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/mmengine/runner/loops.py", line 112, in run_epoch
    self.run_iter(idx, data_batch)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/mmengine/runner/loops.py", line 128, in run_iter
    outputs = self.runner.model.train_step(
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 247, in _fn
    return fn(*args, **kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/mmengine/model/base_model/base_model.py", line 112, in train_step
    with optim_wrapper.optim_context(self):
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/mmengine/model/base_model/base_model.py", line 114, in <resume in train_step>
    losses = self._run_forward(data, mode='loss')  # type: ignore
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/mmengine/model/base_model/base_model.py", line 326, in _run_forward
    results = self(**data, mode=mode)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/gph/Desktop/e2vid_seq/model/e2vid_seq/e2vid_seq.py", line 36, in forward
    self.vis = E2vidVis.get_current_instance()
  File "/home/gph/Desktop/e2vid_seq/model/e2vid_seq/e2vid_seq.py", line 37, in <resume in forward>
    self.vis.update_iter()
  File "/home/gph/Desktop/e2vid_seq/model/e2vid_seq/e2vid_seq.py", line 40, in <resume in forward>
    = self.generator(inputs, self.vis.to_save_vid(), out_preds=False,
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/gph/Desktop/e2vid_seq/model/e2vid_seq/e2vid_seq_net.py", line 47, in forward
    head_seqs, backward_feat_seqs = self.backward_module(input_seqs, cpu_cache_length)
  File "/home/gph/Desktop/e2vid_seq/model/e2vid_seq/e2vid_seq_net.py", line 49, in <resume in forward>
    self.forward_module(input_seqs, head_seqs, backward_feat_seqs,
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/gph/Desktop/e2vid_seq/model/e2vid_seq/e2vid_seq_net_fattn.py", line 124, in forward
    losses = defaultdict(list)
  File "/home/gph/Desktop/e2vid_seq/model/e2vid_seq/e2vid_seq_net_fattn.py", line 143, in <resume in forward>
    zip(self.fusioners[::-1], feats, feats_backward)]
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 394, in catch_errors
    return callback(frame, cache_size, hooks)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 446, in _convert_frame
    result = inner_convert(frame, cache_size, hooks)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 113, in _fn
    return fn(*args, **kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 293, in _convert_frame_assert
    return _compile(
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 169, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/gph/anaconda3/envs/py310_torch200/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 436, in _compile
    raise InternalTorchDynamoError() from e
torch._dynamo.exc.InternalTorchDynamoError

[thfylsty]

Me too~