Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Question] Encountering Unknown Error: [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false. #1409

Open
H-Hisamichi opened this issue Nov 11, 2024 · 6 comments
Open

[Question] Encountering Unknown Error: [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false. #1409

H-Hisamichi opened this issue Nov 11, 2024 · 6 comments
Labels
question Further information is requested

Comments

@H-Hisamichi
Copy link

H-Hisamichi commented Nov 11, 2024 •
edited
Loading

Hello everyone,

I encountered an unknown error while training my original Hexapod using reinforcement learning.

First, I get this warning: [Warning] [omni.ujitso] UJITSO: Build storage validation failed for Storage status: 1, and then the error [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false. appears.
The error occurred after the log [INFO]: Starting the simulation. This may take a few seconds. Please wait....

Sometimes this error causes a buffer error, but other times the training continues without any issues. I couldn't find any information about this error.
Does anyone know what this means?

Thank you!
@RandomOakForest
Copy link
Collaborator

Could you please confirm the version of Isaac Lab you are using? Have you tried this in the 1.2 release? Thanks!

@RandomOakForest RandomOakForest added the question Further information is requested label Nov 12, 2024
@H-Hisamichi
Copy link
Author

Hello @RandomOakForest ,

I'm using Isaac Lab v1.2.0 and Isaac Sim v4.2.0.
Thanks again!

@RandomOakForest
Copy link
Collaborator

Thank you for submitting this. We have take note of this error. If it is causing a crash on your side, please submit a bug report. Thanks.

@H-Hisamichi
Copy link
Author

Hello @RandomOakForest,

Thank you for your response.
The current error code does not cause a crash. However, I am not sure if this error affects policy.
I will provide more information about my environment.
Isaac Sim: 4.2.0
Isaac Lab: 1.2.0
RL code: based on anymal_c(direct), modified for original hexapod.

The display of error codes has also been changed. The cause is unknown.
Example of error code pattern 1:

[INFO] Using python from: /home/robot/anaconda3/envs/iccar2025/bin/python
[INFO][AppLauncher]: Loading experience file: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/apps/isaaclab.python.headless.rendering.kit
Loading user config located at: '/home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/data/Kit/Isaac-Sim/4.2/user.config.json'
[Info] [carb] Logging to file: /home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/logs/Kit/Isaac-Sim/4.2/kit_20241117_101409.log
2024-11-17 01:14:10 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2024-11-17 01:14:10 [673ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:10 [673ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:10 [701ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.

|---------------------------------------------------------------------------------------------|
| Driver Version: 535.183.01    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA RTX A4000                 | Yes: 0 |     | 16622   MB | 10de      | 0          |
|     |                                  |        |     |            | 24b0      | 44ccd3c0.. |
|     |                                  |        |     |            | 6         |            |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 6.8.0-48-generic
| Processor: AMD Ryzen 7 2700 Eight-Core Processor | Cores: 8 | Logical: 16
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 48078 | Free Memory: 41052
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
2024-11-17 01:14:11 [1,319ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2024-11-17 01:14:12 [2,183ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,183ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,193ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,193ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,201ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,201ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,210ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,210ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,212ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,212ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:12 [2,213ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 01:14:12 [2,213ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 01:14:15 [5,363ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
formatted_tuple: (0.5, 0.5, 0.5, 0.5)
[INFO]: Parsing configuration from: omni.isaac.lab_tasks.direct.AT3R.AT3R_rolling_env:AT3RRoughEnvCfg
[INFO]: Parsing configuration from: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/AT3R/agents/rl_games_rough_ppo_cfg.yaml
[INFO] Logging experiment in directory: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct
Setting seed: 42
[INFO]: Base environment:
    Environment device    : cuda:0
    Environment seed      : 42
    Physics step-size     : 0.005
    Rendering step-size   : 0.02
    Environment step-size : 0.02
[INFO] Generating terrains based on curriculum took : 1.656922 seconds
[INFO]: Time taken for scene creation : 4.838669 seconds
[INFO]: Scene manager:  <class InteractiveScene>
    Number of environments: 1024
    Environment spacing   : 4.0
    Source prim name      : /World/envs/env_0
    Global prim paths     : ['/World/ground']
    Replicate physics     : True
[INFO]: Starting the simulation. This may take a few seconds. Please wait...
2024-11-17 01:14:30 [20,747ms] [Warning] [omni.ujitso] UJITSO : Build storage validation failed for  Storage status: 1
2024-11-17 01:14:30 [20,747ms] [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false.
2024-11-17 01:14:30 [20,749ms] [Warning] [omni.ujitso] UJITSO : Build storage validation failed for  Storage status: 1
2024-11-17 01:14:30 [20,749ms] [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false.
[INFO]: Time taken for simulation start : 57.431758 seconds
[INFO] Event Manager:  <EventManager> contains 1 active terms.
+--------------------------------------+
| Active Event Terms in Mode: 'startup' |
+----------+---------------------------+
|  Index   | Name                      |
+----------+---------------------------+
|    0     | physics_material          |
|    1     | add_base_mass             |
+----------+---------------------------+

[INFO]: Completed setting up the environment...
[INFO] Recording videos during training.
        video_folder: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct/2024-11-17_10-14-20/videos/train
        step_trigger: lambda step: step % args_cli.video_interval == 0
        video_length: 400
        disable_logger: True
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/gymnasium/core.py:297: UserWarning: WARN: env.is_vector_env to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.is_vector_env` for environment variables or `env.get_attr('is_vector_env')` that will search the reminding wrappers.
  logger.warn(
self.seed = 42
Started to train
Exact experiment name requested from command line: 2024-11-17_10-14-20
seq_length: 4
current training device: cuda:0
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/common/a2c_common.py:254: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
  self.scaler = torch.cuda.amp.GradScaler(enabled=self.mixed_precision)
build mlp: 252
RunningMeanStd:  (1,)
2024-11-17 01:15:27 [77,183ms] [Warning] [omni.syntheticdata.plugin] OgnSdPostRenderVarToHost : rendervar copy from texture directly to host buffer is counter-performant. Please use copy from texture to device buffer first.
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/algos_torch/a2c_continuous.py:106: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=self.mixed_precision):
fps step: 4422 fps step and policy inference: 4368 fps total: 4190 epoch: 1/5000 frames: 0
fps step: 6197 fps step and policy inference: 6106 fps total: 6055 epoch: 2/5000 frames: 24576

Example of error code pattern 2:

[INFO] Using python from: /home/robot/anaconda3/envs/iccar2025/bin/python
[INFO][AppLauncher]: Loading experience file: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/apps/isaaclab.python.headless.rendering.kit
Loading user config located at: '/home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/data/Kit/Isaac-Sim/4.2/user.config.json'
[Info] [carb] Logging to file: /home/robot/.local/share/ov/pkg/isaac-sim-4.2.0/kit/logs/Kit/Isaac-Sim/4.2/kit_20241117_123841.log
2024-11-17 03:38:41 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2024-11-17 03:38:42 [744ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:42 [744ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:42 [770ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.

|---------------------------------------------------------------------------------------------|
| Driver Version: 535.183.01    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA RTX A4000                 | Yes: 0 |     | 16622   MB | 10de      | 0          |
|     |                                  |        |     |            | 24b0      | 44ccd3c0.. |
|     |                                  |        |     |            | 6         |            |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 6.8.0-48-generic
| Processor: AMD Ryzen 7 2700 Eight-Core Processor | Cores: 8 | Logical: 16
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 48078 | Free Memory: 40730
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2046
|---------------------------------------------------------------------------------------------|
2024-11-17 03:38:43 [1,448ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2024-11-17 03:38:44 [2,239ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,239ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,250ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,250ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,259ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,259ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,269ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,269ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,271ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,271ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:44 [2,272ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-11-17 03:38:44 [2,272ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-11-17 03:38:47 [5,563ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
formatted_tuple: (0.5, 0.5, 0.5, 0.5)
[INFO]: Parsing configuration from: omni.isaac.lab_tasks.direct.AT3R.AT3R_rolling_env:AT3RRoughEnvCfg
[INFO]: Parsing configuration from: /home/robot/IsaacLab_v1.2.0/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/AT3R/agents/rl_games_rough_ppo_cfg.yaml
[INFO] Logging experiment in directory: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct
Setting seed: 42
[INFO]: Base environment:
    Environment device    : cuda:0
    Environment seed      : 42
    Physics step-size     : 0.005
    Rendering step-size   : 0.02
    Environment step-size : 0.02
[INFO] Generating terrains based on curriculum took : 1.709247 seconds
[INFO]: Time taken for scene creation : 5.009588 seconds
[INFO]: Scene manager:  <class InteractiveScene>
    Number of environments: 1024
    Environment spacing   : 4.0
    Source prim name      : /World/envs/env_0
    Global prim paths     : ['/World/ground']
    Replicate physics     : True
[INFO]: Starting the simulation. This may take a few seconds. Please wait...
2024-11-17 03:39:03 [21,512ms] [Warning] [omni.ujitso] UJITSO : Build storage validation failed for  Storage status: 1
2024-11-17 03:39:03 [21,512ms] [Error] [omni.physx.cooking.plugin] opRes == OperationResult::SUCCESS is false.
[INFO]: Time taken for simulation start : 58.231141 seconds
[INFO] Event Manager:  <EventManager> contains 1 active terms.
+--------------------------------------+
| Active Event Terms in Mode: 'startup' |
+----------+---------------------------+
|  Index   | Name                      |
+----------+---------------------------+
|    0     | physics_material          |
|    1     | add_base_mass             |
+----------+---------------------------+

[INFO]: Completed setting up the environment...
[INFO] Recording videos during training.
        video_folder: /home/robot/IsaacLab_v1.2.0/IsaacLab/logs/rl_games/AT3R_rough_direct/2024-11-17_12-38-52/videos/train
        step_trigger: lambda step: step % args_cli.video_interval == 0
        video_length: 400
        disable_logger: True
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/gymnasium/core.py:297: UserWarning: WARN: env.is_vector_env to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.is_vector_env` for environment variables or `env.get_attr('is_vector_env')` that will search the reminding wrappers.
  logger.warn(
self.seed = 42
Started to train
Exact experiment name requested from command line: 2024-11-17_12-38-52
seq_length: 4
current training device: cuda:0
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/common/a2c_common.py:254: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
  self.scaler = torch.cuda.amp.GradScaler(enabled=self.mixed_precision)
build mlp: 252
RunningMeanStd:  (1,)
2024-11-17 03:40:00 [78,469ms] [Warning] [omni.syntheticdata.plugin] OgnSdPostRenderVarToHost : rendervar copy from texture directly to host buffer is counter-performant. Please use copy from texture to device buffer first.
/home/robot/anaconda3/envs/iccar2025/lib/python3.10/site-packages/rl_games/algos_torch/a2c_continuous.py:106: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=self.mixed_precision):
fps step: 4462 fps step and policy inference: 4405 fps total: 4137 epoch: 1/5000 frames: 0
fps step: 6106 fps step and policy inference: 6017 fps total: 5972 epoch: 2/5000 frames: 24576

@RandomOakForest
Copy link
Collaborator

Thanks for posting this. We will review with the team.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@H-Hisamichi @RandomOakForest and others