Packman setup for 5.3.0 broken after 5.3.1 release

[Zeblote]

Getting some errors running our build script on a new computer using version 5.3.0:

********** 1 / 2 REGENERATE PROJECT FILES **********
Running packman in preparation for cmake ...
Processing project file 'F:\BR\UnrealEngine\Engine\Source\ThirdParty\PhysX5\physx\dependencies.xml'
Package 'PhysXGpu' at version '105.1-5.3.4207.33265367-windows-public' is missing from local storage.
Downloading from https://d4i3qtqj3r0z5.cloudfront.net/PhysXGpu%40105.1-5.3.4207.33265367-windows-public
.       (speed 896 KiB/s)
Total of 0.00 seconds
packman(ERROR): Tar archive extension must be one of: [.tar .tar.bz2 .tar.bzip2 .tar.gz .tar.gzip .tar.xz]
'\VsWhere.exe' is not recognized as an internal or external command,
operable program or batch file.
'\VsWhere.exe' is not recognized as an internal or external command,
operable program or batch file.
'\VsWhere.exe' is not recognized as an internal or external command,
operable program or batch file.
Using preset xml: buildtools/presets/public/clangwin64-brickadia.xml
Target platform: win64 using compiler: clang
Traceback (most recent call last):
  File "F:\BR\UnrealEngine\Engine\Source\ThirdParty\PhysX5\physx\buildtools\cmake_generate_projects.py", line 322, in <module>
    main()
  File "F:\BR\UnrealEngine\Engine\Source\ThirdParty\PhysX5\physx\buildtools\cmake_generate_projects.py", line 318, in main
    presetProvided(presetName)
  File "F:\BR\UnrealEngine\Engine\Source\ThirdParty\PhysX5\physx\buildtools\cmake_generate_projects.py", line 249, in presetProvided
    print('PM_PATHS: ' + os.environ['PM_PATHS'])
  File "F:\BR\UnrealEngine\Engine\Source\ThirdParty\PhysX5\physx\packages\python\3.7.9-windows-x86_64\lib\os.py", line 681, in __getitem__
    raise KeyError(key) from None
KeyError: 'PM_PATHS'
Hit ENTER to continue...

The root cause seems to be that something is wrong with the PhysXGpu packman package downloaded before build. This didn't happen about a month ago (when I last tried clean setup).

Got it fixed by copying the packages folder from another computer, but probably something to look at.

[zalo]

Ah, I'm seeing this as well:

Running packman in preparation for cmake ...
Processing project file './dependencies.xml'
Package 'clang-physxmetadata' at version '4.0.0.32489833_1' is missing from local storage.
Downloading from https://d4i3qtqj3r0z5.cloudfront.net/clang-physxmetadata%404.0.0.32489833_1
No continuous progress report because this is not a proper terminal. Be patient ...
... (speed 1.16 MiB/s)
Total of 0.00 seconds
packman(ERROR): Tar archive extension must be one of: [.tar .tar.bz2 .tar.bzip2 .tar.gz .tar.gzip .tar.xz]

It seems like this file is out of date:
https://github.com/NVIDIA-Omniverse/PhysX/blob/main/physx/buildtools/packman/config.packman.xml
@preist-nvidia would you be able to take a look at this?

[preist-nvidia]

Please update packman, and it'll work: #230

We are looking at auto-updating packman as part of the generate projects phase to avoid this in the future.

[preist-nvidia]

Leaving this open for now because the discussion may not be as visible as it should be.

[Zeblote]

Oh I see, completely forgot the discussions tab existed so I didn't think to check there

[gupta-ish]

I'm facing the same issue on running generate_projects.sh :
Preset parameter required, available presets: (0) linux-aarch64 <--- Linux-aarch64 gcc PhysX SDK general settings (1) linux <--- Linux clang PhysX SDK general settings Enter preset number: 0 Running generate_projects.sh linux-aarch64 Running packman in preparation for cmake ... Processing project file '/home/quad/hapticsimulator/src/simulator/src/physx_lib/include/physx/dependencies.xml' Package 'CMakeModules' at version '1.28.trunk.32494385' is missing from local storage. Downloading from https://d4i3qtqj3r0z5.cloudfront.net/CMakeModules%401.28.trunk.32494385 . (speed 648 KiB/s) Total of 0.00 seconds packman(ERROR): Tar archive extension must be one of: [.tar .tar.bz2 .tar.bzip2 .tar.gz .tar.gzip .tar.xz] Traceback (most recent call last): File "/home/quad/hapticsimulator/src/simulator/src/physx_lib/include/physx/buildtools/cmake_generate_projects.py", line 307, in <module> main() File "/home/quad/hapticsimulator/src/simulator/src/physx_lib/include/physx/buildtools/cmake_generate_projects.py", line 298, in main result = subprocess.run(['bash', './generate_projects.sh', presetName], cwd=os.environ['PHYSX_ROOT_DIR'], check=True, universal_newlines=True) File "/home/quad/packman-repo/python/3.7.9-linux-x86_64/lib/python3.7/subprocess.py", line 512, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['bash', './generate_projects.sh', 'linux-aarch64']' returned non-zero exit status 1.

Is there any temporary fix for this? I need the physx SDK files for my project

[preist-nvidia]

@gupta-ish yes please look at the discussion #230 mentioned above.

[gupta-ish]

@gupta-ish yes please look at the discussion #230 mentioned above.

Somehow, I'm still getting the same error : include/physx/buildtools/packman$ ./packman update -y packman(WARNING): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091) packman(WARNING): Retrying after 20 seconds (2 retries left) ... packman(WARNING): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091) packman(WARNING): Retrying after 20 seconds (1 retry left) ... packman(WARNING): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091) packman(ERROR): Connection failure. Retries exhausted. Is network down?
Do you know anything that I can try here?

[preist-nvidia]

That's a different error that I have not seen before, and I cannot repro locally (with 5.3.0).

[zemlifr]

@gupta-ish @preist-nvidia
I have same problem with certificate verification.
Is there some resolution this?

[preist-nvidia]

Yes, please see #230

[zemlifr]

@preist-nvidia
That does not solve it.
I have latest repro, 5.4.0
Packman version is 7.23.1

Still I am unable to fetch dependencies due to invalid server certificate

Package 'PhysXDevice' at version '18.12.7.6' is missing from local storage.
packman(WARNING): SSLError: HTTPSConnectionPool(host='d4i3qtqj3r0z5.cloudfront.net', port=443): Max retries exceeded with url: /PhysXDevice%4018.12.7.6.7z (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')))

[preist-nvidia]

@zemlifr - I cannot repro this on a machine on a public network. I use the same packman version.

I assume you cannot download any of the packages?

[ayoub-belarbi]

I tried from my side as well (and deleted any existing package so Packman is forced to pull it again) on a public network and it works just fine.
Since we cannot reproduce in multiple computers, it seems related to your environment / Network probably. The issue is caused by an untrusted SSL certificate by the way since this GitHub issue seems to discuss multiple errors.

[zemlifr]

Yes, I have tried public network as well and it works. So you are right it is something with our company network setting. I will ask our admins.

[kv1128]

@preist-nvidia Hello :) The same here. I've just downloaded 4.1 version and 5.4 version on the same PC and put their folders in the same directory, as a result generate_projects.bat creates projects on 4.1, but shows the same errors about VsWhere on 5.4. Upgrading packman doesn't fix this. It says that the version is already 7.23.1

[ayoub-belarbi]

Hello @kv1128 ,
It would be great if you could share the logs here. The VSWhere error can be the result of multiple issues. Usually, it's shown in the logs before the VsWhere error.
Thanks

[kv1128]

Hi @ayoub-belarbi. I've edited logs a little bit in order to remove the actual paths that are not significant here. Well, the errors are a little bit different from the topic starter's, since there are some connection errors as well

Running generate_projects.bat vc16win64
Running packman in preparation for cmake ...
Package 'clang-physxmetadata' at version '4.0.0.32489833_1' is missing from local storage.
packman(WARNING): ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
packman(WARNING): Retrying after 20 seconds (2 retries left) ...
packman(WARNING): ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
packman(WARNING): Retrying after 20 seconds (1 retry left) ...
packman(WARNING): ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
packman(ERROR): Connection failure. Retries exhausted. Is service or network down?
"\VsWhere.exe" is not recognized as an internal or external command,
operable program or batch file.
"\VsWhere.exe" is not recognized as an internal or external command,
operable program or batch file.
"\VsWhere.exe" is not recognized as an internal or external command,
operable program or batch file.
Using preset xml: "\physx\buildtools\presets\public\vc16win64.xml
Target platform: win64 using compiler: vc16
Traceback (most recent call last):
File "\physx\buildtools\cmake_generate_projects.py", line 325, in
main()
File "\physx\buildtools\cmake_generate_projects.py", line 322, in main
presetProvided(presetName, physx_root_dir)
File "\physx\buildtools\cmake_generate_projects.py", line 250, in presetProvided
print('PM_PATHS: ' + os.environ['PM_PATHS'])
File "D:\packman-repo\python\3.10.5-1-windows-x86_64\lib\os.py", line 679, in getitem
raise KeyError(key) from None
KeyError: 'PM_PATHS'

[ayoub-belarbi]

Thank you @kv1128 ,
Are you using a VPN or your network is behind a firewall ?

[kv1128]

Thank you @kv1128 , Are you using a VPN or your network is behind a firewall ?

No VPN and no firewall, at least on my PC

[ayoub-belarbi]

Did you try to update to latest packman?
Please try this #230

[kv1128]

Did you try to update to latest packman? Please try this #230

Yes, I tried, but it doesn't seem to update anything

Downloading from https://bootstrap.packman.nvidia.com/packman-command%407.last-known-good.txt (26.00 bytes)
100.00% (speed 208 kb/s)
Total of 0.00 seconds
Version is already 7.23.1. To reinstall same version, specify 'reinstall' option.