From 4c8bb5e2dfba8632a6a88440b5f5ffbb386a8186 Mon Sep 17 00:00:00 2001 From: Mike Hendricks Date: Thu, 29 Feb 2024 13:11:33 -0800 Subject: [PATCH] Re-work handling of batch's random filename If you need to use hab.bat concurrently you can now set the `HAB_RANDOM` env var to prevent conflicts. This is slower so not used by default. --- README.md | 63 +++++++++++++++++++++++++++++++++++++++++++++++++ bin/hab.bat | 68 +++++++++++++++++++++++++++++++++-------------------- 2 files changed, 105 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index 1b9bc05..3ac3d01 100644 --- a/README.md +++ b/README.md @@ -1214,6 +1214,69 @@ to work with Houdini 19.5 that ships with very dated versions of these packages. In practice this just means that we have to cast pathlib objects to strings before passing them to Jinja2. +## Concurrency +This is the recommended way to launch a bunch of alias commands at once from +within a python process. + +```py +import hab + +# Resolve the hab configuration +hab_uri = 'app/aliased' +resolver = hab.Resolver() +cfg = resolver.resolve(hab_uri) + +# Launch the processes concurrently +procs = [] +for _ in range(10): + proc = cfg.launch("as_str", ["-c", "print('done')"]) + procs.append(proc) + +# Example of waiting for all of them to finish and check for issues +errors = 0 +for proc in procs: + out, _ = proc.communicate() + print(proc.returncode, '-' * 50, '\n', out) + if proc.returncode: + print("# BROKEN", proc.returncode) + errors += 1 +print(f'Finished with {errors} errors') +``` + +Using `cfg.launch` directly calls the alias with the correct environment variables +without the need for each subprocess to have to re-evaluate the hab environment, +then launch the actual alias application as its own subprocess. + +If not using python, the preferred method is to use `hab env` to configure your +environment once, and then call the aliases in bulk. + +### Concurrency in Command Prompt + +This only applies to running hab in batch mode on windows using the cli and +launching multiple processes at once using the same user. You may +run into issues where the temp dir each hab command creates gets re-used by +multiple processes. + +By default `hab.bat` uses `%RANDOM%` which is very fast but uses time for its seed +which has a [maximum resolution of seconds](https://devblogs.microsoft.com/oldnewthing/20100617-00/?p=13673). +This is normally not an issue as a user is not able to call hab fast enough to +cause the issue. +``` +A subdirectory or file C:\Users\username\AppData\Local\Temp\hab~7763 already exists. +``` +If you end up seeing this message, or run into deleted file errors, then you can +set the env var `HAB_RANDOM` to one of these values. It is only respected when using +hab in batch mode. + +| Value | Notes | ~ Time | +|---|---|---| +| fast | The default, uses `%RANDOM%` which is fast but may conflict when running concurrently. | 0.07s | +| safe | Uses python to generate a UUID. This is somewhat slower or it would be used by default. | 0.22s | +| [Anything else] | Anything else is a command to run and the output is captured as the unique value. | | + +Approximate time generated using `time cmd.exe /c "hab -h"` in git bash after +omitting the `%py_exe% -m ...` call. + # Glosary * **activate:** Update the current process(shell) for a given configuration. Name taken diff --git a/bin/hab.bat b/bin/hab.bat index 10fc1b5..ba58f19 100644 --- a/bin/hab.bat +++ b/bin/hab.bat @@ -5,31 +5,9 @@ @ECHO OFF -:: Generate a unique temp folder to store hab's short term temp .bat files. -:: Batch's %RANDOM% isn't so random, https://devblogs.microsoft.com/oldnewthing/20100617-00/?p=13673 -:: So if you are calling hab as a batch of subprocesses, you may run into issues -:: where multiple processes use the same random folder causing the dir to get -:: removed while the later processes finish. - -:: 1. To work around this issue we add the current process's PID to the filename. -:: https://superuser.com/a/1746190 -for /f "USEBACKQ TOKENS=2 DELIMS==" %%A in (`wmic process where ^(Name^="WMIC.exe" AND CommandLine LIKE "%%%%TIME%%%%"^) get ParentProcessId /value`) do set "PID=%%A" -rem echo PID: %PID% -rem timeout 10 - -:: 2. We also add a random number to hopefully reduce the chance of name conflicts -:uniqLoop -set "temp_directory=%tmp%\hab~%RANDOM%-%PID%" -:: If the folder already exists, re-generate a new folder name -if exist "%temp_directory%" goto :uniqLoop - -:: Create the launch and config filenames we will end up using -mkdir %temp_directory% -set "temp_launch_file=%temp_directory%\hab_launch.bat" -set "temp_config_file=%temp_directory%\hab_config.bat" +SETLOCAL ENABLEDELAYEDEXPANSION :: Calculate the command to run python with -SETLOCAL ENABLEEXTENSIONS IF DEFINED HAB_PYTHON ( :: If HAB_PYTHON is specified, use it explicitly set py_exe=%HAB_PYTHON% @@ -41,9 +19,45 @@ IF DEFINED HAB_PYTHON ( set "py_exe=py -3" ) +:uniqLoop +:: Generate a unique temp folder to store hab's short term temp .bat files. +:: Batch's %RANDOM% isn't so random, https://devblogs.microsoft.com/oldnewthing/20100617-00/?p=13673 +:: So if you are calling hab as a batch of subprocesses, you may run into issues +:: where multiple processes use the same random folder causing the dir to get +:: removed while the later processes finish. + +:: If not set default to the fast random +IF "%HAB_RANDOM%"=="" (set "HAB_RANDOM=fast") + +IF "%HAB_RANDOM%" == "safe" ( + :: A safe but slower method when calling hab cli concurrently. + for /f %%i in ('%py_exe% -c "import uuid; print(uuid.uuid4())"') do set uuid=%%i +) ELSE IF "%HAB_RANDOM%"=="fast" ( + :: Faster method to generate a random number that is not concurrent safe + set "uuid=!RANDOM!" +) ELSE ( + :: Set uuid to HAB_RANDOM's output if it's set to anything else + set "uuid=%HAB_RANDOM%" + for /f %%i in ('%HAB_RANDOM%') do set uuid=%%i +) + +set "temp_directory=%tmp%\hab~!uuid!" + +:: If the folder already exists, re-generate a new folder name +if exist "!temp_directory!" goto :uniqLoop + +:: Create the launch and config filenames we will end up using +mkdir !temp_directory! +:: There is a chance that between checking if the directory exists and trying +:: to create it, another process created it, Check if that occurred and generate +:: a new temp_directory if so +if errorlevel 1 goto :uniqLoop + +set "temp_launch_file=!temp_directory!\hab_launch.bat" +set "temp_config_file=!temp_directory!\hab_config.bat" + :: Call our worker python process that may write the temp filename -%py_exe% -m hab --script-dir "%temp_directory%" --script-ext ".bat" %* -ENDLOCAL +%py_exe% -m hab --script-dir "!temp_directory!" --script-ext ".bat" %* :: Run the launch or config script if it was created on disk if exist %temp_launch_file% ( @@ -58,7 +72,9 @@ if exist %temp_launch_file% ( :: with @ to prevent printing them to the console :: Remove the temp directory and its contents -@RMDIR /S /Q %temp_directory% +@RMDIR /S /Q !temp_directory! + +ENDLOCAL :: Ensure the errorlevel is reported to the calling process. This is needed :: when calling hab via subprocess/QtCore.QProcess calls to receive errorlevel