Skip to content

Commit

Permalink
Re-work handling of batch's random filename
Browse files Browse the repository at this point in the history
If you need to use hab.bat concurrently you can now set the `HAB_RANDOM`
env var to prevent conflicts. This is slower so not used by default.
  • Loading branch information
MHendricks committed Mar 1, 2024
1 parent 1a555d8 commit 4c8bb5e
Show file tree
Hide file tree
Showing 2 changed files with 105 additions and 26 deletions.
63 changes: 63 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1214,6 +1214,69 @@ to work with Houdini 19.5 that ships with very dated versions of these packages.
In practice this just means that we have to cast pathlib objects to strings before
passing them to Jinja2.

## Concurrency
This is the recommended way to launch a bunch of alias commands at once from
within a python process.

```py
import hab

# Resolve the hab configuration
hab_uri = 'app/aliased'
resolver = hab.Resolver()
cfg = resolver.resolve(hab_uri)

# Launch the processes concurrently
procs = []
for _ in range(10):
proc = cfg.launch("as_str", ["-c", "print('done')"])
procs.append(proc)

# Example of waiting for all of them to finish and check for issues
errors = 0
for proc in procs:
out, _ = proc.communicate()
print(proc.returncode, '-' * 50, '\n', out)
if proc.returncode:
print("# BROKEN", proc.returncode)
errors += 1
print(f'Finished with {errors} errors')
```

Using `cfg.launch` directly calls the alias with the correct environment variables
without the need for each subprocess to have to re-evaluate the hab environment,
then launch the actual alias application as its own subprocess.

If not using python, the preferred method is to use `hab env` to configure your
environment once, and then call the aliases in bulk.

### Concurrency in Command Prompt

This only applies to running hab in batch mode on windows using the cli and
launching multiple processes at once using the same user. You may
run into issues where the temp dir each hab command creates gets re-used by
multiple processes.

By default `hab.bat` uses `%RANDOM%` which is very fast but uses time for its seed
which has a [maximum resolution of seconds](https://devblogs.microsoft.com/oldnewthing/20100617-00/?p=13673).
This is normally not an issue as a user is not able to call hab fast enough to
cause the issue.
```
A subdirectory or file C:\Users\username\AppData\Local\Temp\hab~7763 already exists.
```
If you end up seeing this message, or run into deleted file errors, then you can
set the env var `HAB_RANDOM` to one of these values. It is only respected when using
hab in batch mode.

| Value | Notes | ~ Time |
|---|---|---|
| fast | The default, uses `%RANDOM%` which is fast but may conflict when running concurrently. | 0.07s |
| safe | Uses python to generate a UUID. This is somewhat slower or it would be used by default. | 0.22s |
| [Anything else] | Anything else is a command to run and the output is captured as the unique value. | |

Approximate time generated using `time cmd.exe /c "hab -h"` in git bash after
omitting the `%py_exe% -m ...` call.

# Glosary

* **activate:** Update the current process(shell) for a given configuration. Name taken
Expand Down
68 changes: 42 additions & 26 deletions bin/hab.bat
Original file line number Diff line number Diff line change
Expand Up @@ -5,31 +5,9 @@

@ECHO OFF

:: Generate a unique temp folder to store hab's short term temp .bat files.
:: Batch's %RANDOM% isn't so random, https://devblogs.microsoft.com/oldnewthing/20100617-00/?p=13673
:: So if you are calling hab as a batch of subprocesses, you may run into issues
:: where multiple processes use the same random folder causing the dir to get
:: removed while the later processes finish.

:: 1. To work around this issue we add the current process's PID to the filename.
:: https://superuser.com/a/1746190
for /f "USEBACKQ TOKENS=2 DELIMS==" %%A in (`wmic process where ^(Name^="WMIC.exe" AND CommandLine LIKE "%%%%TIME%%%%"^) get ParentProcessId /value`) do set "PID=%%A"
rem echo PID: %PID%
rem timeout 10

:: 2. We also add a random number to hopefully reduce the chance of name conflicts
:uniqLoop
set "temp_directory=%tmp%\hab~%RANDOM%-%PID%"
:: If the folder already exists, re-generate a new folder name
if exist "%temp_directory%" goto :uniqLoop

:: Create the launch and config filenames we will end up using
mkdir %temp_directory%
set "temp_launch_file=%temp_directory%\hab_launch.bat"
set "temp_config_file=%temp_directory%\hab_config.bat"
SETLOCAL ENABLEDELAYEDEXPANSION

:: Calculate the command to run python with
SETLOCAL ENABLEEXTENSIONS
IF DEFINED HAB_PYTHON (
:: If HAB_PYTHON is specified, use it explicitly
set py_exe=%HAB_PYTHON%
Expand All @@ -41,9 +19,45 @@ IF DEFINED HAB_PYTHON (
set "py_exe=py -3"
)

:uniqLoop
:: Generate a unique temp folder to store hab's short term temp .bat files.
:: Batch's %RANDOM% isn't so random, https://devblogs.microsoft.com/oldnewthing/20100617-00/?p=13673
:: So if you are calling hab as a batch of subprocesses, you may run into issues
:: where multiple processes use the same random folder causing the dir to get
:: removed while the later processes finish.

:: If not set default to the fast random
IF "%HAB_RANDOM%"=="" (set "HAB_RANDOM=fast")

IF "%HAB_RANDOM%" == "safe" (
:: A safe but slower method when calling hab cli concurrently.
for /f %%i in ('%py_exe% -c "import uuid; print(uuid.uuid4())"') do set uuid=%%i
) ELSE IF "%HAB_RANDOM%"=="fast" (
:: Faster method to generate a random number that is not concurrent safe
set "uuid=!RANDOM!"
) ELSE (
:: Set uuid to HAB_RANDOM's output if it's set to anything else
set "uuid=%HAB_RANDOM%"
for /f %%i in ('%HAB_RANDOM%') do set uuid=%%i
)

set "temp_directory=%tmp%\hab~!uuid!"

:: If the folder already exists, re-generate a new folder name
if exist "!temp_directory!" goto :uniqLoop

:: Create the launch and config filenames we will end up using
mkdir !temp_directory!
:: There is a chance that between checking if the directory exists and trying
:: to create it, another process created it, Check if that occurred and generate
:: a new temp_directory if so
if errorlevel 1 goto :uniqLoop

set "temp_launch_file=!temp_directory!\hab_launch.bat"
set "temp_config_file=!temp_directory!\hab_config.bat"

:: Call our worker python process that may write the temp filename
%py_exe% -m hab --script-dir "%temp_directory%" --script-ext ".bat" %*
ENDLOCAL
%py_exe% -m hab --script-dir "!temp_directory!" --script-ext ".bat" %*

:: Run the launch or config script if it was created on disk
if exist %temp_launch_file% (
Expand All @@ -58,7 +72,9 @@ if exist %temp_launch_file% (
:: with @ to prevent printing them to the console

:: Remove the temp directory and its contents
@RMDIR /S /Q %temp_directory%
@RMDIR /S /Q !temp_directory!

ENDLOCAL

:: Ensure the errorlevel is reported to the calling process. This is needed
:: when calling hab via subprocess/QtCore.QProcess calls to receive errorlevel
Expand Down

0 comments on commit 4c8bb5e

Please sign in to comment.