Add docs for CachedResolver

LucaScheller · Oct 30, 2023 · a46b732 · a46b732
1 parent 08d366b
commit a46b732
Show file tree

Hide file tree

Showing 9 changed files with 227 additions and 10 deletions.
diff --git a/README.md b/README.md
@@ -25,7 +25,7 @@ See our [Automatic Installation](https://lucascheller.github.io/VFX-UsdAssetReso
 Asset resolvers that can be compiled via this repository:
 - **Production Resolvers**
     - **File Resolver** - A file system based resolver similar to the default resolver with support for custom mapping pairs as well as at runtime modification and refreshing.
-    - **Cached Resolver** - Still work in progress, more info coming soon.
+    - **Cached Resolver** - A resolver that first consults an internal resolver context dependent cache to resolve asset paths. If the asset path is not found in the cache, it will redirect the request to Python and cache the result. This is ideal for smaller studios, as this preserves the speed of C++ with the flexibility of Python.
 - **RnD Resolvers**
     - **Python Resolver** - Python based implementation of the file resolver. The goal of this resolver is to enable easier RnD by running all resolver and resolver context related methods in Python. It can be used to quickly inspect resolve calls and to setup prototypes of resolvers that can then later be re-written in C++ as it is easier to code database interactions in Python for initial research.
 - **Proof Of Concept Resolvers**

diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md
@@ -14,6 +14,8 @@
     - [Production Resolvers](./resolvers/production.md)
         - [File Resolver](./resolvers/FileResolver/overview.md)
             - [Python API](./resolvers/FileResolver/PythonAPI.md)
+        - [Cached Resolver](./resolvers/CachedResolver/overview.md)
+            - [Python API](./resolvers/CachedResolver/PythonAPI.md)
     - [RnD Resolvers](./resolvers/rnd.md)
         - [Python Resolver](./resolvers/PythonResolver/overview.md)
             - [Python API](./resolvers/PythonResolver/PythonAPI.md)

diff --git a/docs/src/resolvers/CachedResolver/PythonAPI.md b/docs/src/resolvers/CachedResolver/PythonAPI.md
@@ -0,0 +1,177 @@
+## Overview
+You can import the Python module as follows:
+```python
+from pxr import Ar
+from usdAssetResolver import CachedResolver
+```
+
+## Tokens
+Tokens can be found in CachedResolver.Tokens:
+```python
+CachedResolver.Tokens.mappingPairs
+```
+
+## Resolver Context
+You can manipulate the resolver context (the object that holds the configuration the resolver uses to resolve paths) via Python in the following ways:
+
+```python
+from pxr import Ar, Usd
+from usdAssetResolver import CachedResolver
+
+# Get via stage
+stage = Usd.Stage.Open("/some/stage.usd")
+context_collection = stage.GetPathResolverContext()
+cachedResolver_context = context_collection.Get()[0]
+# Or context creation
+cachedResolver_context = CachedResolver.ResolverContext()
+
+# To print a full list of exposed methods:
+for attr in dir(CachedResolver.ResolverContext):
+    print(attr)
+```
+
+### Refreshing the Resolver Context
+```admonish important
+If you make changes to the context at runtime, you'll need to refresh it!
+```
+You can reload it as follows, that way the active stage gets the change notification.
+
+```python
+from pxr import Ar
+from usdAssetResolver import CachedResolver
+resolver = Ar.GetResolver()
+context_collection = pxr.Usd.Stage.GetPathResolverContext()
+cachedResolver_context = context_collection.Get()[0]
+# Make edits as described below to the context.
+cachedResolver_context.AddMappingPair("identifier.usd", "/absolute/file/path/destination.usd")
+# Trigger Refresh (Some DCCs, like Houdini, additionally require node re-cooks.)
+stage.RefreshContext(context_collection)
+```
+
+### Editing the Resolver Context
+We can edit the mapping and cache via the resolver context.
+We also use these methods in the `PythonExpose.py` module.
+
+```python
+import json
+stage = pxr.Usd.Stage.Open("/some/stage.usd")
+context_collection = pxr.Usd.Stage.GetPathResolverContext()
+cachedResolver_context = context_collection.Get()[0]
+
+# Mapping Pairs (Same as Caching Pairs, but have a higher loading priority)
+cachedResolver_context.AddMappingPair("identifier.usd", "/absolute/file/path/destination.usd")
+# Caching Pairs
+cachedResolver_context.AddCachingPair("identifier.usd", "/absolute/file/path/destination.usd")
+# Clear mapping and cached pairs (but not the mapping file path)
+cachedResolver_context.ClearAndReinitialize()
+# Load mapping from mapping file
+cachedResolver_context.SetMappingFilePath("/some/mapping/file.usd")
+cachedResolver_context.ClearAndReinitialize()
+
+# Trigger Refresh (Some DCCs, like Houdini, additionally require node re-cooks.)
+stage.RefreshContext(context_collection)
+```
+When the context is initialized for the first time, it runs the `ResolverContext.Initialize` method as described below. Here you can add any mapping and/or cached pairs as you see fit.
+
+### Mapping/Caching Pairs
+To inspect/tweak the active mapping/caching pairs, you can use the following:
+```python
+ctx.ClearAndReinitialize()                    # Clear mapping and cache pairs and re-initialize context (with mapping file path)
+ctx.GetMappingFilePath()                      # Get the mapping file path (Defaults to file that the context created via Resolver.CreateDefaultContextForAsset() opened")
+ctx.SetMappingFilePath(p: str)                # Set the mapping file path
+ctx.RefreshFromMappingFilePath()              # Reload mapping pairs from the mapping file path
+ctx.GetMappingPairs()                         # Returns all mapping pairs as a dict
+ctx.AddMappingPair(src: string, dst: str)     # Add a mapping pair
+ctx.RemoveMappingByKey(src: str)              # Remove a mapping pair by key
+ctx.RemoveMappingByValue(dst: str)            # Remove a mapping pair by value
+ctx.ClearMappingPairs()                       # Clear all mapping pairs
+ctx.GetCachingPairs()                         # Returns all caching pairs as a dict
+ctx.AddCachingPair(src: string, dst: str)     # Add a caching pair
+ctx.RemoveCachingByKey(src: str)              # Remove a caching pair by key
+ctx.RemoveCachingByValue(dst: str)            # Remove a caching pair by value
+ctx.ClearCachingPairs()                       # Clear all caching pairs
+```
+
+To generate a mapping .usd file, you can do the following:
+```python
+from pxr import Ar, Usd, Vt
+from usdAssetResolver import CachedResolver
+stage = Usd.Stage.CreateNew('/some/path/mappingPairs.usda')
+mapping_pairs = {'assets/assetA/assetA.usd':'/absolute/project/assets/assetA/assetA_v005.usd', '/absolute/project/shots/shotA/shotA_v000.usd':'shots/shotA/shotA_v003.usd'}
+mapping_array = []
+for source_path, target_path in mapping_pairs.items():
+    mapping_array.extend([source_path, target_path])
+stage.SetMetadata('customLayerData', {CachedResolver.Tokens.mappingPairs: Vt.StringArray(mapping_array)})
+stage.Save()
+```
+
+### PythonExpose.py Overview
+As described in our [overview](./overview.md) section, the cache population is handled completely in Python, making it ideal for smaller studios, who don't have the C++ developer resources.
+
+You can find the basic implementation version that gets shipped with the compiled code here:
+[PythonExpose.py](https://github.com/LucaScheller/VFX-UsdAssetResolver/blob/main/src/CachedResolver/PythonExpose.py).
+
+```admonish important
+You can live edit it after the compilation here: ${REPO_ROOT}/dist/cachedResolver/lib/python/PythonExpose.py (or resolvers.zip/CachedResolver/lib/python folder if you are using the pre-compiled releases).
+Since the code just looks for the `PythonExpose.py` file anywhere in the `sys.path` you can also move or re-create the file anywhere in the path to override the behaviour. The module name can be controlled by the `CMakeLists.txt` file in the repo root by setting `AR_CACHEDRESOLVER_USD_PYTHON_EXPOSE_MODULE_NAME` to a different name.
+
+You'll want to adjust the content, so that identifiers get resolved and cached to what your pipeline needs.
+```
+
+```admonish tip
+We also recommend checking out our unit tests of the resolver to see how to interact with it. You can find them in the "<Repo Root>/src/CachedResolver/testenv" folder or on [GitHub](https://github.com/LucaScheller/VFX-UsdAssetResolver/blob/main/src/CachedResolver/testenv/testCachedResolver.py).
+```
+
+Below we show the Python exposed methods, note that we use static methods, as we just call into the module and don't create the actual object. (This module could just as easily been made up of pure functions, we just create the classes here to make it match the C++ API.)
+
+To enable a similar logging as the `TF_DEBUG` env var does, you can uncomment the following in the `log_function_args` function.
+
+```python
+...code...
+def log_function_args(func):
+    ...code...
+    # To enable logging on all methods, re-enable this.
+    # LOG.info(f"{func.__module__}.{func.__qualname__} ({func_args_str})")
+...code...
+```
+
+#### Resolver Context
+
+```python
+class ResolverContext:
+    @staticmethod
+    def Initialize(context):
+        """Initialize the context. This get's called on default and post mapping file path
+        context creation.
+
+        Here you can inject data by batch calling context.AddCachingPair(assetPath, resolvePath),
+        this will then populate the internal C++ resolve cache and all resolves calls
+        to those assetPaths will not invoke Python and instead use the cache.
+
+        Args:
+            context (CachedResolverContext): The active context.
+        """
+        # Very important: In order to add a path to the cache, you have to call:
+        #    context.AddCachingPair(assetPath, resolvedAssetPath)
+        # You can add as many identifier->/abs/path/file.usd pairs as you want.
+        context.AddCachingPair("identifier", "/some/path/to/a/file.usd")
+
+    @staticmethod
+    def ResolveAndCache(assetPath, context):
+        """Return the resolved path for the given assetPath or an empty
+        ArResolvedPath if no asset exists at that path.
+        Args:
+            assetPath (str): An unresolved asset path.
+            context (CachedResolverContext): The active context.
+        Returns:
+            str: The resolved path string. If it points to a non-existent file,
+                 it will be resolved to an empty ArResolvedPath internally, but will
+                 still count as a cache hit and be stored inside the cachedPairs dict.
+        """
+        # Very important: In order to add a path to the cache, you have to call:
+        #    context.AddCachingPair(assetPath, resolvedAssetPath)
+        # You can add as many identifier->/abs/path/file.usd pairs as you want.
+        resolved_asset_path = "/some/path/to/a/file.usd"
+        context.AddCachingPair(assetPath, resolved_asset_path)
+        return resolved_asset_path
+```
diff --git a/docs/src/resolvers/CachedResolver/overview.md b/docs/src/resolvers/CachedResolver/overview.md
@@ -0,0 +1,39 @@
+# Cached Resolver
+## Overview
+```admonish tip
+This resolver first consults an internal resolver context dependent cache to resolve asset paths. If the asset path is not found in the cache, it will redirect the request to Python and cache the result. This is ideal for smaller studios, as this preserves the speed of C++ with the flexibility of Python.
+```
+
+Similar to the FileResolver and USD's default resolver, any absolute and relative file path is resolved as an on-disk file path. That means "normal" USD files, that don't use custom identifiers, will resolve as expected (and as fast as usual as this is called in C++).
+
+All non file path identifiers (anything that doesn't start with "/"/"./"/"../") will forward their request to the `PythonExpose.py` -> `ResolverContext.ResolveAndCache` method.
+If you want to customize this resolver, just edit the methods in PythonExpose.py to fit your needs. You can either edit the file directly or move it anywhere where your "PYTHONPATH"/"sys.path" paths look for Python modules.
+
+We also recommend checking out our unit tests of the resolver to see how to interact with it. You can find them in the "<Repo Root>/src/CachedResolver/testenv" folder or on [GitHub](https://github.com/LucaScheller/VFX-UsdAssetResolver/blob/main/src/CachedResolver/testenv/testCachedResolver.py).
+
+Here is a full list of features:
+- We support adding caching pairs in two ways, cache-lookup-wise they do the same thing, except the "MappingPairs" have a higher priority than "CachedPairs":
+    - **MappingPairs**: All resolver context methods that have `Mapping` in their name, modify the internal `mappingPairs` dictionary. As with the [FileResolver](../FileResolver/overview.md) and [PythonResolver](../PythonResolver/overview.md) resolvers, mapping pairs get populated when creating a new context with a specified mapping file or when editing it via the exposed Python resolver context methods. When loading from a file, the mapping data has to be stored in the Usd layer metadata in an key called ```mappingPairs``` as an array with the syntax ```["sourceIdentifierA.usd", "/absolute/targetPathA.usd", "sourceIdentifierB.usd", "/absolute/targetPathB.usd"]```. (This is quite similar to Rodeo's asset resolver that can be found [here](https://github.com/rodeofx/rdo_replace_resolver) using the AR 1.0 specification.). See our [Python API](./PythonAPI.md) page for more information.
+    - **CachingPairs**: All resolver context methods that have `Caching` in their name, modify the internal `cachingPairs` dictionary. With this dictionary it is up to you when to populate it. In our `PythonExpose.py` file, we offer two ways where you can hook into the resolve process. In both of them you can add as many cached lookups as you want via `ctx.AddCachingPair(asset_path, resolved_asset_path)`:
+        - On context creation via the `PythonExpose.py` -> `ResolverContext.Initialize` method. This gets called whenever a context gets created (including the fallback default context). For example Houdini creates the default context if you didn't specify a "Resolver Context Asset Path" in your stage on the active node/in the stage network. If you do specify one, then a new context gets spawned that does the above mentioned mapping pair lookup and then runs the `PythonExpose.py` -> `ResolverContext.Initialize` method.
+        - On resolve for non file path identifiers (anything that doesn't start with "/"/"./"/"../") via the `PythonExpose.py` -> `ResolverContext.ResolveAndCache` method. Here you are free to only add the active asset path via `ctx.AddCachingPair(asset_path, resolved_asset_path)` or any number of relevant asset paths.
+- In comparison to our [FileResolver](../FileResolver/overview.md) and [PythonResolver](../PythonResolver/overview.md), the mapping/caching pair values need to point to the absolute disk path. We chose to make this behavior different, because in the "PythonExpose.py" you can directly customize the "final" on-disk path to your liking.  
+- The resolver contexts are cached globally, so that DCCs, that try to spawn a new context based on the same mapping file using the [```Resolver.CreateDefaultContextForAsset```](https://openusd.org/dev/api/class_ar_resolver.html), will re-use the same cached resolver context. The resolver context cache key is currently the mapping file path. This may be subject to change, as a hash might be a good alternative, as it could also cover non file based edits via the exposed Python resolver API.
+- ```Resolver.CreateContextFromString```/```Resolver.CreateContextFromStrings``` is not implemented due to many DCCs not making use of it yet. As we expose the ability to edit the context at runtime, this is also often not necessary. If needed please create a request by submitting an issue here: [Create New Issue](https://github.com/LucaScheller/VFX-UsdAssetResolver/issues/new)
+- Refreshing the stage is also supported, although it might be required to trigger additional reloads in certain DCCs.
+
+
+```admonish warning
+While the resolver works and gives us the benefits of Python and C++, we don't guarantee its scalability. If you look into our source code, you'll also see that our Python invoke call actually circumvents the "const" constant variable/pointers in our C++ code. USD API-wise the resolve ._Resolve calls should only access a read-only context. We side-step this design by modifying the context in Python. Be aware that this could have side effects.
+```
+
+## Debug Codes
+Adding following tokens to the `TF_DEBUG` env variable will log resolver information about resolution/the context respectively.
+* `CACHEDRESOLVER_RESOLVER`
+* `CACHEDRESOLVER_RESOLVER_CONTEXT`
+
+For example to enable it on Linux run the following before executing your program:
+
+```bash
+export TF_DEBUG=CACHEDRESOLVER_RESOLVER_CONTEXT
+```
diff --git a/docs/src/resolvers/FileResolver/PythonAPI.md b/docs/src/resolvers/FileResolver/PythonAPI.md
@@ -6,7 +6,6 @@ from pxr import Ar
 from usdAssetResolver import FileResolver
 ```
 
-
 ## Tokens
 Tokens can be found in FileResolver.Tokens:
 ```python
@@ -62,7 +61,7 @@ ctx.SetCustomSearchPaths(searchPaths: list) # Set custom search paths
 To inspect/tweak the active mapping pairs, you can use the following:
 ```python
 ctx.GetMappingFilePath()                      # Get the mapping file path (Defaults file that the context created Resolver.CreateDefaultContextForAsset() opened)
-ctx.SetMappingFilePath()                      # Set the mapping file path
+ctx.SetMappingFilePath(p: str)                # Set the mapping file path
 ctx.RefreshFromMappingFilePath()              # Reload mapping pairs from the mapping file path
 ctx.GetMappingPairs()                         # Returns all mapping pairs as a dict
 ctx.AddMappingPair(src: string, dst: str)     # Add a mapping pair
@@ -89,5 +88,5 @@ To change the asset path formatting before it is looked up in the mapping pairs,
 ctx.GetMappingRegexExpression()               # Get the regex expression
 ctx.SetMappingRegexExpression(regex_str: str) # Set the regex expression
 ctx.GetMappingRegexFormat()                   # Get the regex expression substitution formatting
-ctx.SetMappingRegexFormat()                   # Set the regex expression substitution formatting
+ctx.SetMappingRegexFormat(f: str)             # Set the regex expression substitution formatting
 ```
diff --git a/docs/src/resolvers/PythonResolver/PythonAPI.md b/docs/src/resolvers/PythonResolver/PythonAPI.md
@@ -92,7 +92,7 @@ Below we show the Python exposed methods, note that we use static methods, as we
 
 The method signatures match the C++ signatures, except how the context is injected, as this is necessary due to how the Python exposing works.
 
-To enable a similiar logging as the `TF_DEBUG` env var does, you can uncomment the following in the `log_function_args` function.
+To enable a similar logging as the `TF_DEBUG` env var does, you can uncomment the following in the `log_function_args` function.
 
 ```python
 ...code...

diff --git a/docs/src/resolvers/production.md b/docs/src/resolvers/production.md
@@ -1,4 +1,4 @@
 # Production Resolvers
 Here you can find production ready asset resolvers, checkout our [Resolvers Overview](../resolvers/overview.md) section for an outline of their features:
 - [File Resolver](./FileResolver/overview.md)
-- [Cached Resolver]()
+- [Cached Resolver](./CachedResolver/overview.md)