Proposed RFC Feature: Introduce DirectX Agility SDK

[hosea1008]

Summary:

When using DirectX RHI, some advanced shader features are delivered by high version of shader model. The shader model version has a strong binding to the Windows version, therefore, lower version of Windows cannot use some advanced shader features. The DirectX Agility SDK is the only way to get rid of such restriction.

This proposal plans to introduce DirectX Agility SDK into O3DE, so that when using DX RHI, the engine itself can be independent of the windows system version even after introducing some advanced features that relies on higher version of shader model. (e.g., Windows 10 only supports shader version up to SM 6.5 according to this document).

You may see that Unreal Engine is doing exactly the same thing like this.

What is the relevance of this feature?

The ultimate goal is to make O3DE more compatible to lower version of Windows if running DX RHI, even after introducing some new advanced features needs higher version of shader model, for example, if a Gem requires 64-bit atomic operation, that would need shader model 6.6 and limits the compatible platform to Windows 11+ in DX12 RHI. After introducing DirectX Agility SDK, one may run it in some Windows 10 versions. This point has been verify by some of our real projects.

Feature design description:

According to the official blog of DirectX Agility SDK, to introduce it, I think the best and most efficient way to introduce it is to add the D3D12SDKVersion and D3D12SDKPath keyword to the main.cpp, and use a compile flag to control whether or not enabling it.

As mentioned before, you may see that Unreal Engine is also using this way like this.

Technical design description:

Introduce USE_D3D12_AGILITY_SDK macro to control whether or not enabling DirectX Agility SDK in compile time.

I am planning to introduce USE_D3D12_AGILITY_SDK macro to control whether or not enabling it:

in both Code/Editor/CMakeLists.txt and Code/LauncherUnified/CMakeLists.txt, add

set(USE_D3D12_AGILITY_SDK OFF CACHE BOOL "Introduce DirectX 12 Agility SDK.")
if(USE_D3D12_AGILITY_SDK)
    message(STATUS "Enabling DirectX 12 Agility SDK as default DirectX backend.")
    set(D3D12_AGILITY_SDK_DEPENDENCIES 3rdParty::DirectX12AgilitySDK)
endif()

Add COMPILE_DEFINITIONS $<$<BOOL:${USE_D3D12_AGILITY_SDK}>:USE_D3D12_AGILITY_SDK> in both targets Launcher.Game.Static and Editor

Use DirectX Agility SDK in both Editor and GameLauncher:

in Code/LauncherUnified/Platform/Windows/Launcher_Game_Windows.cpp, append

#if defined(USE_D3D12_AGILITY_SDK)
extern "C" 
{ 
    __declspec(dllexport) extern const unsigned int D3D12SDKVersion = 610;
    __declspec(dllexport) extern const char* D3D12SDKPath = u8".\\D3D12\\"; 
}
#endif() // USE_D3D12_AGILITY_SDK

in Code/Editor/main.cpp, add the following lines before int main(int argc, char* argv[])

#if defined(AZ_PLATFORM_WINDOWS)
#if defined(USE_D3D12_AGILITY_SDK)
extern "C" 
{ 
    AZ_DLL_EXPORT extern const uint32_t D3D12SDKVersion = 610;
    AZ_DLL_EXPORT extern const char* D3D12SDKPath = u8".\\D3D12\\"; 
}
#endif() // USE_D3D12_AGILITY_SDK
#endif() // AZ_PLATFORM_WINDOWS

Introduce DirectX Agility SDK as 3rdParty dependency?

For this part I am not quite sure about the correct way to do it, in our inner version we pre-built the binary and put them in our own 3rdParty server, so that the engine can download and introduce it when compiling, I am not sure what's the correct way to introduce a new 3rdParty package to the community version? And is it a good idea to introduce binary? Would it be better to include the source code of it just like the DirectX Shader Compiler here: o3de/DirectXShaderCompiler?

What are the advantages of the feature?

The engine can be better compatible to different Windows versions, and the developer still have the flexibility to control enable it or not.

What are the disadvantages of the feature?

The changes in main.cpp is a little bit ugly? But I couldn't find a better way to do it since it is required by the design of Agility SDK.

Are there any alternatives to this feature?

No

How will users learn this feature?

Users won't notice this feature, they will only find that in lower version of Windows they can still using some advanced shader features, the producer of the engine have the control of enabling or not.

Are there any open questions?

The best way to introduce the SDK, introduce a pre-built binary or introduce the source code and compile it along with the engine? I myself prefer introducing pre-built SDK, we are already using this way in our inner version, but I don't know how can I do it for the community version.

[Ulrick28]

To get some eyes on this after the holidays, added @moudgils
@VickyAtAZ
VickyAtAZ
@antonmic
antonmic
@galibzon
galibzon

[moudgils]

Overall I like this addition. Dx12 RHI was bottlenecked by Agility SDK when it came to experimentation with new DX api so this is a welcome addition. A few points below

1. Shouldn't D3D12SDKVersion and D3D12SDKPath be part of Dx12 RHI gem as oppose to Launcher_Game_Windows.cpp or Editor/main.cpp? This way you won't have to duplicate it in two places.
2. AgilitySDK can be provided as a 3rd party lib but that requires a lot of work to maintain. You will need to create a script within https://github.com/o3de/3p-package-source and then use that pull in the 3p package at runtime. I wonder if providing a way to download directly via NuGet is an option during Configure. Microsoft provides a way to do this via.
   NuGet.exe install Microsoft.Direct3D.D3D12 -Source <FolderWith.nupkg> -OutputDirectory <DestinationFolder>

Tagging @spham-amzn and @nick-l-o3de to comment on ease of using Nuget for Agility SDK.

[hosea1008]

Overall I like this addition. Dx12 RHI was bottlenecked by Agility SDK when it came to experimentation with new DX api so this is a welcome addition. A few points below

1. Shouldn't D3D12SDKVersion and D3D12SDKPath be part of Dx12 RHI gem as oppose to Launcher_Game_Windows.cpp or Editor/main.cpp? This way you won't have to duplicate it in two places.
2. AgilitySDK can be provided as a 3rd party lib but that requires a lot of work to maintain. You will need to create a script within https://github.com/o3de/3p-package-source and then use that pull in the 3p package at runtime. I wonder if providing a way to download directly via NuGet is an option during Configure. Microsoft provides a way to do this via.
   NuGet.exe install Microsoft.Direct3D.D3D12 -Source <FolderWith.nupkg> -OutputDirectory <DestinationFolder>

Tagging @spham-amzn and @nick-l-o3de to comment on ease of using Nuget for Agility SDK.

1. I've tried adding these in DX12_Windows.h but it doesn't work, I can try somewhere else in DX12 RHI again.
2. That's a good way, I didn't try it before, I can take a look, and any help of using NuGet in O3DE or cmake is appreciated

[spham-amzn]

1. The proposed addition of the USE_D3D12_AGILITY_SDK should be placed in a windows-specific only cmake file, and since it affects multiple targets, perhaps the USE_D3D12_AGILITY_SDK and D3D12_AGILITY_SDK_DEPENDENCIES should be defined in PAL_Windows.cmake

2. Using #if defined(AZ_PLATFORM_WINDOWS) will cause a validation error since its not permitted to do platform-specific gating in general in the common files. The PAL idiom requires that platform specific content reside in their respective PAL specific file, so in the example for Code/Editor/main.cpp, that block should instead be moved somewhere in https://github.com/o3de/o3de/tree/development/Code/Editor/Platform/Windows. Probably a trait header file, and you can follow the pattern that AzCore uses with AzCore_Traits_Platform.h, or you can add a main_Windows.cpp instead.

3. I think if its the case that the Agility SDK DLLs are meant to be project local, and that games that are distributed with it will need it, then create a new 3rd Party package for it is the correct approach. The NuGet.exe steps suggested can actually be used in the 3rd party script to generate the library.
   However, using this approach will require this DLL to be redistributed with any packaged windows game. Normally, for release builds, they are built monolithically and minimally to reduce (or eliminate) and dynamic module loading, so if we want to follow this approach, there will need to be a way to statically link the agility SDK (if possible).
   If this is not possible, then the packaged game will need to separately install this redistributable, unless the Agility SDK license allows for a 3rd party distribution for games in this manner.

The repo for 3rd Party packages is here.

[hosea1008]

Before actually working on introducing DirectX Agility SDK, I am wondering whether the license of it is friendly enough?

https://www.nuget.org/packages/Microsoft.Direct3D.D3D12/1.614.0/License

[moudgils]

Doesnt sound like it. Basically, unless the license is MIT/Apache it is probably not going to work for the core engine. 3p package is the way to go here

[hosea1008]

Considering the discussion in Discord https://discord.com/channels/805939474655346758/821692567355523152/1293152651177099296, I prefer introducing DirectX Agility SDK like we did to introduce PIX, i.e., provide cmake option to enable/disable it, and allow users to configure the SDK's path, but the user have to think about the license issue and download the SDK manually.

I am NOT going to put the DirectX Agility SDK inside O3DE or O3DE's 3rd party repo, and will NOT ship O3DE with DirectX Agility SDK by default.

If the user enables agility SDK and provides the SDK's path, the SDK's dll would be included in the compile output, and I will set the required D3D12SDKVersion and D3D12SDKPath in a platform specific manner.

If this works, I am going to do it in this way.

[moudgils]

This could work as a start. Having a default 3rd party lib is also not a bad idea for the default engine (can be added later on) and people can override it easily with the setup mentioned above. But please add details related to this in the wiki here (https://github.com/o3de/o3de/wiki) or add documentation to the main site if possible. Otherwise folks will have no idea how to override it.