Skip to content

Latest commit

 

History

History
370 lines (252 loc) · 24.2 KB

APPLICATION_DEVELOPMENT.md

File metadata and controls

370 lines (252 loc) · 24.2 KB

Microsoft Ignite Sample Project

This project is designed to be used with the NVIDIA Omniverse kit-app-template. Using the kit-app-template together with the provided ./kit-app-source/source you can create a streaming Kit application that works together with the provided ./web-app client. If you use the kit-app-template, you agree you have read the NVIDIA Software License Agreement (available at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/) and Product-Specific Terms for NVIDIA Omniverse (available at https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-omniverse/) and agree to their terms. If you do not have the authority to enter into the License Agreement and/or do not accept the License Agreement terms and conditions, you should not register to use or use the kit-app-template.

Table of Contents

Overview

This repository provides sample code for creating the applications needed for a Omniverse Kit App Streaming solution. In such a solution, the Kit app is the application that runs on a remote server and the client/web-app is the application that displays the streamed Kit app.

Prerequisites

⚠️ Important: Before proceeding with the cloning step, ensure that Git Large File Storage (Git LFS) is installed on your system. To verify this, run the following command in your terminal:

git lfs --version

If the command returns a version number, Git LFS is installed correctly. If not, you will need to install Git LFS. You can download and install it from the official website here.

Kit App

  • kit-app-template using Kit SDK 106.4 or more recent. Please refer to the kit-app-template project prerequisites as well. The code in ./kit-app-source/source was developed to work with Kit SDK 106.4.

Web App

Quick Start

Kit App Setup

  1. Clone the https://github.com/NVIDIA-Omniverse/kit-app-template repo. The following instructions assume an un-modified kit-app-template project.

  2. Copy the ./kit-app-source/source directory in this project into the root directory of the cloned kit-app-template.

  3. Enable the kit-app-template repo tools to build the source by adding these lines to premake5.lua:

-- Apps: for each app generate batch files and a project based on kit files (e.g. my_name.my_app.kit)
define_app("msft.usd_viewer.kit")
define_app("msft.usd_viewer_streaming.kit")
  1. Add this line to kit_args.txt.
"--/app/auto_load_usd=${USD_PATH}"
  1. Enable precaching of the Kit app by replacing the [repo_precache_exts] section repo.toml with this:
[repo_precache_exts]
# Apps to run and precache
apps = [
"${root}/source/apps/msft.usd_viewer.kit",
"${root}/source/apps/msft.usd_viewer_streaming.kit",
]
  1. Refer to kit-app-template instructions to build the project. For kit-app-template based on Kit SDK 106.4, open a terminal in the root of the project and execute:
./repo.sh build

This completes the setup for the Kit app. You now have a built application that can be launched. Let's continue with the web app.

Web App Setup

  1. Open a terminal in ./web-app.

  2. Install dependencies:

npm install

If there is an error when running npm install:

  1. Delete ./web-app/node_modules and ./web-app/package-lock.json.
  2. Run npm install again.
  3. Run npm run build to create a distribution.

The web app is now ready to be launched locally or deployed.

Run on Workstation

Pushing a container to the Omniverse App Streaming deployment and testing functionality can be time consuming. You can easily run the Kit and web app locally for quick iteration.

  1. Launch the web app by opening a terminal in ./web-app and run npm run dev.
    1. Terminal displays a localhost URL such as localhost:5173. Open the URL in a Chromium browser.
    2. Enter 127.0.0.1 in the Server field and click Start Local Server. The web app is now trying to connect to a Kit app running locally.

  1. Promptly, launch the Kit app by opening a terminal in the root of kit-app-template and run ./repo.sh launch.

    1. When prompted, select the msft.usd_viewer_streaming.kit file.

  2. You can now see the Kit app streamed into the web app. The Kit app may be just a black rectangle because we didn't load a file.

  3. Click the Set Asset 1 URL or Set Asset 2 URL to send a message from the web app to the Kit app with a request to load a file.

Note: You can launch the Kit app with an argument to load a file using ./repo.sh launch -- --/app/auto_load_usd="\${app}/../samples/stage01.usd".

Additional information on running solution locally can be found in https://docs.omniverse.nvidia.com/embedded-web-viewer/latest/create/run-the-solution.html documentation.

Containerize for Deployment

  1. Follow the instructions in above Kit App Setup section.
  2. For kit-app-template based on Kit SDK 106.4, open a terminal in the root of the project and execute:
    1. When prompted, select the msft.usd_viewer_streaming.kit file.
./repo.sh package --container
  1. List containers:
docker image list
  1. Use docker tag and docker push commands to push the appropriately named container to a registry of your choosing.

Run Web App Locally Against Omniverse App Streaming Service

  1. Launch the web app by opening a terminal in ./web-app and run npm run dev.
  2. Terminal displays a localhost URL such as localhost:5173. Open the URL in a Chromium browser.
  3. Enter the URL for the Omniverse App Streaming Streaming Service in the Service field.
  4. Click the Start Service button.

Requesting a stream can take some time - especially if a new container needs to be pulled by the service. Have patience and monitor the browser console for progress. There can be errors while the web app attempts to connect to a running Kit app container.

How it Works

Debug Panel

The Debug Panel provides controls for modifying and viewing the state variables. It also provides a listing of active sessions and the ability to end them. The panel can be toggled with CTRL+M. To understand the code a developer should use this interface and follow the path of execution.

Controls

Use either the Service or Server controls to start a stream. Do not use both together. The Service control requests a remote session while the Server enables local streaming.

The Assets buttons shows how to send a request to load a USD file. The first two makes use of the assets bundled with the Kit application. Those assets are included in the Kit container. The third button executes a function that sends a request to load a file using SAS token. You'll need to edit the function variables to make this work.

The Selection buttons assume that one of the two bundled files are loaded. The files contain 4 USD Prims that have been tagged with asset_id attributes with the values cube, sphere, cone, and group. Clicking a button will toggle the given asset_id to be included in the selection list and a message will be sent to the Kit app to change the current selection.

The Asset State buttons also assumes that one of the two bundled files are loaded. These buttons send a request to change the Kit selection group for a given prim: nominal applies default USD Prim rendering, warning and fault provides different emphasis of the USD Prim rendering.

State

This section allows you to inspect the current state variable values. For example, if you load one of the two bundled files and select the cube prim in the viewport - not using the above mentioned buttons - you can see how the assetIds in the Selection group updates to list the cube. This is because Kit sends a message to the web app about the selection change and thus the state is updated.

Sessions

Here you can click the Get Sessions button to list the active sessions. You can use the buttons to delete them one by one or all of them. Note that one of them could be a session you are currently using.

Starting a Remote Stream

As mentioned in above Run Web App Locally Against Omniverse App Streaming Service, a session can be started using the Service field and Start Session button.

Clicking the Start Session button executes:

dispatch(setURL(`[some URL]`));
dispatch(setService({ address: addressValue, desiredState: 'connected' }));

The above does two things. First, it sets the UsdStorageState.url state to the given URL. That by itself does not send a request to the a streamed Kit application because we have not connected yet. Second, it sets the ServiceState.sessionService state and this triggers the ServiceComponent to request a streaming session.

Getting an active streaming session requires a number of steps to happen:

  1. Request the service. ServiceComponent makes use of the createStreamingSession API. The URL mentioned above is passed as an argument for the Kit app to load a stage.
    1. The request causes the remote service to pull and start a container.
  2. The web app polls the service using getStreamingSessionInfountil it receives information about an active stream.
  3. Once the ServiceComponent receives information about an active stream it sets the AppStreamState.remoteStream state.
  4. A change in AppStreamState.remoteStream state triggers AppStreamComponent to connect to the stream (see usage of AppStreamer.connect usage).
  5. Connecting to the remotely running container is a series of connection attempts (see usage of maxReconnects usage for number of attempts).
  6. Once connected the web client will display the streamed application.

Opening Files

Executing setURL updates the UsdStorageState.url state and causes the USDStorageComponent to send a request to the Kit app for opening a file.

The Kit msft.usd_viewer.messaging handles loading of files in stage_loading.py.

Viewport Navigation

The streamed viewport is interactive. Mouse events and camera navigation is automatically sent to the streamed application.

Selection and Status Indication

A USD Stage contains USD Prims. Each Prim has can be identified by a path. A solution could be created around USD paths but this creates restraints around USD authoring because paths would not be allowed to change unless the web app is also updated. Instead, this sample makes use of asset_ids. Relevant USD Prims are expected to have an asset_id attribute. Selection and "Status" changes can be communicated about using the asset_id instead of Prim paths.

Selection

Selection changes in the USD Stage can happen two ways: by selecting something in the viewport or by sending a message to select something.

Viewport selection is automatically handled. The streamed Kit app will send a message to the web app which updates the SelectionState.assetIds state.

Likewise, the web app can change the SelectionState.assetIds state and cause the SelectionComponent to send a message requesting a selection change to the Kit app.

The Kit msft.usd_viewer.messaging handles selection changes in stage_management.py.

Status

Changes in StatusState.assetId and StatusState.assetStatus causes the StatusComponent to send a message to the Kit app to change the status of an asset.

The Kit msft.usd_viewer.messaging handles the status in stage_status.py.

Ending a Stream

Clicking the End Session button executes:

dispatch(setService({ address: addressValue, desiredState: 'destroyed' }));

This updates the ServiceState.sessionService which triggers AppStreamComponent to terminate the stream via AppStreamer.terminate() and the ServiceComponent to request the service to destroy the current session via destroyStreamingSession API.

Kit Source Code Overview

Kit Files

Location: ./kit-app-source/source/apps.

File Description
msft.usd_viewer.kit Base application providing an RTX 3D viewport.
msft.usd_viewer_streaming.kit Extends msft.usd_viewer.kit by adding streaming and messaging support.

Extension: msft.usd_viewer.setup

Location: ./kit-app-source/source/extensions/msft.usd_viewer.setup.

Feature Description
Auto loading Supports application to be launched with --/app/auto_load_usd argument. When running within a deployment the container is passed this argument.
Sample files Two small sample USD files are included.
Layout Sets up the viewport-only application layout.

Extension: msft.usd_viewer.messaging

Location: ./kit-app-source/source/extensions/msft.usd_viewer.messaging.

Feature Description
Loading Enables a client to request loading a stage.
Stage Management Allows changing selection change per request. Informs client of selection changes when user interacts in viewport.
Asset Status Enables a client to indicate a status for a USD Prim. Makes use of selection groups to apply special rendering.

Web App Code Overview

Top Level

File Description
./web-app/index.html Top level html. Uses ./web-app/src/main.tsx
./web-app/src/main.tsx Makes use of various components. The critical one is <App/> - other are optional.
./web-app/src/App.tsx Makes use of required components described below.

State

Location: ./web-app/src/state.

State is presented using Redux. These files presents the state and allow other components to change it.

File Description
store.ts Provides the Redux reducers.
appStreamSlice.ts server state for local streaming.
remoteServer for remote stream.
kitState indicates the responsiveness of the streamed Kit application.
selectionSlice.ts assetIds state for selected assets.
selectionChangeOrigin is used to avoid loops in selection changes.
serviceSlice.ts sessionService state for controlling what service to use to and whether to create or destroy a service.
appId, version, and profile needs to be set to match the container to launch when requesting a stream session.
sessionId is the identifier of a current session and is provided by service.
status is the status code for when polling service for a session.
statusSlice.ts assetId is the identifier to change the status for.
assetStatus is the status for a given asset to send to the streamed Kit application.
usdStorageSlice.ts url is the most recently requested asset to load.
loadingState indicates current activity.
loadedURL is the most recently loaded asset.

Components

Location: ./web-app/src/components.

The components react to state changes and controls some of the states as well. They provide a streamed Omniverse 3D viewport and manages the communication with it.

File Description
AppStreamComponent.tsx Provides a streamed Omniverse 3D viewport.
SelectionComponent.ts Sends messages to streamed Omniverse 3D viewport to change selection.
ServiceComponent.ts Communicates with App Streaming service to create and destroy Omniverse 3D viewport sessions. Supports passing a USD URL to load as part of container startup.
StatusComponent.ts Sends messages to streamed Omniverse 3D viewport to change asset status.
USDStorageComponent.ts Sends messages to streamed Omniverse 3D viewport to load a stage.

Service

Location: ./web-app/src/service.

These files provide API for connecting to the Streaming and the Application service. In this implementation, only the Streaming service is used.

File Description
EndPoint.tsx Provides the main API for communicating with the service. Other files supports this API.

Debug

Location: ./web-app/src/debug.

The debug code provides sample code for controlling and inspecting the state. This code could be replaced by other code and the solution would still work.

File Description
Controls.tsx User interface elements that modifies the state.
SessionManagement.tsx Supports listing sessions and destroying them.
State.tsx Displays state values.