[Feature] Camera rendering configuration (#2137)

* Add an initial set of masking utilities.

* Add an initial set of masking utilities.

* Push background improvements

* Demonstrate more of observation conversion

* Fix heightfield.py

* Improve performance of kernel

* Format

* Add edge generation file.

* Show minicity.

* Update bitmap

* Improve kernel padding.

* Remove default padding from mesh2bitmap.

* Fix perlin generator.

* Update perlin with shift.

* Rename perlin bitmap tool.

* Constrain perlin tool table cache.

* Move graphics tools to sstudio.graphics.

* Update terminology

* Warn if map source cannot be resolved.

* Rename method to clarify use.

* Fix observation issues.

* Updated changes.

* Add offroad noise.

* Add loop driving example.

* Initial gpu texture generation.

* Update shaders.

* Prepare geometry for base solution.

* Add headless display back to options.

* Update faq.

* Investigate offscreen cameras.

* Update configuration.

* Update

* Update shader step with padding information.

* Update.

* Fix logical errors.

* Fix missing observation and style issues.

* Update observations.

* Hook up variable custom render dependencies.

* Change blank space to green.

* Scale simplex noise in simplex shader.

* Fix formatting.

* Optimize waypoint conversion.

* Formatting and cleanup.

* Fix math utility function.

* Fix type return error with interpolate_waypoints.

* Remove redundant method.

* Fix type checking.

* Remove cpu version of perlin noise generator.

* Fix broken import.

* Fix more types issues.

* Switch on engine debug mode.

* Fix bug with configuration utility.

* make format

* Add buffer configuration options.

* Pass buffer dependencies.

* Plan next change.

* Add types information to sensor.

* Add the current render look.

* Solve some docs errors.

* Update mask example to use constant dependency.

* Reduce lidar array allocations.

* Remove rogue print.

* Determine shader input types.

* Fix bug with rendering on smarts reset.

* Embed pybullet frequency default.

* Generalize offscreen camera update signature.

* Add updating buffers.

* Force low dimension observations to use standard
 python types instead of numpy.

* Fix renderer test.

* Ensure waypoint types types are exact.

* Add remaining observation buffers.

* Send only observation buffers that are in use.

* Update mask.py.

* Update shader file names.

* Add shader uniforms test.

* Update engine configuration documentation.

* Further update documenation

* Fix benchmark examples.

* Remove rendundant utilities

* Fix test

* Update agent documentation.

* Add missing uniform to test.

* Rename to fix variable names.

* Remove unused graphics.

* Fix tuple addition.

* Fix test failure.

* Close in on release.

* Fix doc.

* Fix type check in renderer.

* Fix type errors.

* Fix docs tests.

* Add missing docstring.

* Fix final type issues.

* Fix point distance check.

* Fix shutdown test.

* Run renderer test.

* Update dockerfiles to add renderer dockerfile.

* Update install options.

* Log renderer backend.

* Restore renderer test.

* Enable model caching.

* Use software render for tests.

* Debug rendering.

* Update dockerfiles.

* Remove unused files.

* Try running xorg in earlier step.

* Add test duration information to pytest commands.

* Remove model cache optimization.

* Fix ray and parallel sensor resolvers.

* Update faq.

* Update engine configuration.

* Fix broken configuration.

* Make final changes.

* Add missing details to changelog.

* Fix docs test.

* Remove document reference.

* Fix sensor assignment bug.

.dockerignore

@@ -34,12 +34,14 @@
 **/.mypy_cache/
 **/nosetests.xml
 **/OpEn_build/
+**/.panda3d_cache
 **/pip-delete-this-directory.txt
 **/pip-log.txt
 **/pip-wheel-metadata/
 **/__pycache__
 **/.pytest_cache
 **/.python-version
+**/.pytype
 **/*.rou.alt.xml
 **/*.rou.xml
 **/*.sif
@@ -54,7 +56,7 @@
 **/.tox/
 **/traffic_histories.pkl
 **/*.trips.xml
-**/.venv
+**/.venv*
 **/.vscode
 **/wheels/
 **/xdummy.log

.github/workflows/ci-auto-commit-linux.yml

@@ -33,7 +33,7 @@ jobs:
           . ${{env.venv_dir}}/bin/activate
           pip install --upgrade pip
           pip install wheel==0.38.4
-          pip install .[camera_obs,rllib,test,torch,train]
+          pip install .[camera-obs,rllib,test,torch,train]
       - name: Update requirements
         run: |
           . ${{env.venv_dir}}/bin/activate

.github/workflows/ci-auto-commit-mac.yml

@@ -28,7 +28,7 @@ jobs:
           . ${{env.venv_dir}}/bin/activate
           pip install --upgrade pip
           pip install wheel==0.38.4
-          pip install .[camera_obs,rllib,test,torch,train]
+          pip install .[camera-obs,rllib,test,torch,train]
           pip freeze | grep -v 'smarts' | grep -v 'pkg-resources==0.0.0' > utils/setup/mac_requirements.txt
       - name: Commit changes
         uses: EndBug/add-and-commit@v7

.github/workflows/ci-base-tests-linux.yml

@@ -9,7 +9,7 @@ jobs:
   base-tests:
     runs-on: ubuntu-20.04
     if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
-    container: ghcr.io/smarts-project/smarts:v0.6.1-minimal
+    container: ghcr.io/smarts-project/smarts:v2.0.0-software_render
     strategy:
       matrix:
         tests:
@@ -30,10 +30,11 @@ jobs:
           . ${{env.venv_dir}}/bin/activate
           pip install --upgrade pip
           pip install wheel==0.38.4
-          pip install -e .[camera_obs,opendrive,test,test_notebook,torch,train,gif_recorder,gymnasium,argoverse,envision,sumo]
+          pip install -e .[camera-obs,opendrive,test,test-notebook,torch,train,gif-recorder,gymnasium,argoverse,envision,sumo]
           if echo ${{matrix.tests}} | grep -q -e "test_rllib_hiway_env.py"; then pip install -e .[rllib]; fi
           if echo ${{matrix.tests}} | grep -q -e "test_examples.py"; then pip install -e .[examples,rllib]; fi
           if echo ${{matrix.tests}} | grep -q -e "/smarts/ray"; then pip install -e .[ray]; fi
+          if echo ${{matrix.tests}} | grep -q -e "/smarts/core"; then (/usr/bin/Xorg -noreset +extension GLX +extension RANDR +extension RENDER -logfile ./xdummy.log -config /etc/X11/xorg.conf -novtswitch :1 &); fi
       - name: Build scenarios
         run: |
           . ${{env.venv_dir}}/bin/activate
@@ -55,6 +56,7 @@ jobs:
             --doctest-modules \
             --forked \
             --dist=no \
+            --durations=10 \
             -n auto \
             --ignore-glob="**/ros.py" \
             --ignore-glob="**/waymo_map.py" \
@@ -85,7 +87,7 @@ jobs:
           . ${{env.venv_dir}}/bin/activate
           pip install --upgrade pip
           pip install wheel==0.38.4
-          pip install -e ./../../.[camera_obs,argoverse,sumo,test]
+          pip install -e ./../../.[camera-obs,argoverse,sumo,test]
           pip install -e ./inference/
       - name: Run smoke tests
         run: |
@@ -117,7 +119,7 @@ jobs:
           . ${{env.venv_dir}}/bin/activate
           pip install --upgrade pip
           pip install wheel==0.38.4
-          pip install -e .[camera_obs,argoverse,test,ray,sumo]
+          pip install -e .[camera-obs,argoverse,test,ray,sumo]
           scl zoo install examples/${{matrix.tests}}/inference
       - name: Run smoke tests
         run: |

.github/workflows/ci-base-tests-mac.yml

@@ -21,7 +21,7 @@ jobs:
           - ./examples/tests --ignore=./examples/tests/test_learning.py
           - ./smarts/sstudio
           - ./smarts/env/tests/test_rllib_hiway_env.py
-          - ./smarts/core --nb-exec-timeout 65536 --ignore=./smarts/core/tests/test_notebook.py
+          - ./smarts/core --nb-exec-timeout 65536 --ignore=./smarts/core/tests/test_notebook.py --ignore=./smarts/core/tests/test_renderers.py::test_custom_shader_pass_buffers
           - ./smarts/env --ignore=./smarts/env/tests/test_rllib_hiway_env.py
           - ./smarts/ray
     steps:
@@ -46,7 +46,7 @@ jobs:
           pip install --upgrade pip
           pip install wheel==0.38.4
           pip install -r utils/setup/mac_requirements.txt
-          pip install -e .[camera_obs,opendrive,rllib,test,test_notebook,torch,train,argoverse,envision,sumo]
+          pip install -e .[camera-obs,opendrive,rllib,test,test-notebook,torch,train,argoverse,envision,sumo]
           if echo ${{matrix.tests}} | grep -q -e "/env"; then pip install -e .[rllib]; fi
           if echo ${{matrix.tests}} | grep -q -e "/examples"; then pip install -e .[examples,rllib]; fi
           if echo ${{matrix.tests}} | grep -q "/ray"; then pip install -e .[ray]; fi

.github/workflows/ci-format.yml

@@ -59,7 +59,7 @@ jobs:
           python3.8 -m venv ${{env.venv_dir}}
           . ${{env.venv_dir}}/bin/activate
           pip install wheel==0.38.4
-          pip install -e .[dev,camera_obs,train,test]
+          pip install -e .[dev,camera-obs,train,test]
       - name: Get changed files on branch since branching
         id: changed-files
         shell: bash
@@ -98,7 +98,7 @@ jobs:
           . ${{env.venv_dir}}/bin/activate
           pip install --upgrade pip "setuptools<58.3.0"
           pip install wheel==0.38.4
-          pip install .[camera_obs,doc,train,ray,envision,argoverse,opendrive,waymo,sumo]
+          pip install .[camera-obs,doc,train,ray,envision,argoverse,opendrive,waymo,sumo]
           cd ${GITHUB_WORKSPACE}/docs
           make html SPHINXOPTS="-W -T -E -n --keep-going -b spelling -b linkcheck"
       - name: Check build output

.github/workflows/ci-pull-request.yml

@@ -22,7 +22,7 @@ jobs:
           . ${{env.venv_dir}}/bin/activate
           pip install --upgrade pip
           pip install wheel==0.38.4
-          pip install .[camera_obs,rllib,test,torch,train]
+          pip install .[camera-obs,rllib,test,torch,train]
       - name: SMARTS benchmark
         run: |
           cd $GITHUB_WORKSPACE
@@ -32,6 +32,6 @@ jobs:
           scl scenario build-all --clean ./scenarios
           pytest --benchmark-save=previous --benchmark-min-rounds=10 --benchmark-timer=time.process_time ./smarts/env/tests/test_benchmark.py
           git checkout -
-          pip install .[camera_obs,rllib,test,torch,train]
+          pip install .[camera-obs,rllib,test,torch,train]
           scl scenario build-all --clean ./scenarios
           pytest --benchmark-compare=0001_previous --benchmark-compare-fail=mean:10% --benchmark-min-rounds=10 --benchmark-timer=time.process_time ./smarts/env/tests/test_benchmark.py

.github/workflows/ci-python-version-test.yml

@@ -27,4 +27,4 @@ jobs:
           . ${{env.venv_dir}}/bin/activate
           pip install --upgrade pip
           pip install wheel==0.38.4
-          pip install .[camera_obs,rllib,sumo,test,torch,train]
+          pip install .[camera-obs,rllib,sumo,test,torch,train]

.github/workflows/ci-test-learning.yml

@@ -27,7 +27,7 @@ jobs:
           . ${{env.venv_dir}}/bin/activate
           pip install --upgrade pip
           pip install wheel==0.38.4
-          pip install .[camera_obs,rllib,test,torch,train]
+          pip install .[camera-obs,rllib,test,torch,train]
       - name: Verify learning
         run: |
           cd $GITHUB_WORKSPACE

.github/workflows/ci-test-long-determinism.yml

@@ -26,7 +26,7 @@ jobs:
           python3.8 -m venv ${{env.venv_dir}}
           . ${{env.venv_dir}}/bin/activate
           pip install --upgrade pip
-          pip install .[camera_obs,rllib,test,torch,train]
+          pip install .[camera-obs,rllib,test,torch,train]
       - name: Verify long determinism
         run: |
           cd $GITHUB_WORKSPACE

.github/workflows/ci-test-memory-growth.yml

@@ -28,7 +28,7 @@ jobs:
           pip install --upgrade pip
           pip install wheel==0.38.4
           pip install pympler
-          pip install .[camera_obs,rllib,test,torch,train]
+          pip install .[camera-obs,rllib,test,torch,train]
       - name: Test memory growth
         run: |
           cd $GITHUB_WORKSPACE

.gitignore

@@ -66,6 +66,9 @@ target/
 # VSCode
 .vscode
 
+# panda3d
+.panda3d_cache
+
 # pyenv
 .python-version
 
@@ -153,4 +156,5 @@ collected_observations/
 **/diagnostic/reports/*
 
 # Experiments
-outputs/
+outputs/
+vaw/

.readthedocs.yaml

@@ -22,7 +22,7 @@ python:
     - method: pip
       path: .
       extra_requirements:
-        - camera_obs
+        - camera-obs
         - doc
         - train
         - ray

CHANGELOG.md

@@ -14,6 +14,8 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - The following methods now exist explicitly `Vehicle.{add_sensor|detach_sensor|subscribed_to|sensor_property|}`.
 - Resources loaded with `load_yaml_config_with_substitution()` now substitute in SMARTS configuration with single squiggle bracket `${}` syntax. This is currently restricted to environment variable names prefixed with `"SMARTS_"`. This extends to benchmark configuration and vehicle configuration.
 - Default vehicle definitions can be now modified using `assets:default_vehicle_definitions_list`/`SMARTS_ASSSETS_DEFAULT_VEHICLE_DEFINITIONS_LIST` or by providing a `vehicle_definitions_list.yaml` in the scenario. These vehicle types are related to the `AgentInterface.vehicle_type` attribute.
+- New `CustomRender` agent interface option added. This allows using `glsl` fragment scripts to generate images from camera textures and simulation buffers.
+- New `ObfuscationMap` agent interface option added. This uses the `OccupancyGridMap` to help generate an image of ground viewable area from the ego vehicle's perspective.
 - There is now a centralized `TraCI` server mananger that can be used to prevent port collisions. It can be run using `python smarts.core.utils.sumo_server` and the use of the server can be enabled with `SMARTS_SUMO_TRACI_SERVE_MODE="central"`.
 ### Changed
 - `VehicleIndex.build_agent_vehicle()` no longer has `filename` and `surface_patches` parameters.
@@ -30,6 +32,12 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - `Vehicle.build_agent_vehicle()` and `Vehicle.build_social_vehicle()` moved to `VehicleIndex`.
 - `smarts.core.configuration.Configuration.get_settings()` now uses the `PyYAML` default instead of forcefully casting to `str`.
 - Added `AgentInterface.vehicle_class` which allows selection of a dynamics vehicle from the vehicle definitions list file.
+- Waypoints now have a `position` property (which will eventually replace `pos`).
+- You must now implement `act()` for any agent inheriting from `smarts.core.agent.Agent`.
+- `FunctionAgent` is now no longer dynamically defined.
+- `Vias.hit_via_points` is now a property.
+- `ViaPoint` now has an attribute `hit` which determines if the point has been "collected".
+- Dependencies switched back to using `-` instead of `_` (e.g. "camera-obs").
 ### Deprecated
 - Module `smarts.core.models` is now deprecated in favour of `smarts.assets`.
 - Deprecated a few things related to vehicles in the `Scenario` class, including the `vehicle_filepath`, `tire_parameters_filepath`, and `controller_parameters_filepath`. The functionality is now handled through the vehicle definitions.
@@ -57,6 +65,8 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - The via sensor and trip meter sensor now work without a mission.
 - Fixed a bug with `VehicleIndex.attach_sensors_to_vehicle()` that would generate an invalid plan.
 - Fixed a bug where vehicle sensor meta attributes would reference the wrong vehicle.
+- Resolved issue with road waypoints not showing waypoints if the horizon was larger than the start of the lane.
+- Fixed an issue where `SMARTS.reset()` would be unable to render cameras.
 - Squashed TraCI "retrying" stdout messages.
 ### Removed
 ### Security
@@ -126,6 +136,8 @@ Copy and pasting the git commit messages is __NOT__ enough.
 - Interest vehicles now show up in Envision.
 - Seed of `hiway-v1` env can be retrieved through a new property `seed`.
 - Added `TrafficEngineActor` to describe a scenario studio defined actor that is controlled by a traffic engine.
+- Docker images from now on out base from ``ubuntu:focal``.
+- A new Docker image has been added for software rendering for use cases where there is no display and GPU.
 ### Changed
 - Changed waypoints in sumo maps to use more incoming lanes into junctions.
 - Increased the cutoff radius for filtering out waypoints that are too far away in junctions in sumo maps.

Makefile

@@ -7,6 +7,7 @@ test: build-all-scenarios
 		--doctest-modules \
 		--forked \
 		--dist=loadscope \
+		--durations=0 \
 		-n `expr \( \`nproc\` \/ 2 \& \`nproc\` \> 3 \) \| 2` \
 		--nb-exec-timeout 65536 \
 		./examples/tests ./smarts/env ./envision ./smarts/core ./smarts/sstudio \
@@ -48,6 +49,7 @@ test-long-determinism:
 	scl scenario build --clean scenarios/sumo/minicity
 	PYTHONHASHSEED=42 pytest -v \
 		--forked \
+		--durations=0 \
 		./smarts/env/tests/test_determinism.py::test_long_determinism
 
 .PHONY: test-memory-growth
@@ -63,7 +65,7 @@ test-memory-growth: build-all-scenarios
 
 .PHONY: benchmark
 benchmark: build-all-scenarios
-	pytest -v ./smarts/env/tests/test_benchmark.py
+	pytest -v --durations=0 ./smarts/env/tests/test_benchmark.py
 
 .PHONY: test-zoo
 test-zoo: build-all-scenarios

docs/benchmarks/driving_smarts_2023_1.rst

@@ -256,7 +256,7 @@ Train
     $ source ./.venv/bin/activate
     $ pip install --upgrade pip
     $ pip install wheel==0.38.4
-    $ pip install -e ./../../.[camera_obs,argoverse,envision,sumo]
+    $ pip install -e ./../../.[camera-obs,argoverse,envision,sumo]
     $ pip install -e ./inference/
 
 + Train locally without visualization
@@ -307,7 +307,7 @@ Evaluate
     $ source ./.venv/bin/activate
     $ pip install --upgrade pip
     $ pip install wheel==0.38.4
-    $ pip install -e .[camera_obs,argoverse,envision,sumo]
+    $ pip install -e .[camera-obs,argoverse,envision,sumo]
     $ scl zoo install examples/e10_drive/inference
     # For Driving SMARTS 2023.1
     $ scl benchmark run driving_smarts_2023_1 examples.e10_drive.inference:contrib-agent-v0 --auto-install

docs/benchmarks/driving_smarts_2023_3.rst

@@ -234,7 +234,7 @@ Train
     $ source ./.venv/bin/activate
     $ pip install --upgrade pip
     $ pip install wheel==0.38.4
-    $ pip install -e ./../../.[camera_obs,argoverse,envision,sumo]
+    $ pip install -e ./../../.[camera-obs,argoverse,envision,sumo]
     $ pip install -e ./inference/
 
 + Train locally without visualization
@@ -285,7 +285,7 @@ Evaluate
     $ source ./.venv/bin/activate
     $ pip install --upgrade pip
     $ pip install wheel==0.38.4
-    $ pip install -e .[camera_obs,argoverse,envision,sumo]
+    $ pip install -e .[camera-obs,argoverse,envision,sumo]
     $ scl zoo install examples/e11_platoon/inference
     $ scl benchmark run driving_smarts_2023_3 examples.e11_platoon.inference:contrib-agent-v0 --auto-install
 

docs/conf.py

@@ -60,7 +60,6 @@
         "%s",
     ),
 }
-
 # Configuring automated generation of api documentation.
 # See: https://github.com/sphinx-contrib/apidoc
 apidoc_module_dir = ".."
@@ -72,6 +71,8 @@
     "scenarios",
     "smarts/ros",
     "zoo/policies/interaction_aware_motion_prediction",
+    "smarts/waymo/waymo_open_dataset/protos",
+    "zoo/evaluation/metrics",
 ]
 apidoc_extra_args = [
     "--force",
@@ -126,6 +127,7 @@
     ("py:class", "ObsType"),
     ("py:class", "smarts.env.gymnasium.wrappers.metric.utils.T"),
     ("py:class", "enum.Enum"),
+    ("py:class", "bc.BulletClient"),
 }
 nitpick_ignore_regex = {
     (r"py:.*", r"av2\..*"),

docs/resources/faq.rst

@@ -11,19 +11,33 @@ This is a list of frequently asked questions.  Feel free to suggest new entries!
 2. Where can I find debug logs?
     In most cases SMARTS debug logs are located at ``~/.smarts``. These can be helpful to diagnose problems.
 
-3. Exception: Could not open window.
-    This may be due to some old dependencies of ``Panda3D``. Try the following instructions to solve it.
+3. Exception: Could not open display. (Ubuntu)
+    This may be due to needing a display to render with a ``GL`` renderer backend setting. Try the following instructions to solve it.
 
     .. code-block:: bash
 
-        # Set DISPLAY 
-        $ vim ~/.bashrc
-        $ export DISPLAY=":1"
+        # Set DISPLAY, can be as needed
+        $ echo export DISPLAY=":1" >> ~/.bashrc
         $ source ~/.bashrc
 
-        # Set xorg server
+        # Do once: Install x11 dummy which allows creating a fake display
+        $ sudo apt-get install -y xserver-xorg-video-dummy x11-apps
+        # Potentially the following if you need software rendering:
+        # sudo apt-get install -y mesa-utils
+
+        # Do once: set xorg server
         $ sudo wget -O /etc/X11/xorg.conf http://xpra.org/xorg.conf
-        $ sudo /usr/bin/Xorg -noreset +extension GLX +extension RANDR +extension RENDER -logfile ./xdummy.log -config /etc/X11/xorg.conf $DISPLAY & 0
 
-4. The simulation keeps crashing on connection in ``SumoTrafficSimulation``. How do I fix this?
+        # Do as needed:
+        $ sudo /usr/bin/Xorg -noreset +extension GLX +extension RANDR +extension RENDER -logfile ./xdummy.log -config /etc/X11/xorg.conf $DISPLAY &
+
+
+    Note that ``mesa-utils`` installs ``llvm``, which is one option out of several that emulate ``OpenGL`` using software. ``llvm`` is not needed if a GPU is available.
+
+4. Custom rendering and Obfuscation maps show completely blank. (Ubuntu)
+    This is due to needing ``OpenGL`` to render using scripts. If you have a GPU make sure ``OpenGL`` is installed and the GPU has the necessary drivers for rendering. 
+
+    See the previous question if you need software rendering.
+
+5. The simulation keeps crashing on connection in ``SumoTrafficSimulation``. How do I fix this?
     This is likely due to using large scale parallelization. You will want to use the centralized management server. See :ref:`centralized_traci_management`.

docs/setup.rst

@@ -35,7 +35,7 @@ This includes SMARTS but none of the examples.
     $ bash utils/setup/install_deps.sh
 
     # This should install the latest version of SMARTS from package index (generally PyPI).
-    $ pip install 'smarts[camera_obs,sumo,example]'
+    $ pip install 'smarts[camera-obs,sumo,example]'
 
 
 Development
@@ -79,13 +79,13 @@ Run the following commands to setup the SMARTS simulator.
     $ pip install --upgrade pip
 
     # Install smarts with extras as needed. Extras include the following: 
-    # `camera_obs` - needed for rendering camera observations, and for testing.
+    # `camera-obs` - needed for rendering camera observations, and for testing.
     # `sumo` - needed for using SUMO scenarios.
     # `test` - needed for running tests.
     # `example` - needed for running examples.
     # `--config-settings editable_mode=strict` - may be needed depending on version of setuptools. 
     #      See https://github.com/huawei-noah/SMARTS/issues/2090.
-    $ pip install -e '.[camera_obs,sumo,test,example]' --config-settings editable_mode=strict
+    $ pip install -e '.[camera-obs,sumo,test,example]' --config-settings editable_mode=strict
 
     # Run sanity-test and verify they are passing.
     # If tests fail, check './sanity_test_result.xml' for test report.