Plan to Restructure the O3DE Engine Programming Guide

[chanmosq]

This project originated from a SIG Docs and Community meeting: #15 (comment)

---

The goal is to discuss the proposal to restructure the O3DE Engine Programmer Guide that requires an action plan from the Documentation and Community Special Interest Group (docs-community-sig).

Users can't find information related to engine programming in the O3DE Docs. The O3DE Amazon Documentation team searched through many pages before we found engine programming docs that were supposedly missing, but were actually divided throughout the O3DE Docs. Also, the community's questions in discord are often responded with a link to existing docs.

We should begin realigning the documentation structure for engine developers because we expect more engine programming docs contributions soon. This will makes it easier for users to find docs and for contributors to author docs in a suitable place.

Considerations in the design of O3DE Docs

User roles

O3DE users' learning goals are the primary motivation for determining how to organize the O3DE Docs. There are three common user roles when using O3DE:

• Artist
• Designer
• Developer

The artist and designer both use the O3DE Docs to learn how to use existing tools and functions in the engine to create a product. The developer uses O3DE Docs to learn how to develop those tools and functions. Documentation for a developer contains more technical detail than an artist or designer should be concerned about. To focus the learning experience for each role, the O3DE Docs should organize information for O3DE users or O3DE developers.

Design problems

Current docs structure

Engine programming information coexists with user information in different sections of the O3DE Docs.

- Welcome Guide
  - Key Concepts
    - Entry point APIs
    - O3DE source code structure
    - Engine systems graph
    - Working with Gems
    - ... more programming concepts
- User Guide
  - Gems
    - Development
  - Code
    - Development
  - Engine Core
    - Intro to C++, Memory management, Event System, and more
- API Reference

Discussion

<to discuss>

For each of these topics:

• Does the section they are under make sense?
• Why should or shouldn't they be in this section?
• What is an alternative section to put them under? Why should or shouldn't they be in that alternative section?

Section name

The section dedicated to engine programming information is called "Engine Core".

Discussion

<to discuss>

• Is this a suitable and non-ambiguous name?

Possible solution: A separate guide for engine programming topics

This solution centralizes engine programming information into a single place in the O3DE docs.

The "Engine Programmer Guide" is for developers who want to change or extend the engine's functionalities. All of the information in this guide teaches developers how to do that, specifically:

• Engine architecture, its many systems, and how they operate
• Core tools and applications that make up O3DE
• Core Gems that the engine depends on
• An introduction to programming C++ for O3DE
• Gem APIs and how to create them
• Component APIs and how to create them

Arguments

<to discuss>

Additional possible solutions

<to discuss>

Action plan

<to discuss>

[FiniteStateGit]

I gathered from the Lumberyard community that there was a discoverability issue with separate user and developer guides. Users transitioning to gem and component development overlooked the developer guide, as they were accustomed to the user guide as the engine's source of truth. This issue is somewhat mitigated in O3DE by the top level domain and the presence of header navigation. Personally, the left-side TOC is the only place I habitually look for navigation options, I forget about header links and splash page structure. I only bring this up here because it would move more content away from the user guide TOC, but I wonder if the guides should be collapsed, or site layout changed, so that more site content was visible from a given page's TOC.

Browsing the welcome guide, I noticed that it jumped into engine details geared towards the developer crowd. I could see that content moving to the introductory sections of a developer guide. At the very least, it needs to be balanced with more content to get an artist or developer oriented to the engine. A primary focus of the welcome guide should be to orient all roles to the structure of the documentation and possibly provide a suggested learning approach. It should define what information will and will not be found in a particular guide.

I agree that the engine programming content should be centralized compared to its current distribution, particularly away from topics within the user guide. A dilemma: some of the developer content is the only information available on functionality that artists and designers also use. Topics that might fit into this category: assets, component APIs, input, json serialization, file i/o, Ebus, datatype APIs, cvars. This could also be an opportunity to present these topics on a level appropriate for artists and designers.

Engine Core could use a renaming either way.

I found a few items that might be considered in this discussion. I will edit this list as I can locate them.

• https://o3deorg.netlify.app/docs/tools-ui/
• The first section of https://o3deorg.netlify.app/docs/user-guide/components/reference/gameplay/input-event-bus-interface/, the remainder of the page will be duplicated on the page for the Input component (following the current format for component reference pages).
• https://o3deorg.netlify.app/docs/user-guide/components/reference/shape/vertex-container/
• https://o3deorg.netlify.app/docs/user-guide/interactivity/user-interface/fonts/
• https://o3deorg.netlify.app/docs/user-guide/interactivity/input/

[chanmosq]

10/26/2021 - Meeting notes

New! Programming section

We will revamp the previous Engine core section of the User Guide and call it Programming. This section will inherit and reorganize the topics in the Engine Core section and adopt some new subsections.

Example of what the Programming page's structure might look like:

- Programming
  - Core Frameworks
  - Editor Extensions
  - Memory Management
  - C++ 
  - Python
  - Gem Development
  - Component Development
  - Asset Pipeline Development

Why did we decide to keep the Programming section in the User Guide?

Better discoverability. It's common for some of the guides to not appear on the website when using different resolutions and window sizes. Also, there's already many guides listed in the header nav; adding another guide will cause the guides to move to the hamburger menu (three horizontal lines).

Action items

• Rename the section!

There will be a bunch of other subtasks to clean up the subsections. It's a huge undertaking and should be dealt with in chunks.

Split the Key Concepts page

The purpose of the Key Concepts page is to provide a holistic view of O3DE for all audiences: artists, designers, developers, and everything in between or outside. We've identified sections in this page that need to be broadly expanded to encompass other features in O3DE or reduced and moved to an overview page in the Programming section.

Action items

In the Key Concepts page, we'll restructure the following subsections:

• Intro

  • Edit pass.
  • Create a simplified version of the "structure of project runtimes" diagram for this page and move the original diagram into the Programming section.

• Overview of the O3DE SDK

  We don't need to specify the names and descriptions here. We only need an overview - that O3DE has APIs we can develop tools/features with.

  • Move the list of core modules to the Programming section's overview page.
  • Keep the dependency graph for core modules and examples of "high-level products using these frameworks".
  • Edit content to mention that we'll dive deeper into the core modules in the Programming section.

• O3DE Directory Structure

  • The O3DE directory structure for the source engine and the pre-built SDK is about the same, so no need to differentiate that.

• Working with Gems

  • Keep.
  • Mention that developers can create custom Gems and link to proper page in Programming section.

• The EBus Messaging System

  • Keep and reduce. Artists and designers may need to use the EBus Messaging system if they are scripting in their project, including scripting game play logic and animation.
  • Move technical information better suited for only developers to the Programming section.
  • Mention that developers who are creating custom components or Gems can use the EBus Messaging system, and link to the correct page in the Programming section.
  • Do research on the state of O3DE's EBus Messaging system and AZ::Event framework. Some parts of O3DE uses the EBus Messaging system, while others use the AZ::Event. This page should differentiate the two.

• Component Entity system

  • Keep.

• Asset Pipeline

  • Keep.
  • Mention that the developers can create custom Asset Pipelines and link to the Programming section.

• Scripting for O3DE

  • Keep.
  • Add Python.

• Available Release Runtimes

  • ?

Component and Gem development

Action Item

• Move the Development pages of the Component and Gem sections of the User Guide to the Programming section.

Other action items

• Move all the guides to the left-hand nav. This improves discoverability because users often have to switch between the different guides.

Summary

• Let's create a rename the section "Engine Core" to "Programming" and keep it in the User Guide!
• Let's restructure the Key Concepts page in the Getting Started docs!
• Let's move the Gems > Development page and Components > Development page into the Programming section.

[sptramer]

Some supplementary notes to the above:

• Users mostly go to the Welcome Guide and User Guide at this point

• Information in the user guide is poorly organized - especially for new users

• sig-docs-community/discussions/16

• How do we solve this problem of discoverability and where should we locate these documents?

• Different guides:

  • The Welcome Guide hosts the "key concepts" page, which is currently our hub for the overview of the whole Engine (and programming)
    • Most users won't need this information! Only engine programmers really need it.
      • Scale it back: Give every developer a casual overview of what everything is and how it fits together. Holistic overview.
      • Looks like there's a mix of content on this page - detailed information on how libraries link and load, but also pure-text high-level descriptions of EBuses.
    • Missing information about Python (editor extension/editor driver) scripting
    • Mention Scene importing (Az::Scene)
    • Mention Python asset building
    • Component Entity System (CES) - also important high-level overview of the
    • Look at Colin's talk for the directory structure
    • SDK overview should be focused more on marketing-ish materials, but the library connection is useful for engineers
  • A lot of O3DECon content would be good for Getting Started or surface-level overviews of the available systems

• Should a Programmer Guide live within the user guide, or be its own separate guide?

  • Will's vote: Section in user guide based on past research and user discussions about discoverability
  • LY_DM: Should be located inside the user guide so that there's easy cross-linking
  • stramer: Also adding a new card to the landing page for guides causes a problem
  • VERDICT: Programmer's Guide fits into the User Guide.

• What should it be named in the ToC/Index?

  • "Programming"
    • "Bootstrapping / launching"
    • "Memory managemnt"
    • "Core frameworks"
    • "Networking"
    • "Gems"
    • "Editor extensions"
    • "Asset builders"
    • etc.

• Should we move the content like Gem dev / Component dev to the Programming section?

  • All in attendance agree

[chanmosq]

Based on our discussion, here are the issues to restructure the O3DE Engine Programming Guide:

• [Programming Guide] Restructure "Programming" section directory in User Guide o3de.org#1120
• [Programming Guide] Update Key Concepts page | Intro o3de.org#1121
• [Programming Guide] Update Key Concepts page | Overview of O3DE SDK o3de.org#1122
• [Programming Guide] Update Key Concepts page | Working with Gems o3de.org#1123
• [Programming Guide] Update Key Concepts page | The EBus Message System o3de.org#1124
• [Programming Guide] Update Key Concepts page | Asset Pipeline o3de.org#1125
• [Programming Guide] Update Key Concepts page | Scripting for O3DE o3de.org#1126
• [Programming Guide] Update Key Concepts page | O3DE Directory Structure o3de.org#1127
• [Programming Guide] Update Key Concepts page | Scene Importing o3de.org#1128

[ghost]

---

I'm adding my comments here. This was originally a registered issue #1092, but they want us to gather everything here :)

After reading most of the parts under https://o3de.org/docs/api/ and https://o3de.org/docs/tools-ui/ there are some big gaps in lacking documentation that need some attention.

If we were trying to strive for a certain level on documentation, I believe Unity has some good structure and great examples on how to reach out to different levels of information based on your day to day needs in coding,

https://docs.unity3d.com/ScriptReference/index.html

Sessions that should be highlighted:

• Where to expand your main game project code by creating new classes and writing code (not gems)?

• The Doxygen documentation is very limited. Could we get a revision sometime soon to know what all public functions do, besides names?

• What IDE's are meant to be used (if that really matters), Visual Studio, CLion, other? The CMakeLists.txt files are having commands that do not contain only CMake standard commands? We are getting errors for example when I reload a CMake command and it says, ly_get_list_relative_pal_filename is an unknown command. Please advice in these cases how a developer can proceed with gem development outside the scope of the engine. A tutorial/example on how to initially setup and do something would be great.

• Which parts of the code should a game creator retain in a repository (whether its github, bitbucket, azure devops, etc.)? Do we really need to retain all these Gem cpp/cxx/hxx/h classes around?

• When we compile, do we really need to compile the entire solution (it is a very time sinking process especially if you're making minor changes you need to quickly to test out something), or would certain parts of the solution suffice? We would need a C++ tutorial to explain how to save time by just choosing out which stuff to be compiled without the need to recompile the entire solution to test/debug something more effectively...

• Can I develop Gems under my own project folder or do we have to register and code version them in the same location as other external public Gems? The project is currently spread out all over the place. For example ProjectName.GameLauncher.vcxproj file is retained in a project-centric structure under C:\ProjectFolder\build\windows_vs2019\o3de\Code\LauncherUnified while the LauncherProject.cpp class within the same project is existing in an engine-centric location C:\o3de\Code\LauncherUnified\LauncherProject.cpp could we please get all involved project files in one and the same location of our choice?

• We need sessions to call out for deprecated functions/libraries and potential come-soon features/libraries coming out to replace them. There's a way for functions to be marked as deprecated (soon-to-be-replaced-with...) that could be part of Doxygen documentation (read more, https://docs.microsoft.com/en-us/cpp/cpp/deprecated-cpp?view=msvc-160) That would be enormously helpful for developers to capture deprecated areas in the code and/or mark them with //TODO: comments.

The project structure in Visual Studio 2019,

Thank you in advance,

Sincerely,
Niklas Henricson

[amznestebanpapp]

Can you convert the checkboxes to numbers? I will reply in order here

1. the way things are setup is for you to create at least a gem for you game and put your game logic there. If you need something that cannot be solved within a gem, we should tackle it.

2. that opens a bigger question around public API. Currently is not that well defined.

3. No, and you should not need to maintain O3DE's code for your game. However, its likely that you will want to keep a fork of the engine for your game. In that scenario, the easiest is to keep all of it to be easier to merge/push changes back.
   If you want to filter the things that your team sees, you can remove entries in Gems and just remove those entries from the engine.json file. That removes gems and allows you to simply delete them. Note that the installer includes all the gems in the repository, so you will have to distribute your version of the engine to the rest of the team.

4. No, you dont need to compile the entire solution. You should compile the target that you want/need. For example, if you want to run the Editor, just set it as your startup project, hit play and done. If you want to run the game, just set up <game>.GameLauncher as your startup project, hit run. If you want to run the AzCore unit tests, just setup AzCore.Tests as your startup project and run it. Dependencies are set up properly to compile all the dependencies. Compiling the entire solution means "compile everything", including tests, tools and gems that you are not using. Test targets are setup to run the tests.

5. Yes, you can develop gems inside your project. Your project can actually just be a gem that you are developing. A project doesnt even need to have "levels". The most practical scenario is to develop your gem in some sandbox game where you can test it and validate it within the editor/gamelauncher.

6. Agree. There are currently a couple of things that maybe you are not aware:

• there is a RETIRED_CODE.md at the root of the repository
• there is a AZ_DEPRECATED macro to mark things that are deprecated (and some other ones like DEPRECATE_EBUS
• there is a list of things that will be removed: rest of code in Code/Legacy, Code/GridMate (AzNetworking replaces it)
• there is code that is just old and is useless. For example, I recently removed all the driller code. I also have a PR that finds files that are not being used. These sort of changes improve compilation times and reduce binary sizes.
  Then there are things that as a community we have to define if it is worthy to keep and maintain or not. For example, there have been talks for the LuaIDE to be replaced with VSCode and a plugin. Those kind of decisions will likely need RFCs and approval.