diff --git a/site/docs/connectors/unity-catalog/unity-catalog.drawio b/site/docs/connectors/unity-catalog/unity-catalog.drawio index fdc64f88e4..fa82d23362 100644 --- a/site/docs/connectors/unity-catalog/unity-catalog.drawio +++ b/site/docs/connectors/unity-catalog/unity-catalog.drawio @@ -127,8 +127,8 @@ - - + + @@ -145,17 +145,17 @@ - - + + - + - + - + diff --git a/site/docs/user-interfaces/mermaid/daily-goverance-action-process-definition.png b/site/docs/user-interfaces/mermaid/daily-goverance-action-process-definition.png deleted file mode 100644 index 57bcca6109..0000000000 Binary files a/site/docs/user-interfaces/mermaid/daily-goverance-action-process-definition.png and /dev/null differ diff --git a/site/docs/user-interfaces/mermaid/daily-governance-action-process-definition.png b/site/docs/user-interfaces/mermaid/daily-governance-action-process-definition.png new file mode 100644 index 0000000000..b9afe7a499 Binary files /dev/null and b/site/docs/user-interfaces/mermaid/daily-governance-action-process-definition.png differ diff --git a/site/docs/user-interfaces/mermaid/daily-governance-action-process-instance.png b/site/docs/user-interfaces/mermaid/daily-governance-action-process-instance.png index 315f7a9cbb..6f0ce5ab3e 100644 Binary files a/site/docs/user-interfaces/mermaid/daily-governance-action-process-instance.png and b/site/docs/user-interfaces/mermaid/daily-governance-action-process-instance.png differ diff --git a/site/docs/user-interfaces/mermaid/lineage-graph-coco-sus.png b/site/docs/user-interfaces/mermaid/lineage-graph-coco-sus.png new file mode 100644 index 0000000000..1275f184c8 Binary files /dev/null and b/site/docs/user-interfaces/mermaid/lineage-graph-coco-sus.png differ diff --git a/site/docs/user-interfaces/mermaid/lineage-graph-unity-default.png b/site/docs/user-interfaces/mermaid/lineage-graph-unity-default.png new file mode 100644 index 0000000000..dcd926e81e Binary files /dev/null and b/site/docs/user-interfaces/mermaid/lineage-graph-unity-default.png differ diff --git a/site/docs/user-interfaces/mermaid/overview.md b/site/docs/user-interfaces/mermaid/overview.md index a1fdfceb12..c4df0708dc 100644 --- a/site/docs/user-interfaces/mermaid/overview.md +++ b/site/docs/user-interfaces/mermaid/overview.md @@ -1,9 +1,10 @@ -# Mermaid diagrams +# Graph Visualization -[Mermaid](https://mermaid.js.org/) is a markdown extension for representing diagram using structured text that can then be rendered in a JavaScript environment. Selected [Open Metadata View Services (OMVSs)](/services/omvs) return a Mermaid markdown string that can be used to render a visual representation of the elements returned on the same request. This markdown can be rendered in a JavaScript environment that supports the Mermaid markdown, such as [JupyterLab Notebooks](#jupyterlab-notebooks) and [Draw.io](#drawio). +Egeria visualizes collections of related metadata elements as graphs using Mermaid. +[Mermaid](https://mermaid.js.org/) is a markdown extension for representing diagram using structured text that can then be rendered in a JavaScript environment. Selected [Open Metadata View Services (OMVSs)](/services/omvs) REST API calls return a Mermaid markdown string that can be used to render a visual representation of the elements returned on the same request. ## Asset graphs @@ -46,32 +47,54 @@ The asset is on the far left-hand side. Each box on the graph is a metadata elem } ``` + +## Lineage graphs + +[Asset Catalog OMVS](/services/omvs/asset-catalog/overview) also supports a REST API called *getAssetLineageGraph(assetGUID)* which returns a graph of metadata elements that are linked via lineage relationships with the requested asset. In addition it returns a Mermaid string that can be used to visualise this structure. Below is a screen capture of a rendered lineage graph. + +![Lineage graph showing data flows](lineage-graph-coco-sus.png) + +The graph above shows [DataFlow](/types/7/0750-Data-Passing) relationships between systems. The next image shows a lineage graph of the data sources that make up the resources in a Unity Catalog schema. It is showing [DataContentForDataSet](/types/2/0210-Data-Stores) relationships. + +![lineage graph showing data dependencies](lineage-graph-unity-default.png) + +The full list of relationships supported are: + +* [DataFlow](/types/7/0750-Data-Passing) - how is data flowing +* [ProcessCall](/types/7/0750-Data-Passing) - which system is calling another +* [ControlFlow](/types/7/0750-Data-Passing) - how is control passed between processes that are under the control of a workflow manager +* [LineageMapping](/types/7/0770-Lineage_Mapping) - how do elements catalogued from different lineage systems relate to one another +* [DataContentForDataSet](/types/2/0210-Data-Stores) - how are data sources combined to create a data set +* [DerivedSchemaTypeQueryTarget](/types/5/0512-Derived-Schema-Elements) - what are the queries used to create derived data fields and views + +You can specify any number of these relationship type on the query. If you specify no relationships, it searches for any of these relationships. + ## Governance Action Process graphs -[Governance Action Processes](/concepts/governance-action-process) describe a linked series of steps that run in an [engine host](/concepts/engine-host). Their definition is maintained in the open metadata repositories. In addition, when a governance action process runs, a [*GovernanceActionProcessInstance*](/type/4/0462-Governance-Action-Processes) element is created and linked to the [engine actions](/concepts/engine-action) that controls the execution of the process. +[Governance Action Processes](/concepts/governance-action-process) describe a linked series of steps that run in an [engine host](/concepts/engine-host). Their definition is maintained in the open metadata repositories. In addition, when a governance action process runs, a [*GovernanceActionProcessInstance*](/types/4/0462-Governance-Action-Processes) element is created and linked to the [engine actions](/concepts/engine-action) that controls the execution of the process. The [Automated Curation OMVS](/services/omvs/automated-curation/overview) has a REST API called *getGovernanceActionProcessGraph(processGUID)*. It can either be passed the unique identifier of a governance action process element, or a governance action process instance element. The mermaid graph returned for these processes is as follows. This first image shows the sample *DailyGovernanceActionProcess* which determines the day of the week and runs a specific task for that day. Each box is a step in the process. The relationships show possible links to the next step to run. The label on the relationship shows which [guard](/concepts/guard) must be produced to cause the linked step to run. The *wait_time* is the minutes to wait before running the step and *mult_trig* indicates whether the step can be triggered multiple times in the same process instance, or not. -![DailyGovernanceActionProcess definition](daily-goverance-action-process-definition.png) +![DailyGovernanceActionProcess definition](daily-governance-action-process-definition.png) The next image show a single execution of this process. Notice that the shape is different because each time it runs, it only outputs a single task. ![DailyGovernanceActionProcess instance](daily-governance-action-process-instance.png) -## Lineage graphs - -*Work in progress* - ## Rendering mermaid diagrams Mermaid is a mature standard that has widespread support. Below are some examples of where it can be used. -### JupyterLab Notebooks +### Jupyter Notebooks -If you are using *JupyterLab Notebooks* with [pyegeria](user-interfaces/python-widgets/overview/#installation) installed, such as with the [egeria-workspaces](https://github.com/odpi/egeria-workspaces), it is possible to render Mermaid strings using the `render_mermaid(mermaid_string)` function call. +If you are using *Jupyter Notebooks* with [pyegeria](user-interfaces/python-widgets/overview/#installation) installed, such as with the [egeria-workspaces](https://github.com/odpi/egeria-workspaces), it is possible to render Mermaid strings using the `render_mermaid(mermaid_string)` function call. ### Draw.io The *draw.io* diagramming tool used extensively in this site can render Mermaid diagrams. It is described in the [draw.io help](https://www.drawio.com/blog/mermaid-diagrams). +### Mermaid live editor + +The Mermaid community provide a web page where you can paste a mermaid string, and it will render it dynamically. The live editor can be found at [https://mermaid.live/edit](https://mermaid.live/edit) + --8<-- "snippets/abbr.md" \ No newline at end of file diff --git a/site/mkdocs.yml b/site/mkdocs.yml index ac30832d11..2d265743cf 100644 --- a/site/mkdocs.yml +++ b/site/mkdocs.yml @@ -210,7 +210,7 @@ nav: - User Interfaces: - User Interface overview: user-interfaces/index.md - Python Widgets: user-interfaces/python-widgets/overview.md - - Mermaid Diagrams: user-interfaces/mermaid/overview.md + - Graph Visualization: user-interfaces/mermaid/overview.md - Explorers: user-interfaces/brain-explorers/overview.md - Features: - Features Overview: features/index.md @@ -694,7 +694,6 @@ nav: - Favorite Collections: concepts/favorite-things-collection.md - Feedback: concepts/feedback.md - File Type: concepts/file-type.md - - Glossary Term: practices/common-data-definitions/anatomy-of-a-glossary.md - GUID: concepts/guid.md - Governance Action: concepts/governance-action.md - Governance Action Engine: concepts/governance-action-engine.md