Extract subgraphs in visualisation

[jl-wynen]

Some subgraphs don't need to be visualised in detail. For example, the metadata-related graph in scipp/essreflectometry#27 is relatively large to make it customisable. But, as long as it contains the default providers, there is no need to show it in its full glory in pipeline.visualize. We could contract it into a single node and only show that.

If we support showing the contracted subgraph separately, we wouldn't even lose information but would allow the user to 'zoom out' a little. There are also other graphs that contain subgraphs that could conceptually be represented by a single node.

Supporting this would require identifying contractable graphs. This needs domain knowledge, i.e., configuration by a human. One possibility would be the tag providers and / or data types and group and contract based on those tags.

To be clear, I'm only talking about visualisation, the actual task graph would remain unchanged.

Example

Only showing providers for brevity.

graph LR;
load-->filter_[filter];
filter_-->reduce_[reduce];
reduce_-->save_[save];
load-->extract_meta;
extract_meta-->bundle_meta;
bundle_meta-->save;
reduce_-->bundle_meta;

Loading

Could become

graph TD;
subgraph main
load-->filter_[filter];
filter_-->reduce_[reduce];
reduce_-->save_[save];
load-->build_meta;
build_meta-->save_;
reduce_-->build_meta;
end

subgraph extracted_meta_sub
extract_meta-->bundle_meta;
end

Loading

I'm not sure what's the best way of indicating inputs and outputs of subgraphs. We could, e.g. duplicate all connects nodes from the main graph in the subgraph.

[SimonHeybrock]

Work on Cyclebane may solve some cases of this (not via subgraphs), as we want to support __getitem__ on task graphs. Examples:

graph[TransmissionFraction]  # get the branch computing the transmission fraction (with all ancestors)
graph[RawMonitor:WavelengthMonitor]  # get everything in path between start and stop, including ancestors (except those of RawMonitor)

[jl-wynen]

How does this relate to visualisation?

[SimonHeybrock]

If you extract a subgraph, you can visualize the subgraph, instead of everything.

[jl-wynen]

This is the opposite of my suggestion. I want to visualise the large graph but simplify subgraphs to make the overall structure more readable.