Infinite tree

[kinow]

These changes close #330, and close #346, and close #256 (no recursion - no need for so many events, it's now controlled by the infinite-tree library), and close #227 (memory is about the same initially, but doesn't skyrocket over time), and close #222

Now also closes #403 , as I used it to improve the JS code, and avoid having to build cycle points by looking at all the families (thanks to @dwsutherland 's tip).

Reverts #232 because the offline mode uses cylc client graphql to fetch data, and it doesn't get the __typename, which is set only by the JS ApolloClient.

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Appropriate tests are included (unit and/or functional).
• Appropriate change log entry included.
• No documentation update required.

Pending

• avoid data duplication
• update the template to use the latest layout improvements Fix leaf and other tree nodes layout. #396
• try to add the Infinite Tree as a separate component, instead of replacing the current Tree, and use as an extra Lumino widget
• check again that event listeners are being properly removed, widgets are being removed from DOM and telling the Vue components to be removed too, double check the scroll is working as expected, etc (there was a recent rebase)
• add note about extra application size (~130 KB if I remember well, better double check later how much more we are adding here)
• fix add view menu (I think my rebase/merge messed it and I lost graph/mutations)
• keep user state, instead of overriding it with every websockets update (blocked by deltas UIS Delta subscriptions & GraphQL null stripping back-end cylc-flow#3500 & UIS Delta subscriptions & GraphQL null stripping back-end cylc-uiserver#118)

[kinow]

Benchmarks

Setup

Backend

There are multiple ways to prepare the environment for these tests. I will describe how I prepare my environment. While there are probably many better ways of achieving the same result (e.g. git worktree, conda environments, etc) I tend to use the tools I am more familiar with.

$ cd ~/Development/python/workspace/cylc-uiserver
$ git fetch --all ; git checkout master ; git rebase upstream/master
$ virtualenv venv
$ source venv/bin/activate
$ (venv) pip install -e .

This should install Cylc UI Server and Cylc Flow. I don't change this, and I call it the "backend". I use the same venv while testing multiple Cylc UI branches.

Frontend

I don't use git worktree, but this is probably a good scenario where it would be useful. Here's my workflow.

$ cd ~/Development/python/workspace
$ cp -r cylc-ui cylc-ui-master # I know, I know... I am sorry. Bear with me.
$ cd cylc-ui-master
$ git fetch --all ; git checkout master ; git rebase upstream/master
$ npm ci
$ npm run build # production mode!

Now for the branch that I want to benchmark against.

$ cd ~/Development/python/workspace/cylc-ui
$ git fetch --all ; git checkout infinite-tree ; git rebase kinow/infinite-tree
$ npm ci
$ npm run build # production mode!

Starting two Cylc UI Servers

This is one of the rare cases where I think you will want to run two Cylc UI Servers. I do that occasionally, and because of that have included instructions how to do that in the Cylc UI Server README.md.

# `master`
$ cd ~/Development/python/workspace/cylc-uiserver/
$ source venv/bin/activate
$ (venv) cd /tmp
$ (venv) jupyterhub --JupyterHub.spawner_class="jupyterhub.spawner.LocalProcessSpawner" --Spawner.args="['-s', '/home/kinow/Development/python/workspace/cylc-ui-master/dist/']" --Spawner.cmd="cylc-uiserver" --JupyterHub.bind_url="http://:7000" --JupyterHub.hub_bind_url="http://127.0.0.1:7878" --JupyterHub.proxy_api_port="9001"

Cylc UI master is now available at http://localhost:7000/user/kinow/#/. Then in another terminal (or tmux window, tab, etc).

$ cd ~/Development/python/workspace/cylc-uiserver/
$ source venv/bin/activate
$ (venv) jupyterhub # it works because my cylc-ui for the infinite-tree branch is located at ../cylc-ui, the default static folder!

Cylc UI infinite-tree branch is now available at http://localhost:8000/user/kinow/.

Note 1: One of the two browser sessions must be in incognito.
Note 2: The other session must be in a browser with vue-dev-tools disabled! (the extension may retain objects in memory), or you can just use incognito for both (most times I am using a single branch in incognito, as enabling and disabling extensions takes time).
Note 3: You can look at the footer to confirm the version changed. (I changed it in the infinite-tree branch.)

Cylc Flow workflows

I use the following three workflows for testing. But you are free to use other workflows.

five

This is my favorite workflow, for its simplicity and predicability.

[meta]
    title = "Inter-cycle dependence + a cold-start task"
[cylc]
    UTC mode = True
[scheduling]
    initial cycle point = 20130808T00
    [[graph]]
        R1 = "prep => foo"
        PT12H = "foo[-PT12H] => foo => bar"

families2

This was an example Hilary posted in an issue on GitHub I believe, and it is quite simple too, with the difference it has a good and nested structure with families and tasks. Useful to validate the Tree view is displaying families (in the beginning it was only capable of displaying cycle points, tasks, and jobs).

[scheduling]
  cycling mode = integer
  initial cycle point = 1
  [[dependencies]]
     [[[R/^/P1]]]  # 1, 2, 3, 4, 5, ...
        graph = """foo[-P1] => foo
                 foo => f1 & f2 => bar"""
     [[[R/^/P2]]]  # 1, 3, 5, ...
        graph = "foo => f3 & f4 => bar"

[runtime]
  [[root]]
      script = "sleep $((5 + RANDOM % 10))"
  [[FAM]]
  [[f1, f2, f3, f4]]
     inherit = FAM

complex

This workflow was my karma in 2019. Most problems that forced me to rewrite or drop parts of the initial code of the Tree view were caused by tests with this workflow. This is from the Cylc Flow 7.x examples I believe, but may have some modifications (there are further changes that could be applied, such as queues I think - Hilary might know - but I use the same as in the Cylc 7 branch I think).

(code too long, see gist)

one (mocked)

Cylc UI comes with a mocked workflow, "one", that can be visualized with a command like NODE_ENV="offline" npm run serve.

While this workflow is static (this is a snapshot of a real workflow, recorded as JS for development), it is useful for testing, or for comparing the performance of components without networking, or without having to wait for things to be updated.

It is possible to start two servers for master and infinite-tree branches. e.g.

$ cd cylc-ui && NODE_ENV="offline" npm run serve &
$ cd cylc-ui-master && NODE_ENV="offline" npm run serve -- --port=9090 &

Another way to test without any network, is running NODE_ENV="offline" npm run build to get a production optimized build, with the mocked data. This can then be accessed with the file:/// protocol in the browser, and the absolute location of the dist/index.html file.

[kinow]

Measuring performance

Chromium performance tab

I use the Chromium browser for tests normally. Without the extensions, and with no other tabs open (either in normal or incognito mode).

Then clear any previous results, and press "Start recording and reload" (looks like a reload symbol). That should reload the page and initialize a performance benchmark of a few (4-5) seconds. During that time, it will measure several metrics, including JS Heap, number of listeners, time spent loading and evaluating scripts, as well as other things such as reflow, idle time, etc.

Here's a screenshot of Cylc UI master with the families2 workflow running.

And the same workflow now with Cylc UI infinite-tree.

I have grouped by category. "Scripting" includes things like converting the GraphQL JS object to our tree object, and to the InfiniteTree data as well. While "Rendering" includes things like creating the canvas, painting, putting the elements in position and, in special, re-organizing the elements when there are changes in the DOM and the styles need to be updated.

You may want to compare memory, number of event listeners, drill-in and inspect what other threads were doing during the test.

Another useful test is to click "Record", then scroll down, and expand and collapse a few nodes (preferably a cycle point or family to test if the browser will cascade many events).

What improved

DOM element hierarchy (See ref 1)

Slightly trickier — reduce the size of your DOM tree and the number of elements in each branch. The smaller and shallower your document, the quicker it can be reflowed. It may be possible to remove unnecessary wrapper elements if you’re not supporting older browsers.

Before we had a recursive component, based on Vuetify tree, and on existing tutorials. While it worked, a change in the job details panel could trigger a reflow of its parent, and of its parents' parents, and so it goes. All the way up to the cycle point element, or the tree component element itself.

Now each entry in the tree is a node, with a margin-left proportional to its depth - the higher the depth, the wider the margin is. Updates to a job panel, now should happen within the job panel element. If a reflow occurs, it will be a bug, that can be fixed by changes in the CSS or HTML elements.

Less elements affected when recalculating styles - so less time spent as well

Below is a comparison of "Rendering" and "Painting" for infinite-tree and master branches respectively - note the number of elements affected when recalculating styles.

The same workflow, but for master it had almost twice the amount of elements to be updated. That's because the infinite tree hides other elements, so the browser does not need to go through them updating their style.

So while master takes 38.8ms, infinite-tree does the same work in 0.6ms.

Less time spent "painting" rendering

With less elements, the browser also spends less time painting the screen. For master, it may take near 2.0ms.

While for infinite-tree it never passes 0.3ms.

About the same memory - but winner is still infinitre-tree

When running the one mocked workflow, the memory is about the same, around 22.8 MB in my environment. However, it is important we have less VNodes and Deps. These are classes from Vue.js.

The more VNodes you have, the more work for Vue in maintaining the virtual nodes in sync with the DOM. And Deps are - I believe - observables - or at least they are involved in how Vue reactivity works. The more Deps, the more Vue may spend updating values based on reactivity.

Memory leaks

There are less detached elements, which can be responsible for memory leaks. I took a 4-6 seconds scrolling the page from the top to the bottom on both branches, profiling memory using the "Allocation instrumentation on timeline" option.

After that we have 40 elements (links) detached in infinite-tree.

While we have 1000s detached elements in master (mea culpa maxima!!!).

What got worse

The browser is now taking longer in the Scripting steps. That's because for master once the GraphQL data is retrieved, we create a workflow tree structure, that is used to recursively create the components on the UI.

On the other hand, for infinite-tree, we have an extra step. After we have the infinite-tree, we give this data to the infinite-tree component, which flattens the structure and prepares it for the UI.

So here we have 395.8ms for master, against 525.8ms for infinite-tree. We can't do much here. The larger the data, the longer infinite-tree will take flattening it. But once we have incremental updates & deltas, we should have it happening just once.

References

1 - https://www.sitepoint.com/10-ways-minimize-reflows-improve-performance/

[hjoliver]

Initial subjective tests in my environment with the complex suite (queue-limited to avoid killing my box with jobs) ... shows this performs pretty damned well 🎉

[oliver-sanders]

BTW I haven't forgotten about this PR, just taking a while for me to get my head around it. Should get it signed off on Today or Monday. A couple of comments so far.

[oliver-sanders]

Thought for later work, for default expansion/collapsing we could use:

• A maximum expand depth
  • Which is what we currently do with Cylc7 [and in my Cylc Monitor PR] (i.e. expand the first N levels)*
• A function or list of conditionals.
  • Perhaps slower but very powerful, would allow us to "filter" the display, fits into the infinite tree model well.
  • e.g. collapse parameterisations by default.
  • If nodes know their depth we could implement "maximum expand depth" in this way.

[kinow]

Before that would be a bit harder, as the collapse/extend state was controlled in the component.

The InifiniteTree object keeps state in the data. So we have full control by simply choosing the initial/default state and setting the value in the JavaScript data.

[dwsutherland]

If nodes know their depth we could implement "maximum expand depth" in this way.

Nodes do know their depth 😉

[kinow]

True! But I should have explained better. I think I know one case that doesn't translate directly from the GraphQL results to the Tree view.

I don't remember how we are using depth (i.e. don't remember which nodes in our schema have it). But say we have a workflow with depth=0, then the next step would be cycle points with depth=1 in the tree.

But in our schema, I think the cycle point has a different depth?

We also have special cases, like families. I am not sure if the depth in families is using the correct hierarchy that we need for the tree—which is good I think; our GraphQL schema is independent of component.

Another corner case is the root family, which is hidden by default. So the depth has to be corrected for that. So in the end, we need the depth matching what we want to display in the Tree component, which is too specific for the component I think?

[dwsutherland]

I don't remember how we are using depth (i.e. don't remember which nodes in our schema have it). But say we have a workflow with depth=0, then the next step would be cycle points with depth=1 in the tree.

Depth is defined on all nodes except jobs, and based on Family hierarchy.. Cycle points in the tree view are really the root family in my mind... (BTW - maxDepth is a field under workflow if it helps with working out minimum margins or something)

[oliver-sanders]

Should we use em for margin-left (and also for padding-left in the infinite tree code)?

[kinow]

Oh, will have to look with more calm why I used 12px. 👍

[oliver-sanders]

It's not a blocker or anything just a quick comment, because the line-height is set in em above.

[oliver-sanders]

Thanks, this one was clearly an epic effort 🚀

[kinow]

Build passing on GitHub. Happy to report that it was easier than expected!

Worked locally on "offline" mode, with dummy data. Also worked with no issues serving two workflows (five & families2). I used the branch of the pull request that fixes websockets in cylc-uiserver.

Tested on Ubuntu with Chrome & Firefox. No issues.

Also tested on Windows 10Pro with IE Edge. No issues.

It appears to be a bit slower to render the infinite scroll when scrolling very fast, but could be that it was that way before too. It does look very similar to facebook & instagram when you scroll very fast.

Also fixed two possible memory leaks, where data in a map and event listeners were not properly cleaned up.

Ready for review again 👍

[oliver-sanders]

It does look very similar to facebook & instagram when you scroll very fast.

Would be nice to use a skeleton for scrolling, for future work...

Ready for review again

Code looks good.

Unfortunately the tree is rendering really slowly for me with the complex suite - even when held (~5s when I press Page-Down).

Last time I tested it the complex suite was rendering quickly. Are you seeing different performance.

I also got Opera using >2GiB memory which triggered the debugger to hold the session (which may or may not be related to the infinite tree):

Sorry :(

[kinow]

Thanks for testing it @oliver-sanders ! I may have introduced some bug after rebasing it, or maybe using Lumino with it needs some further work.

It does look very similar to facebook & instagram when you scroll very fast.

Would be nice to use a skeleton for scrolling, for future work...

Good idea! There's a Vuetify component for that too: https://vuetifyjs.com/en/components/skeleton-loaders

[hjoliver]

Hmmm, I agree - sorry Bruno! - something seems to have gone horribly wrong with the performance on this branch (just tested on the complex suite)

[kinow]

Hmmm, I agree - sorry Bruno! - something seems to have gone horribly wrong with the performance on this branch (just tested on the complex suite)

@hjoliver I wonder if accessing the Tree view directly is also not working? From the Workflows Table, there should be a link to view the tree view I think.

[hjoliver]

From the Workflows Table, there should be a link to view the tree view I think.

The table rows now link to the lumino tabs view, with the default tree view tab. (I did that last week).

Is there a direct URL I can use to the tree view component (I know there is for the graph view.)

[kinow]

Sorry, my environment is slightly broken while I finish updating some deps. Just had a look at src/router/paths.js, and even though the view src/views/Tree.vue exists, it is not bound to any route - meaning we can remove it later, unless it's added back as per #376

Maybe if you copy the Graph entry in paths.js and binds it to some route like /#/tree/:workflowName it could work?

[kinow]

Rebased

[oliver-sanders]

On em/rem instead of px, that's not possible due to how the vue-virtual-scroller creates the virtual elements in the visible area

Fair enough, makes sense.

It appears to be working OK when I change the browser zoom and viewport area.

Ah so it does, and the style seems to hold together perfectly.

So long as accessibility is fine I'm happy, looks good to me:

Sorry, but I've found another bug :(

If I expand items in the tree, then scroll up or down, sometimes items get cut off of the front or end of the tree. I can replicate this with the "one" suite in the mocked data:

Expand/collapse or switching tab seems to be enough to un-stick the tree.

I guess this sort of thing is always a risk with a virtual tree so it might be very difficult to fix this entirely. If we could reduce the likelyhood, and make it clearer to the user that there is content that they can't see (e.g. with a skeleton for the missing items) that would be enough.

[hjoliver]

The expand/collapse triangles and in-line job icons aren't independent between lumino tabs (is this a known issue?). Here I expanded cfoo in the tab on the right:

[oliver-sanders]

The expand/collapse triangles and in-line job icons aren't independent between lumino tabs

Ah, dang good spot, I guess the expand/collapse state is stored in the shared tree rather than locally in the view instance. Might be related to the change from JS to CSS logic?

[hjoliver]

If I expand items in the tree, then scroll up or down, sometimes items get cut off of the front or end of the tree.

I've played around quite extensively with the complex suite and a smaller suite. So far I've seen this all the time in a second tab (if I have a second tab open) but never in the first (or only) tab.

If I have a second tab open, the view never updates on scrolling. If I scroll the most of the rendered items off-screen, then expand or collapse one of the remaining nodes, at that point the view in the tab updates (but scrolling more results in the same problem again).

[hjoliver]

(Scrolling performance seems really good 👍 )

[kinow]

I tested with one and multiple workflows, and with tree, and tree+graph. But never tested tree+tree 😁 good catch @oliver-sanders ! And @hjoliver is right about the expand/collapse not being independent.

There is a single tree, backing both tree components. When you expand one, it emits an event telling the infinite tree that we want to expand a node.

The node's state is changed, reflecting the new state: { open : true } modification. And it emits some other events and performs a few more actions, which I am currently digging into the code to understand more of how it works.

Both issues reported here are due to this state-sharing issue. When you expand on one tree component, the other component receives the new state, and changes the expand/collapse icon. However, the elements are not rendered.

So if you have a limited viewport, with scroll active, and expand one one component, the other will change its state without rendering the elements. When you scroll on the other window, the component will think it needs to show the children elements of the expanded node, but as these elements were not rendered, you see only a blank area.

I believe if we are able to tell both components to change the state, and render the newly expanded nodes, it should work correctly. Then we can discuss having separate states per widget... or maybe having a toggle button to link the widgets.

[kinow]

Actually there's one more issue. Related to the double scroll bars. If you scroll up or down, and you have the view area scroll, and also the component scroll, both active. Then looks like the outer scroll bar needs to emit an event to tell the tree we need to refresh it.

Without that happening, you can end up with empty areas in the tree, due to elements not being rendered.

We already have some code taking care of events when you add other widgets, to tell our tree we want it to update. It might be just a matter of finding what else we need to listen to and tell the tree to update.

p.s. the tree component uses vue-virtual-scroller, which uses the intersection observer API, which has a few issues with browsers. It could be one of these issues too 🤓

[kinow]

Scrollbar issue

Here's how I'm troubleshooting this first issue. First I resized the browser view port area so that I had both scroll bars visible, and successfully reproduced the issue (chrome/chromium/firefox on Linux - i.e. not a browser issue).

Then I used VueDevTools to find the RecycleScroller component, and stored a reference as $vm0. Then started testing the methods of that component instance, to see what would fix the issue.

• $vm0.handleResize() fixes it
• $vm0.handleScroll() fixes it
• $vm0.updateVisibleItems() fixes it

These are the three methods that I found in the object that fixed it.

The RecycleScroller.vue component listens for two events, the scroll and resize.

The scroll will trigger handleScroll (mentioned above), and resize handleResize (also mentioned above). If we reproduce this issue, and then resize the window, it also renders the missing items.

WIP

Workaround

If this issue is too complicated to be fixed, one possible workaround is to support only users with a viewport large enough to display the widgets without scroll bar (i.e. support at least the minimum height of the widgets, plus toolbar and padding/margin)

[kinow]

shared state (expand/collapse) issue

The CylcTree object, that is created for and with the deltas, stores in each node created an attribute with the state for the collapsed/expanded featured.

tree-entry: {
  id: 'unique-id',
  size: 32,
  node: notReplicatedNodeFromGraphQL,
  state: {
    open: !['job', 'job-details'].includes(node.type) // this is kind of what we are doing in the code, expanding everything but jobs and leaf nodes
  }
}

If we have two or more Lumino widgets added to the Workflow view, each widget will have a TreeWrapper. And each wrapper has an InfiniteTree, but the InfiniteTrees are created using the same CylcTree, without duplicating the data.

When you expand or collapse a node, we call the InfiniteTree.openNode(node) or tree.closeNode(node). That changes the state of the node and tells the tree it needs to recompute what is displayed or not.

If you have two infinite trees, A and B, and you expand a node in A, the node will be expanded in both A and B.

But only A.openNode(node) is called. So the tree B doesn't understand it needs to update. I believe that's causing the issue at the moment.

I think we have a few possible solutions so far:

1. call openNode for each tree in the Workflow.vue component, i.e. iterate the widgets, if type of widget is tree, call $component.tree.openNode(node).
2. move the state out of the tree, and into the component, so each widget controls the state separately
3. check if the infinite tree and/or vue-recycle-scroller support the state in some other way (a function perhaps, that is applied only to the local instance of the flattree)
4. explore hacking the flattree object, injecting the state or intercepting the change of state if coming from a different tree?

1. call openNode for each tree in the Workflow.vue component, i.e. iterate the widgets, if type of widget is tree, call $component.tree.openNode(node).

2. move the state out of the tree, and into the component, so each widget controls the state separately

@hjoliver suggested this one. I had a look and my first test was to remove the state.open from the nodes. The issue persisted 🤔

Then I realized that even if we removed node.state, and tried to move that somewhere else, the InfiniteTree operates with the node.state. Meaning that later when we expanded it, it would still call node.state.open = true.

3. check if the infinite tree and/or vue-recycle-scroller support the state in some other way (a function perhaps, that is applied only to the local instance of the flattree)

It does support an autoOpen, but that's global. You specify a boolean that is passed to the flatten function as openAllNodes. That's only for expanding all the nodes or not.

flattree supports an array of openNodes, but the InfiniteTree still works with node.state.open.

4. explore hacking the flattree object, injecting the state or intercepting the change of state if coming from a different tree?

WIP

Workaround

So far the only workaround I can think of, is limiting the number of tree widgets to one, which doesn't sound like a good idea.

[kinow]

Closed as per Riot message 🎉

tree view decision

With agreement from the UK end, via oliver sanders , we are going to shelve the infinite tree for now.

We may come back to virtual scrolling in future if we need it for large workflows, but perhaps not with a collapsing tree view. For the moment, delta-updates seem to have nicely revitalized the simpler plain tree view, and low-n windows will make it even better again soon. My guess is virtual scrolling works well for an ever-extending flat list of items, but our data (particularly in collapsing tree view) is not so simple.