Allow references in Folium

[hansthen]

There are a few issues open that it is impossible to reuse elements such as Icons for layers. Apart from the CustomIcons, other things that are currently impossible are circular references. E.g. I would like to pass a reference to a LayerControl from a Layer and vice versa, so that they can update the other's state during events. This is currently not possible.

The underlying cause of this is the Branca object model. Branca only allows containment relations. This is fine for modelling HTML or any other document object model. However, it does not much the Leaflet object model, which uses references instead of containment.

The following is a two part proposal on how to make references possible. I will first describe the required changes to the object model. Second I will describe how we can make this proposal work without breaking existing client code. Out of scope will be developers who may have made custom plugins or inherited from our object model outside of the Folium project.

[hansthen]

First part, establish new object model

1. Create a new Branca Element to represent a Javascript code block.
2. Make Folium Leaflet objects no longer recursive.
3. When creating Folium objects add them to the Javascript code block
4. Method invocations on Folium objects that modify the structure (like FeatureGroup.addTo and Map.addTo) should generate the equivalent javascript method call. This is similar to how we implement the on method. We can probably generalize method invocations further. Instead of create a new Branca element for each new method name, we can create a generalized MethodInvocation Branca element.
5. MethodInvocations are added to the Javascript code block.
6. HTML generation (render) is invoked on the Javascript code block.

[hansthen]

Second part, make it backwards compatible

1. Define a (thread local) variable referring to the Javascipt code block. This variable should be automatically initialized if it is not there.
2. Inside the Folium Class constructor, the object is added to the Javascript code block.
3. Inside the MethodInvocation constructor, the object is added to the Javascript code block.
4. Branca methods that currently exist on any folium objects are forwarded to the Javascript code block.

[Conengmo]

First I'll try to see if I understood correctly.

Seems like you define two goals:

• allow reusing elements (add an icon once, use it in multiple markers)
• allow circular referencing (references are not limited to the order in which objects are added to the map)

Backwards compatibility is only maintained for users of the documented Folium features, not for external plugins or users extending Folium in custom ways.

To understand the steps you are describing, it would really help me to see some code. Maybe you can make a very barebones example of this setup? Like, separately from the existing Folium code, just a dummy example to show the proposed architecture? It's interesting and I very much would like to understand it fully.

[hansthen]

Yes. A very concrete use case where I ran into a brick wall is the following. I have a Folium app that combines a LayerControl with a few Realtime layers. When a Realtime layer fails to load, I want to change the color of the name of that Layer inside the LayerControl. To do so, I want to add a reference to the LayerControl (via javascript options) when initializing the Realtime layer.

[Conengmo]

Question about this use case: if elements are rendered in order, how would a circular reference work?

Either layer control doesn't know about realtime when it is initialized:

layer_control = LayerControl().add_to(m)  # this initializes with all existing layers
RealTime(color_option=layer_control)

or realtime doesn't konw about layer control yet:

RealTime(color_option=layer_control)  # error
layer_control = LayerControl().add_to(m)  # this initializes with all existing layers

[Conengmo]

I'm also inspired to try out adding multiple parents to the Branca element and still allow rendering. I.e. changing Element._parent to Element._parents and make sure each element is only rendered once. That would probably only address the first goal of reusing elements, but not allow circular referencing. I'll see if I can make a demo.

[Conengmo]

Awesome, I get this idea. Makes a lot of sense. We should try it out to see where the issues lie!

[hansthen]

One of the issues I foresee is that Layer now has a show variable, specifically as a workaround for this issue. (So that an item can be added to the map in Folium without adding it to the map in Leaflet).

I don't know what the best approach will be for those.

I have some time available during the holidays. I can write a small PoC and see where I get stuck.

[Conengmo]

One more disjointed thought: right now it's a bit confusing that from a user perspective Map is the root object. But underwater it adds a Figure as parent of Map. Maybe we can do without that, and Map can be the actual root that collects elements. Doesn't mean that all code has to live in the Map class, it could delegate that through composition.

[hansthen]

I was thinking about that, but what will we do with DualMap?

[Conengmo]

good point, would have to look into that!

[Conengmo]

Another thought: seems the Github Issues are just about adding icon and divicon to markers. Maybe the simplest thing is to just do something custom for those particular elements. That's not unheard of, we have other classes that do some special stuff around rendering.

[hansthen]

Yes, that is surely the most pragmatic approach. However, it does not solve the issue of circular references. I also ran in some other limitations due to the current structure: e.g. creating popups for geojson layers.

[Conengmo]

Created a prototype for this in #2053. B.t.w. I don't mean this as saying we shouldn't look at other approaches.

Note that it's not meant to solve circular references or popups or tooltips. This is just for icons, since the thought was that all three issues we got we're explicitly about icons, so maybe that's the only thing we have to solve for.

[hansthen]

@Conengmo

I made some progress for a PoC. There is one problem that I immediately ran into. I cannot easily use a thread local variable as the new root element. This will work in most typical use cases, but not for running the tests. However, we can probably use the Figure element that is used as the parent of Map to work as global root element.

There are however a few things I do not really understand in Branca that hinder my progress. How does the render method work in Branca? On Element the render method returns its generated template, which recursively calls render for each of its children. So far so good. When I get to Figure I see this:

      def render(self, **kwargs):
          """Renders the HTML representation of the element."""
          for name, child in self._children.items():
              child.render(**kwargs)
          return self._template.render(this=self, kwargs=kwargs)

Where does the output of the rendered children go? I also see that some render methods do not render anything at all, but just do structural manipulation (adding children to their parents and stuff). Am I correct that there are two versions of render

1. that invokes a jinja template to generate html code.
2. one that prepares the parent to be rendered?

[Conengmo]

That's a great question. The way rendering works is very convoluted and still confuses me after all these years. At times I really dive into it and then feel like I understand, only for that knowledge to then leave my brain again.

I'll try if I can write a short step guide...

A thought I've also had is to simplify this part and move it back to Folium. That way this aspect would be easier to understand, and we can change it more easily without having to deal with separate releases.

[hansthen]

@Conengmo I just started a new discussion about this oddity, since it is also the thing that makes mypy complain. I think we should do something about this, since I have the exact same feeling that the truth is just a bit beyond my grasp. Can you have a look at it to see if my writeup makes sense? I also have a modest proposal to make it simpler.