Use "lazy" parsing of netlink messages to improve performance

[crosser]

After introducing BGP-EVPN in our environment, we now have hundreds of network interfaces and tens of thousands of routes on our systems. This made pyroute's "dump" operations unusably slow. Analysis revealed that a lot of CPU time is spent on parsing netlink messages.

Situation was so bad that we decided to reimplement dump operations from scratch. We used "on demand" parsing, skipping all NLAs except those that we need. This gave us a couple of orders of magnitude speedup. But as a consequence of "on demand" approach, the API to our implementation is quite different from that of pyroute (and much lower level), so it it not possible to contribute our optimisations directly to pyroute.

However, I have an idea that may allow to use similar optimisation with existing, or only slightly modified, pyroute API. The idea is to not parse the message that was received, keeping it as bytes blob. And use a custom implementation of getitem() to look up the NLA only when it is requested. (And possibly cache the result in case it will be requested again.) This would also allow to revive the possibility to fetch NLA attributes as python object attributes, by implementing getattr() in the same way, without loss of performance.

What do you think about such idea, @svinota ?

[svinota]

I believe it's perfectly what we need.

There are already two approaches in the library that may help with that:

• custom parser API,

  pyroute2/pyroute2/iproute/linux.py

  Line 817 in 59b4a72

  parser=default_routes,

• the parser by itself tries to use the lazy approach:

  pyroute2/pyroute2/netlink/__init__.py

  Line 1803 in 59b4a72

  def __getitem__(self, key):

But the current implementation is quite slow anyways and I'm ready to refactor it.

[crosser]

I've pushed through our internal implementation to be open to the public from the company's org:

https://github.com/ionos-cloud/netlinklib

So if you are interested @svinota you might take a look now. It is actually very small! (The API is very different though.)

(And I think we could potentially squeeze another half an order of magnitude speedup if we use a slotted object instead of the dict for the accumulator here.)