The migration for stateless NAT is complicated as it cannot be done
during operations. Hence the game plan is to reboot the passive device
on 17.6, let it be synced with NAT stateless, then failover and do the
dance again.

For this procedure it comes in handy for us to completely ignore any
stateless NAT config and just make it mandatory when 17.6 is running.

Static NAT creates an implicit arp alias if the NAT IP resides in the ip
nat outside interface's subnet. We also create an explicit arp alias for
every static NAT statement, regardless of subnet affinity. Cisco stated
that on cleanup there might be a condition under which the deletion of
the explicity arp alias and the deletion of the implicit arp alias
(triggered by the deletion of the static arp statement) challange each
other and lead to the arp alias still residing in hardware. This in turn
means the device still replies to arp for that IP.

As a workaround we add each static NAT statement without the implicit
arp alias. We handle the explicit arp alias already, hence now there is
only a single arp alias in the house and the deletion should work as
expected.

match-in-vrf is a YANG_TYPE.EMPTY, if the tag is present it is true, if
it is absent, it is false. Found this after writing a parsing test for
this class. Also the yang-key argument is redundant as NyBase will
replace all underscores in the key by dashes.

…outer_id, network_id

Python 3.8: github workflow and tox

[yoga] neutron callback system passing dbevent payload instead of context, router_id, network_id

We support extraroute-atomic, but don't advertise it properly, which
means it does not appear in the available OpenStack network extensions.
This advertising is needed for clients which want to decide if they can
use that API or not.

Reviewing the whole way we manage supported_extension_aliases shows that
we should probably rework this code and orient ourselves a bit at what
happened in the reference implementation in neutron, but today is not
that day.

Neutron dropped registry.notify() and migrated to using payloads and
publish(), so we're doing the same. This change is very much oriented on
what can be found in neutron in neutron/agent/l3/agent.py.

admin-shutting an agent leads to the agent being completely vacated.
However we have cases in which we want an agent to stay online, but
prevent any new workloads to be scheduled on it. This can be a capacity
consideration or because it is meant to be decomissioned.

When a router/VRF is part of a BGPVPN, the BGPVPN portion is configured
first. If a BGPVPN happens to have a peer on the local agent (hardware
router) this peer VRF's routes are leaked into the to be configured VRF.
Due to the order of operation, this leaking happens before any
VRF-attached l3 interfaces are configured.

Now say, a customer also configured an extra route. That extra route
also happens to be contained in a leaked route and that extra routes's
next hop also happens to be another BD-VIF in a BGPVPN-peered-VRF that
is local on that hardware router.

Even though the to be configured VRF has no l3 interface yet, it already
has a BGP RIB path pointing to that next hop address of the extra route.
It also believes it's a local interface, which it is, but it also is
not, because that BD-VIF lives in another VRF.

The hardware router not caring about such technicalities, will refuse to
configure the extra route with `%Invalid next hop address (it's this
router)` and all subsequent configuration fails.

The fact that the next hop lives on the same route can however be
obscured if the next hop is directly connected. It becomes directly
connected when the BD-VIFs are configure prior. So let's do that.

OpenStack also prevents you from creating extra routes where the next
hop is not local to the virtual router, so I think all cases should be
covered. Even on interface removal, OpenStack checks if a route still
utilizes that interface.

See https://github.com/sapcc/neutron/blob/5fb5653ffe700a8f9fb0949d16ecbf0978b61fb6/neutron/db/extraroute_db.py#L74-L86
https://github.com/sapcc/neutron/blob/5fb5653ffe700a8f9fb0949d16ecbf0978b61fb6/neutron/db/extraroute_db.py#L154-L164

We can now specify an ARP timeout that gets configured on all router
interfaces, configured separately for external and internal interfaces.
This can be used facilitate faster ARP cleanup of old IP-mac
combinations. The main target of this feature is to get faster FIP reuse
working, where routers are in the same external network. Therefore the
current defaults are 30 minutes (1800s) for external interfaces and
unset for internal interfaces. I'm not currently aware of any problems
inside internal networks, but when the time comes it might be nice to
already have this option waiting.

The asr1ks in our infrastructure have a relatively high ARP timeout
(default is set to 4 hours). If in this time a fip moves from one router
to another, it changes mac address. This means that routers where this
FIP is not configured on might have a stale entry and will send out IP
packets to the wrong mac in the Ethernet frame. Normally this would be
solved by sending out a gratuitous ARP packet from the router, but the
router does not do this for NAT statements (a long open feature
request). Sending out "fake" ARP requests have either resulted in the
ASR not learning the new mac (IP packet from behind the ASR via network
agent as broadcast) or caused the ACI to either discard the ARP packet
(as it "knows better" what we want) or to relearn the endpoint onto
another device, where we would not want it. Therefore we now try to find
these stale entries on other routers and clean them.

The agent now as an ARP cleaning syncloop. By default every two minutes
(though it is configurable). The agent fetches all fip --> router mac
associations. We expect each fip - if we see it - to be on the mac
address of the gateway interface of our router. Then, for each device on
the agent, we fetch the ARP cache and look for IPs that we know about
and are on the wrong mac. Everything wrong will then be cleared.

Fetching all ARP entries from a device takes about 1.2-4.0s in my
tests, depending on the ARP table size on the device. Selecting only a
subset of fields makes the query go something like 20-40% faster.

If the device doesn't give us an ARP cache (e.g. if netconf-yang is
overloaded or disabled) we now don't throw an Attribute error anymore,
but instead log a warning that we couldn't fetch the ARP cache. I opted
for a warning instead of an error as I think YANG problems either are or
should be handled elsewhere, this new could should not be the place to
alert or throw out Sentry errors, as they are (or should be) duplicates.

The method is called labels(), not label(). Using the latter results in
a ValueError "Incorrect label names" by the prometheus_client library.

The external gateway of a router normally has an address scope set and
if that address scope cannot be found in our driver config we log an
error message. This is the behavior we want for everything that has an
address scope. For setups where the external gateway it not an external
network and therefore also has no address scope is set we actually don't
want this to be logged, as everything should be okay.

We'll silence the error for everything that has no address scope. This
also means that we won't log the error for external networks that have
been configured without any subnet with an address scope, but as this is
more of an admin error and I don't want to fetch the external gateway
only for deciding if we want to log this error I'll leave it this way
for now.

The delete action will error out if the attribute that we wanted to
delete does not exist. Remove will just silently ignore it, which is
exactly what we want.

In some cases we want to delete a bridge member, maybe even as a "side
operation", i.e. triggered by another operation. If the interface has
already been removed (by the cleaning loop, another thread or whatever)
we don't want to fail, but silently ignore this. Therefore "remove"
seems to be the operation which is more fitting.

One could argue that this is the operation we want to use in most cases
in this driver... well, we're getting there.

When instantiating the Router model with router data from Neutron where
the gateway port is unbound we might not have a primary subnet. We
already handle the case that the subnet has no CIDR, but if the subnet
is None, we don't have the .get() method available, resulting in an
AttributeError in _build_pbr_acl(). To fix this we simply check if the
given subnet is None beforehand.

When a port has no primary subnet we don't want to lookup the cidr of
None, resulting in a TypeError. See last commit
d89db00 for details.

The argument for the failing entity is called entity, not entity_name.

We used to check for special features and versions by looking at the
revision date of the YANG models, specifically the Cisco-IOS-XE-native
model. Newer versions (e.g. 17.3) apparently don't advertise the
revision dates of these models without extra queries, so this won't
work. As most feature switches we require can be set by looking at the
major/minor version of the current firmware. Therefore we now query for
each connection what the version is via "native/version". As we might
require more fine grained feature switches in the future I'll leave in
the machinery to do capability checks based on YANG model versions.

The "missing address scope for router" message was missing a space and a
closing bracket at the end.

Firmware 17.13 starts to support the garp-interface option for static
NAT configuration. This means that on configuration a gratuitous ARP
will be send out on configuration of the statement. To use this, we need
to be at least on 17.13. The target interface will always be the gateway
interface. Without a gateway interface, the flag will be removed.