Deleting routing configuration#

The appliance generates FRR's configuration from its own and applies it to the running routing daemons by merging it in. Merging adds and changes; it does not remove. This page says what the appliance removes from FRR for you when you delete configuration, what it leaves, and how to see and clear what is left.

What is removed for you#

You delete…	Removed from FRR
a VRF	its BGP instance (and with it everything the VRF announced) and its `vrf` block — unless the VRF originated a default route, see below
an address pool, a CPE circuit, an IGP	their own statements, as before

That is all. A deleted CGN IP group, static route or BGP peer stays in FRR's running configuration until you remove it, as described below. The commit tells you (FRR_REMOVAL_SKIPPED) and show protocols frr-residue lists it.

What the appliance reports as removed (FRR_STATEMENTS_REMOVED) it has confirmed from FRR's running configuration afterwards. A statement FRR was asked to remove and still holds is reported as not removed (FRR_REMOVAL_FAILED), with what FRR still holds in it.

A commit that changes nothing in routing sends FRR nothing.

A VRF with subscribers on it cannot be deleted#

The commit is refused, naming the VRF and how many sessions it has. Move or disconnect them first. commit check gives the same refusal, so you can find out without committing.

A VRF may not be named default, all or lo.

A VRF is also left in FRR — and the log says so — when it is the management VRF, when a CGN IP group uses it as its outside VRF, or when anything else in the configuration still names it.

One case gets past the refusal: a commit confirmed that is not confirmed rolls back unconditionally. If the change being rolled back added a VRF and subscribers are already on it, the rollback still happens, but the VRF is left in FRR so that their routes are not withdrawn.

Seeing what is left#

show protocols frr-residue

lists what FRR's running configuration holds, of the kinds the appliance generates, that your configuration no longer generates, and what the last change removed, did not attempt, or asked for and FRR did not remove. It only reports. Use it after deleting routing configuration and after a rollback. An entry may be a leftover, or may be something added to FRR by hand.

After a VRF has been deleted the list is never empty again: the VRF's route-maps (BNG-REDIST-STATIC4-V-<vrf>, BNG-REDIST-STATIC6-V-<vrf>) and prefix-lists stay, by design. They are harmless — see Route-maps and prefix-lists. The view is clean when those are all it lists.

Static routes and the appliance's own routes#

A static route for a prefix the appliance routes itself replaces the appliance's route

The appliance puts its own routes into FRR: one for every subscriber's address and prefixes, and one for each VRF's L2TP endpoint address. FRR holds them as static routes, exactly like the ones you configure — and it holds one static route per table and prefix.

So a configured static route for the same prefix does not sit beside the appliance's route. It replaces it, the moment it is committed: traffic for that subscriber follows your static route instead. And when the configured route is later removed from FRR, there is no route left for the prefix. The appliance does not notice and does not put its own back; the subscriber stays unreachable from the network until the route is restored as described below.

A commit is refused if it adds a static route that would do this to an address the configuration shows to be the appliance's:

a host route (a /32; for IPv6 a /64 or longer in a WAN pool, the delegation length or longer in a prefix-delegation pool) inside an address pool of the default table — unless the pool's exclude list covers it;
the announced subnet of a default-table address pool, or anything equal to or inside a CGN IP group's public subnet in the default table.

A static route that merely covers a pool — a default route, an aggregate — is ordinary routing and is accepted. So is everything already in your configuration: only a commit that newly creates the collision is refused. Resizing a pool or a CGN public subnet that already covers such a static route is not a new collision; resizing one so that it comes to cover a static route it did not cover before is. A static route for one of the appliance's own addresses (protocols ipoe server-ip, the global L2TP endpoint) is accepted with a WARNING line.

The refusal cannot see an address that the configuration does not name — one RADIUS assigns outside every pool, a framed route — nor a route typed into vtysh by hand. For those, check:

Telling the two apart#

show ip route P
show running-config | include ip route P

(in vtysh; show ipv6 route and ipv6 route for IPv6, and show ip route vrf X P for a VRF).

	`show ip route P` says	a line in `show running-config`
the appliance's own route	`directly connected, <interface>`	none
a configured static route	`via <address>` (or the device you configured)	`ip route P …`

If show running-config has a line for exactly P and P is a subscriber's address or prefix (show subscribers) or a VRF's L2TP endpoint address, the configured route has replaced the appliance's.

Restoring the appliance's route#

Delete the static route from the appliance's configuration and commit, if it is there.
Remove it from FRR (A deleted static route). Now there is no route for P.
Make the appliance put its own route back. There is no command that does only that. What re-creates the route:
- For a subscriber: disconnect the session — request subscribers disconnect id <id> (or username <name>). The route is created when the session comes back up. A DHCP renewal or a PPP keepalive does not re-create it.
- For a VRF's L2TP endpoint address, or for many subscribers at once: restart the control plane (systemctl restart bng-controlplane). At start-up it puts back every session's routes and every L2TP endpoint route. This interrupts L2TP tunnels, which then re-establish; plan it.
The appliance also puts all its routes back by itself whenever its connection to FRR is lost and returns — but restarting FRR to cause that drops every BGP session, and is not a way to restore one route.
show ip route P now says directly connected.

A subscriber that is given a prefix you route statically#

The other way round: the configuration routes a prefix — a static route, an announced address-pool subnet, a CG-NAT public subnet, a VRF's originated default, a VRF's L2TP endpoint address — and a subscriber is then given exactly that prefix, typically as a framed route from RADIUS for a site that has since moved. This is a misconfiguration: correct the subscriber's record. (A subscriber address inside a prefix you route is not this: a /32 within a configured /24 is ordinary.)

The appliance does not install such a subscriber route, and never deletes a prefix the configuration routes: the subscriber comes up without the route and your route stands. show subscribers sessions <id> has a Not installed: line under Routes:, and the log has ROUTE_INJECT_SKIPPED.

If the subscriber's route was installed before the configuration routed the prefix — the session was already up when you committed the static route, or came up on earlier software — FRR may be holding the subscriber's next hop where yours should be. The appliance does not remove or rewrite a configured route. When that subscriber's route would have been withdrawn (the session ends, or loses the route) it is left alone and the alarm routing.configured_route_displaced is raised, naming the prefix, the VRF and the session. Then, in vtysh:

show ip route 192.0.2.0/24
show ip route vrf residential 198.18.0.0/24

If the route shown does not go through your configured next hop, re-enter the static route statement in vtysh (remove it and enter it again, as in A deleted static route), or reload FRR. The alarm clears when the route is deleted from the configuration, or with clear alarms routing.configured_route_displaced <prefix>@<vrf> (for example 192.0.2.0/24@default) once you have checked.

Removing what is left#

These are typed in vtysh on the appliance. Take the values from the lines show protocols frr-residue prints.

A deleted VRF that originated a default route#

A VRF with originate-default-route has a blackhole default route inside its vrf block. The appliance does not remove static routes, and FRR keeps a vrf block for as long as it has anything in it — it accepts no vrf X without an error and leaves the block where it is. So after such a VRF is deleted, its BGP instance is gone (nothing is announced for it any more) and the commit reports the vrf X block as not removed (FRR_REMOVAL_FAILED: kept by FRR … it still holds ip route 0.0.0.0/0 blackhole). The view lists vrf X: ip route 0.0.0.0/0 blackhole and vrf X.

To clear it, for a VRF X:

configure terminal
vrf X
no ip route 0.0.0.0/0 blackhole
exit-vrf
no vrf X
end

This no ip route is safe: the VRF has been deleted and has no sessions, so the appliance has no routes in it.

A deleted or re-addressed CGN IP group#

CGN groups are read when the control plane starts, so a change takes effect at the next restart, and until then the running box is still translating subscribers onto the old public subnet. The appliance therefore never stops announcing the old subnet by itself.

Drain the group. A commit that deletes a group, or makes its public subnets, outside VRF or chunk size differ from what the running box is using, while it has live bindings is refused, naming the group and the number of bindings. (A commit that puts a group back to what the running box is using is always accepted.) commit check gives the same refusal.
Commit the change and restart the control plane.

A commit that changes only CGN does not touch FRR. FRR is given the group's change by the next commit that also changes something in routing, or by the restart — whichever comes first. From that moment a re-addressed group's new subnet is announced, which can be before the restart that makes the box translate onto it; do both in the same maintenance window.
show protocols frr-residue lists the old subnet's network statement and blackhole route — once FRR has been given the change, as in step 2. Before that it has nothing to report.
Remove them. For a subnet P in the default table, with BGP AS N:
```
configure terminal
router bgp N
address-family ipv4 unicast
no network P
exit-address-family
exit
no ip route P blackhole
end
```
For a group whose outside VRF was X, use router bgp N vrf X and no ip route P blackhole vrf X.

The old subnet is withdrawn from your peers as soon as no network is entered.

A deleted static route#

First read Static routes and the appliance's own routes: if the appliance also routes P, removing the configured route leaves no route for P until you restore the appliance's.

For a route to P via NH:

configure terminal
no ip route P NH
end

Use no ipv6 route for IPv6, and add vrf X for a route in a VRF.

A deleted BGP peer#

The commit reports it (FRR_REMOVAL_SKIPPED: BGP peers are not removed automatically). For peer address A, with BGP AS N:

configure terminal
router bgp N
no neighbor A
exit
end

The session to A closes at once. Other sessions are not affected.

Route-maps and prefix-lists#

Leave them. The appliance's own (BNG-…) are harmless when unused, and removing one that something still refers to stops routes being announced.

Before an upgrade#

frr reload-method should be vtysh (the default). With restart, any change to the generated FRR configuration restarts FRR and drops every BGP session.
A change to the set of FRR daemons — switching the IGP between IS-IS and OSPF — restarts FRR whatever the reload method. Do not combine it with an upgrade you want to be hitless.
An upgrade is rolled back if the box is not healthy within 210 seconds, and a box whose subscriber routes have not been put back into FRR is not healthy. If FRR is down when the new control plane starts, the management plane comes up and the frr.disconnected alarm is raised; if FRR is still not back at the deadline, the upgrade rolls back.