--- /dev/null
+BGPD
+=========================
+
+.. toctree::
+ :maxdepth: 2
+
+ next-hop-tracking
+
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
- (master_doc, 'FRR.tex', u'FRR Developer\'s Documentation',
+ (master_doc, 'FRR.tex', u'FRR Developer\'s Manual',
u'FRR', 'manual'),
]
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
- (master_doc, 'frr', u'FRR Developer\'s Documentation',
+ (master_doc, 'frr', u'FRR Developer\'s Manual',
[author], 1)
]
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
- (master_doc, 'FRR', u'FRR Developer\'s Documentation',
+ (master_doc, 'FRR', u'FRR Developer\'s Manual',
author, 'FRR', 'One line description of project.',
'Miscellaneous'),
]
workflow
library
+ bgpd
building
+++ /dev/null
-## Topology
-
-The goal of this test is to verify that the all the basic functionality
-of ldpd is working as expected, be it running on Linux or OpenBSD. In
-addition to that, more advanced features are also tested, like LDP
-sessions over IPv6, MD5 authentication and pseudowire signaling.
-
-In the topology below there are 3 PE routers, 3 CE routers and one P
-router (not attached to any consumer site).
-
-All routers have IPv4 addresses and OSPF is used as the IGP. The
-three routers from the bottom of the picture, P, PE2 and PE3, are also
-configured for IPv6 (dual-stack) and static IPv6 routes are used to
-provide connectivity among them.
-
-The three CEs share the same VPLS membership. LDP is used to set up the
-LSPs among the PEs and to signal the pseudowires. MD5 authentication is
-used to protect all LDP sessions.
-
-```
- CE1 172.16.1.1/24
- +
- |
- +---+---+
- | PE1 |
- | IOS XE|
- | |
- +---+---+
- |
- | 10.0.1.0/24
- |
- +---+---+
- | P |
- +------+ IOS XR+------+
- | | | |
- | +-------+ |
- 10.0.2.0/24 | | 10.0.3.0/24
-2001:db8:2::/64 | | 2001:db8:3::/64
- | |
- +---+---+ +---+---+
- | PE2 | | PE3 |
- |OpenBSD+-------------+ Linux |
- | | | |
- +---+---+ 10.0.4.0/24 +---+---+
- | 2001:db8:4::/64 |
- + +
- 172.16.1.2/24 CE2 CE3 172.16.1.3/24
-```
-
-## Configuration
-
-#### Linux
-1 - Enable IPv4/v6 forwarding:
-```
-# sysctl -w net.ipv4.ip_forward=1
-# sysctl -w net.ipv6.conf.all.forwarding=1
-```
-
-2 - Enable MPLS forwarding:
-```
-# modprobe mpls-router
-# modprobe mpls-iptunnel
-# echo 100000 > /proc/sys/net/mpls/platform_labels
-# echo 1 > /proc/sys/net/mpls/conf/eth1/input
-# echo 1 > /proc/sys/net/mpls/conf/eth2/input
-```
-
-3 - Set up the interfaces:
-```
-# ip link add name lo1 type dummy
-# ip link set dev lo1 up
-# ip addr add 4.4.4.4/32 dev lo1
-# ip -6 addr add 4:4:4::4/128 dev lo1
-# ip link set dev eth1 up
-# ip addr add 10.0.4.4/24 dev eth1
-# ip -6 addr add 2001:db8:4::4/64 dev eth1
-# ip link set dev eth2 up
-# ip addr add 10.0.3.4/24 dev eth2
-# ip -6 addr add 2001:db8:3::4/64 dev eth2
-```
-
-4 - Set up the bridge and pseudowire interfaces:
-```
-# ip link add type bridge
-# ip link set dev bridge0 up
-# ip link set dev eth0 up
-# ip link set dev eth0 master bridge0
-# ip link add name mpw0 type dummy
-# ip link set dev mpw0 up
-# ip link set dev mpw0 master bridge0
-# ip link add name mpw1 type dummy
-# ip link set dev mpw1 up
-# ip link set dev mpw1 master bridge0
-```
-
-> NOTE: MPLS support in the Linux kernel is very recent and it still
-doesn't support pseudowire interfaces. We are using here dummy interfaces
-just to show how the VPLS configuration should look like in the future.
-
-5 - Add static IPv6 routes for the remote loopbacks:
-```
-# ip -6 route add 2:2:2::2/128 via 2001:db8:3::2
-# ip -6 route add 3:3:3::3/128 via 2001:db8:4::3
-```
-
-6 - Edit /etc/frr/ospfd.conf:
-```
-router ospf
- network 4.4.4.4/32 area 0.0.0.0
- network 10.0.3.4/24 area 0.0.0.0
- network 10.0.4.4/24 area 0.0.0.0
-!
-```
-
-7 - Edit /etc/frr/ldpd.conf:
-```
-debug mpls ldp messages recv
-debug mpls ldp messages sent
-debug mpls ldp zebra
-!
-mpls ldp
- router-id 4.4.4.4
- dual-stack cisco-interop
- neighbor 1.1.1.1 password opensourcerouting
- neighbor 2.2.2.2 password opensourcerouting
- neighbor 3.3.3.3 password opensourcerouting
- !
- address-family ipv4
- discovery transport-address 4.4.4.4
- label local advertise explicit-null
- !
- interface eth2
- !
- interface eth1
- !
- !
- address-family ipv6
- discovery transport-address 4:4:4::4
- ttl-security disable
- !
- interface eth2
- !
- interface eth1
- !
- !
-!
-l2vpn ENG type vpls
- bridge br0
- member interface eth0
- !
- member pseudowire mpw0
- neighbor lsr-id 1.1.1.1
- pw-id 100
- !
- member pseudowire mpw1
- neighbor lsr-id 3.3.3.3
- neighbor address 3:3:3::3
- pw-id 100
- !
-!
-```
-
-> NOTE: We have to disable ttl-security under the ipv6 address-family
-in order to interoperate with the IOS-XR router. GTSM is mandatory for
-LDPv6 but the IOS-XR implementation is not RFC compliant in this regard.
-
-8 - Run zebra, ospfd and ldpd.
-
-#### OpenBSD
-1 - Enable IPv4/v6 forwarding:
-```
-# sysctl net.inet.ip.forwarding=1
-# sysctl net.inet6.ip6.forwarding=1
-```
-
-2 - Enable MPLS forwarding:
-```
-# ifconfig em2 10.0.2.3/24 mpls
-# ifconfig em3 10.0.4.3/24 mpls
-```
-
-3 - Set up the interfaces:
-```
-# ifconfig lo1 alias 3.3.3.3 netmask 255.255.255.255
-# ifconfig lo1 inet6 3:3:3::3/128
-# ifconfig em2 inet6 2001:db8:2::3/64
-# ifconfig em3 inet6 2001:db8:4::3/64
-```
-
-4 - Set up the bridge and pseudowire interfaces:
-```
-# ifconfig bridge0 create
-# ifconfig bridge0 up
-# ifconfig em1 up
-# ifconfig bridge0 add em1
-# ifconfig mpw0 create
-# ifconfig mpw0 up
-# ifconfig bridge0 add mpw0
-# ifconfig mpw1 create
-# ifconfig mpw1 up
-# ifconfig bridge0 add mpw1
-```
-
-5 - Add static IPv6 routes for the remote loopbacks:
-```
-# route -n add 4:4:4::4/128 2001:db8:4::4
-# route -n add 2:2:2::2/128 2001:db8:2::2
-```
-
-6 - Edit /etc/frr/ospfd.conf:
-```
-router ospf
- network 10.0.2.3/24 area 0
- network 10.0.4.3/24 area 0
- network 3.3.3.3/32 area 0
-!
-```
-
-7 - Edit /etc/frr/ldpd.conf:
-```
-debug mpls ldp messages recv
-debug mpls ldp messages sent
-debug mpls ldp zebra
-!
-mpls ldp
- router-id 3.3.3.3
- dual-stack cisco-interop
- neighbor 1.1.1.1 password opensourcerouting
- neighbor 2.2.2.2 password opensourcerouting
- neighbor 4.4.4.4 password opensourcerouting
- !
- address-family ipv4
- discovery transport-address 3.3.3.3
- label local advertise explicit-null
- !
- interface em3
- !
- interface em2
- !
- !
- address-family ipv6
- discovery transport-address 3:3:3::3
- ttl-security disable
- !
- interface em3
- !
- interface em2
- !
- !
-!
-l2vpn ENG type vpls
- bridge br0
- member interface em1
- !
- member pseudowire mpw0
- neighbor lsr-id 1.1.1.1
- pw-id 100
- !
- member pseudowire mpw1
- neighbor lsr-id 4.4.4.4
- neighbor address 4:4:4::4
- pw-id 100
- !
-!
-```
-
-8 - Run zebra, ospfd and ldpd.
-
-#### Cisco routers
-CE1 (IOS):
-```
-interface FastEthernet0/0
- ip address 172.16.1.1 255.255.255.0
- !
-!
-```
-
-CE2 (IOS):
-```
-interface FastEthernet0/0
- ip address 172.16.1.2 255.255.255.0
- !
-!
-```
-
-CE3 (IOS):
-```
-interface FastEthernet0/0
- ip address 172.16.1.3 255.255.255.0
- !
-!
-```
-
-PE1 - IOS-XE (1):
-```
-mpls ldp neighbor 2.2.2.2 password opensourcerouting
-mpls ldp neighbor 3.3.3.3 password opensourcerouting
-mpls ldp neighbor 4.4.4.4 password opensourcerouting
-!
-l2vpn vfi context VFI
- vpn id 1
- member pseudowire2
- member pseudowire1
-!
-bridge-domain 1
- member GigabitEthernet1 service-instance 1
- member vfi VFI
-!
-interface Loopback1
- ip address 1.1.1.1 255.255.255.255
-!
-interface pseudowire1
- encapsulation mpls
- neighbor 3.3.3.3 100
-!
-interface pseudowire2
- encapsulation mpls
- neighbor 4.4.4.4 100
-!
-interface GigabitEthernet3
- ip address 10.0.1.1 255.255.255.0
- mpls ip
-!
-router ospf 1
- network 0.0.0.0 255.255.255.255 area 0
-!
-```
-
-P - IOS-XR (2):
-```
-interface Loopback1
- ipv4 address 2.2.2.2 255.255.255.255
- ipv6 address 2:2:2::2/128
-!
-interface GigabitEthernet0/0/0/0
- ipv4 address 10.0.1.2 255.255.255.0
-!
-interface GigabitEthernet0/0/0/1
- ipv4 address 10.0.2.2 255.255.255.0
- ipv6 address 2001:db8:2::2/64
- ipv6 enable
-!
-interface GigabitEthernet0/0/0/2
- ipv4 address 10.0.3.2 255.255.255.0
- ipv6 address 2001:db8:3::2/64
- ipv6 enable
-!
-router static
- address-family ipv6 unicast
- 3:3:3::3/128 2001:db8:2::3
- 4:4:4::4/128 2001:db8:3::4
- !
-!
-router ospf 1
- router-id 2.2.2.2
- address-family ipv4 unicast
- area 0
- interface Loopback1
- !
- interface GigabitEthernet0/0/0/0
- !
- interface GigabitEthernet0/0/0/1
- !
- interface GigabitEthernet0/0/0/2
- !
- !
-!
-mpls ldp
- router-id 2.2.2.2
- neighbor
- 1.1.1.1:0 password clear opensourcerouting
- 3.3.3.3:0 password clear opensourcerouting
- 4.4.4.4:0 password clear opensourcerouting
- !
- address-family ipv4
- !
- address-family ipv6
- discovery transport-address 2:2:2::2
- !
- interface GigabitEthernet0/0/0/0
- address-family ipv4
- !
- !
- interface GigabitEthernet0/0/0/1
- address-family ipv4
- !
- address-family ipv6
- !
- !
- interface GigabitEthernet0/0/0/2
- address-family ipv4
- !
- address-family ipv6
- !
- !
-!
-```
-
-## Verification - Control Plane
-
-Using the CLI on the Linux box, the goal is to ensure that everything
-is working as expected.
-
-First, verify that all the required adjacencies and neighborships sessions
-were established:
-
-```
-linux# show mpls ldp discovery
-Local LDP Identifier: 4.4.4.4:0
-Discovery Sources:
- Interfaces:
- eth1: xmit/recv
- LDP Id: 3.3.3.3:0, Transport address: 3.3.3.3
- Hold time: 15 sec
- LDP Id: 3.3.3.3:0, Transport address: 3:3:3::3
- Hold time: 15 sec
- eth2: xmit/recv
- LDP Id: 2.2.2.2:0, Transport address: 2.2.2.2
- Hold time: 15 sec
- LDP Id: 2.2.2.2:0, Transport address: 2:2:2::2
- Hold time: 15 sec
- Targeted Hellos:
- 4.4.4.4 -> 1.1.1.1: xmit/recv
- LDP Id: 1.1.1.1:0, Transport address: 1.1.1.1
- Hold time: 45 sec
- 4:4:4::4 -> 3:3:3::3: xmit/recv
- LDP Id: 3.3.3.3:0, Transport address: 3:3:3::3
- Hold time: 45 sec
-
-linux# show mpls ldp neighbor
-Peer LDP Identifier: 1.1.1.1:0
- TCP connection: 4.4.4.4:40921 - 1.1.1.1:646
- Session Holdtime: 180 sec
- State: OPERATIONAL; Downstream-Unsolicited
- Up time: 00:06:02
- LDP Discovery Sources:
- IPv4:
- Targeted Hello: 1.1.1.1
-
-Peer LDP Identifier: 2.2.2.2:0
- TCP connection: 4:4:4::4:52286 - 2:2:2::2:646
- Session Holdtime: 180 sec
- State: OPERATIONAL; Downstream-Unsolicited
- Up time: 00:06:02
- LDP Discovery Sources:
- IPv4:
- Interface: eth2
- IPv6:
- Interface: eth2
-
-Peer LDP Identifier: 3.3.3.3:0
- TCP connection: 4:4:4::4:60575 - 3:3:3::3:646
- Session Holdtime: 180 sec
- State: OPERATIONAL; Downstream-Unsolicited
- Up time: 00:05:57
- LDP Discovery Sources:
- IPv4:
- Interface: eth1
- IPv6:
- Targeted Hello: 3:3:3::3
- Interface: eth1
-```
-
-Note that the neighborships with the P and PE2 routers were established
-over IPv6, since this is the default behavior for dual-stack LSRs, as
-specified in RFC 7552. If desired, the **dual-stack transport-connection
-prefer ipv4** command can be used to establish these sessions over IPv4
-(the command should be applied an all routers).
-
-Now, verify that there's a remote label for each PE address:
-```
-linux# show mpls ldp binding
-1.1.1.1/32
- Local binding: label: 20
- Remote bindings:
- Peer Label
- ----------------- ---------
- 1.1.1.1 imp-null
- 2.2.2.2 24000
- 3.3.3.3 20
-2.2.2.2/32
- Local binding: label: 21
- Remote bindings:
- Peer Label
- ----------------- ---------
- 1.1.1.1 18
- 2.2.2.2 imp-null
- 3.3.3.3 21
-3.3.3.3/32
- Local binding: label: 22
- Remote bindings:
- Peer Label
- ----------------- ---------
- 1.1.1.1 21
- 2.2.2.2 24003
- 3.3.3.3 imp-null
-4.4.4.4/32
- Local binding: label: imp-null
- Remote bindings:
- Peer Label
- ----------------- ---------
- 1.1.1.1 22
- 2.2.2.2 24001
- 3.3.3.3 22
-10.0.1.0/24
- Local binding: label: 23
- Remote bindings:
- Peer Label
- ----------------- ---------
- 1.1.1.1 imp-null
- 2.2.2.2 imp-null
- 3.3.3.3 23
-10.0.2.0/24
- Local binding: label: 24
- Remote bindings:
- Peer Label
- ----------------- ---------
- 1.1.1.1 20
- 2.2.2.2 imp-null
- 3.3.3.3 imp-null
-10.0.3.0/24
- Local binding: label: imp-null
- Remote bindings:
- Peer Label
- ----------------- ---------
- 1.1.1.1 19
- 2.2.2.2 imp-null
- 3.3.3.3 24
-10.0.4.0/24
- Local binding: label: imp-null
- Remote bindings:
- Peer Label
- ----------------- ---------
- 1.1.1.1 23
- 2.2.2.2 24002
- 3.3.3.3 imp-null
-2:2:2::2/128
- Local binding: label: 18
- Remote bindings:
- Peer Label
- ----------------- ---------
- 2.2.2.2 imp-null
- 3.3.3.3 18
-3:3:3::3/128
- Local binding: label: 19
- Remote bindings:
- Peer Label
- ----------------- ---------
- 2.2.2.2 24007
-4:4:4::4/128
- Local binding: label: imp-null
- Remote bindings:
- Peer Label
- ----------------- ---------
- 2.2.2.2 24006
- 3.3.3.3 19
-2001:db8:2::/64
- Local binding: label: -
- Remote bindings:
- Peer Label
- ----------------- ---------
- 2.2.2.2 imp-null
- 3.3.3.3 imp-null
-2001:db8:3::/64
- Local binding: label: imp-null
- Remote bindings:
- Peer Label
- ----------------- ---------
- 2.2.2.2 imp-null
-2001:db8:4::/64
- Local binding: label: imp-null
- Remote bindings:
- Peer Label
- ----------------- ---------
- 3.3.3.3 imp-null
-```
-
-Check if the pseudowires are up:
-```
-linux# show l2vpn atom vc
-Interface Peer ID VC ID Name Status
---------- --------------- ---------- ---------------- ----------
-mpw1 3.3.3.3 100 ENG UP
-mpw0 1.1.1.1 100 ENG UP
-```
-
-Check the label bindings of the pseudowires:
-```
-linux# show l2vpn atom binding
- Destination Address: 1.1.1.1, VC ID: 100
- Local Label: 25
- Cbit: 1, VC Type: Ethernet, GroupID: 0
- MTU: 1500
- Remote Label: 16
- Cbit: 1, VC Type: Ethernet, GroupID: 0
- MTU: 1500
- Destination Address: 3.3.3.3, VC ID: 100
- Local Label: 26
- Cbit: 1, VC Type: Ethernet, GroupID: 0
- MTU: 1500
- Remote Label: 26
- Cbit: 1, VC Type: Ethernet, GroupID: 0
- MTU: 1500
-```
-
-## Verification - Data Plane
-
-Verify that all the exchanged label mappings were installed in zebra:
-```
-linux# show mpls table
- Inbound Outbound
- Label Type Nexthop Label
--------- ------- --------------- --------
- 17 LDP 2001:db8:3::2 3
- 19 LDP 2001:db8:3::2 24005
- 20 LDP 10.0.3.2 24000
- 21 LDP 10.0.3.2 3
- 22 LDP 10.0.3.2 24001
- 23 LDP 10.0.3.2 3
- 24 LDP 10.0.3.2 3
- 25 LDP 10.0.3.2 3
-
-linux# show ip route ldp
-Codes: K - kernel route, C - connected, S - static, R - RIP,
- O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel, L - LDP,
- > - selected route, * - FIB route
-
-L>* 1.1.1.1/32 [0/0] via 10.0.3.2, eth2 label 24000
-L>* 3.3.3.3/32 [0/0] via 10.0.3.2, eth2 label 24001
-```
-
-Verify that all the exchanged label mappings were installed in the kernel:
-```
-$ ip -M ro
-17 via inet6 2001:db8:3::2 dev eth2 proto zebra
-19 as to 24005 via inet6 2001:db8:3::2 dev eth2 proto zebra
-20 as to 24000 via inet 10.0.3.2 dev eth2 proto zebra
-21 via inet 10.0.3.2 dev eth2 proto zebra
-22 as to 24001 via inet 10.0.3.2 dev eth2 proto zebra
-23 via inet 10.0.3.2 dev eth2 proto zebra
-24 via inet 10.0.3.2 dev eth2 proto zebra
-25 via inet 10.0.3.2 dev eth2 proto zebra
-$
-$ ip route | grep mpls
-1.1.1.1 encap mpls 24000 via 10.0.3.2 dev eth2 proto zebra metric 20
-3.3.3.3 encap mpls 24001 via 10.0.3.2 dev eth2 proto zebra metric 20
-```
-
-Now ping PE1's loopback using lo1's address as a source address:
-```
-$ ping -c 5 -I 4.4.4.4 1.1.1.1
-PING 1.1.1.1 (1.1.1.1) from 4.4.4.4 : 56(84) bytes of data.
-64 bytes from 1.1.1.1: icmp_seq=1 ttl=253 time=3.02 ms
-64 bytes from 1.1.1.1: icmp_seq=2 ttl=253 time=3.13 ms
-64 bytes from 1.1.1.1: icmp_seq=3 ttl=253 time=3.19 ms
-64 bytes from 1.1.1.1: icmp_seq=4 ttl=253 time=3.07 ms
-64 bytes from 1.1.1.1: icmp_seq=5 ttl=253 time=3.27 ms
-
---- 1.1.1.1 ping statistics ---
-5 packets transmitted, 5 received, 0% packet loss, time 4005ms
-rtt min/avg/max/mdev = 3.022/3.140/3.278/0.096 ms
-```
-
-Verify that the ICMP echo request packets are leaving with the MPLS
-label advertised by the P router. Also, verify that the ICMP echo reply
-packets are arriving with an explicit-null MPLS label:
-```
-# tcpdump -n -i eth2 mpls and icmp
-tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
-listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes
-10:01:40.758771 MPLS (label 24000, exp 0, [S], ttl 64) IP 4.4.4.4 > 1.1.1.1: ICMP echo request, id 13370, seq 1, length 64
-10:01:40.761777 MPLS (label 0, exp 0, [S], ttl 254) IP 1.1.1.1 > 4.4.4.4: ICMP echo reply, id 13370, seq 1, length 64
-10:01:41.760343 MPLS (label 24000, exp 0, [S], ttl 64) IP 4.4.4.4 > 1.1.1.1: ICMP echo request, id 13370, seq 2, length 64
-10:01:41.763448 MPLS (label 0, exp 0, [S], ttl 254) IP 1.1.1.1 > 4.4.4.4: ICMP echo reply, id 13370, seq 2, length 64
-10:01:42.761758 MPLS (label 24000, exp 0, [S], ttl 64) IP 4.4.4.4 > 1.1.1.1: ICMP echo request, id 13370, seq 3, length 64
-10:01:42.764924 MPLS (label 0, exp 0, [S], ttl 254) IP 1.1.1.1 > 4.4.4.4: ICMP echo reply, id 13370, seq 3, length 64
-10:01:43.763193 MPLS (label 24000, exp 0, [S], ttl 64) IP 4.4.4.4 > 1.1.1.1: ICMP echo request, id 13370, seq 4, length 64
-10:01:43.766237 MPLS (label 0, exp 0, [S], ttl 254) IP 1.1.1.1 > 4.4.4.4: ICMP echo reply, id 13370, seq 4, length 64
-10:01:44.764552 MPLS (label 24000, exp 0, [S], ttl 64) IP 4.4.4.4 > 1.1.1.1: ICMP echo request, id 13370, seq 5, length 64
-10:01:44.767803 MPLS (label 0, exp 0, [S], ttl 254) IP 1.1.1.1 > 4.4.4.4: ICMP echo reply, id 13370, seq 5, length 64
-```
--- /dev/null
+Next Hop Tracking
+==================
+
+Next hop tracking is an optimization feature that reduces the processing time
+involved in the BGP bestpath algorithm by monitoring changes to the routing
+table.
+
+Background
+-----------
+
+Recursive routes are of the form:
+
+::
+
+ p/m --> n
+ [Ex: 1.1.0.0/16 --> 2.2.2.2]
+
+where 'n' itself is resolved through another route as follows:
+
+::
+
+ p2/m --> h, interface
+ [Ex: 2.2.2.0/24 --> 3.3.3.3, eth0]
+
+Usually, BGP routes are recursive in nature and BGP nexthops get resolved
+through an IGP route. IGP usually adds its routes pointing to an interface
+(these are called non-recursive routes).
+
+When BGP receives a recursive route from a peer, it needs to validate the
+nexthop. The path is marked valid or invalid based on the reachability status
+of the nexthop. Nexthop validation is also important for BGP decision process
+as the metric to reach the nexthop is a parameter to best path selection
+process.
+
+As it goes with routing, this is a dynamic process. Route to the nexthop can
+change. The nexthop can become unreachable or reachable. In the current BGP
+implementation, the nexthop validation is done periodically in the scanner run.
+The default scanner run interval is one minute. Every minute, the scanner task
+walks the entire BGP table. It checks the validity of each nexthop with Zebra
+(the routing table manager) through a request and response message exchange
+between BGP and Zebra process. BGP process is blocked for that duration. The
+mechanism has two major drawbacks:
+
+- The scanner task runs to completion. That can potentially starve the other
+ tasks for long periods of time, based on the BGP table size and number of
+ nexthops.
+
+- Convergence around routing changes that affect the nexthops can be long
+ (around a minute with the default intervals). The interval can be shortened
+ to achieve faster reaction time, but it makes the first problem worse, with
+ the scanner task consuming most of the CPU resources.
+
+The next-hop tracking feature makes this process event-driven. It eliminates
+periodic nexthop validation and introduces an asynchronous communication path
+between BGP and Zebra for route change notifications that can then be acted
+upon.
+
+Goal
+----
+
+Stating the obvious, the main goal is to remove the two limitations we
+discussed in the previous section. The goals, in a constructive tone,
+are the following:
+
+- **Fairness**: the scanner run should not consume an unjustly high amount of
+ CPU time. This should give an overall good performance and response time to
+ other events (route changes, session events, IO/user interface).
+
+- **Convergence**: BGP must react to nexthop changes instantly and provide
+ sub-second convergence. This may involve diverting the routes from one
+ nexthop to another.
+
+Overview of changes
+------------------------
+
+The changes are in both BGP and Zebra modules. The short summary is
+the following:
+
+- Zebra implements a registration mechanism by which clients can
+ register for next hop notification. Consequently, it maintains a
+ separate table, per (VRF, AF) pair, of next hops and interested
+ client-list per next hop.
+
+- When the main routing table changes in Zebra, it evaluates the next
+ hop table: for each next hop, it checks if the route table
+ modifications have changed its state. If so, it notifies the
+ interested clients.
+
+- BGP is one such client. It registers the next hops corresponding to
+ all of its received routes/paths. It also threads the paths against
+ each nexthop structure.
+
+- When BGP receives a next hop notification from Zebra, it walks the
+ corresponding path list. It makes them valid or invalid depending
+ on the next hop notification. It then re-computes best path for the
+ corresponding destination. This may result in re-announcing those
+ destinations to peers.
+
+Design
+------
+
+Modules
+~~~~~~~
+
+The core design introduces an "nht" (next hop tracking) module in BGP
+and "rnh" (recursive nexthop) module in Zebra. The "nht" module
+provides the following APIs:
+
++----------------------------+--------------------------------------------------+
+| Function | Action |
++============================+==================================================+
+| bgp_find_or_add_nexthop() | find or add a nexthop in BGP nexthop table |
++----------------------------+--------------------------------------------------+
+| bgp_find_nexthop() | find a nexthop in BGP nexthop table |
++----------------------------+--------------------------------------------------+
+| bgp_parse_nexthop_update() | parse a nexthop update message coming from zebra |
++----------------------------+--------------------------------------------------+
+
+The "rnh" module provides the following APIs:
+
++----------------------------+----------------------------------------------------------------------------------------------------------+
+| Function | Action |
++============================+==========================================================================================================+
+| zebra_add_rnh() | add a recursive nexthop |
++----------------------------+----------------------------------------------------------------------------------------------------------+
+| zebra_delete_rnh() | delete a recursive nexthop |
++----------------------------+----------------------------------------------------------------------------------------------------------+
+| zebra_lookup_rnh() | lookup a recursive nexthop |
++----------------------------+----------------------------------------------------------------------------------------------------------+
+| zebra_add_rnh_client() | register a client for nexthop notifications against a recursive nexthop |
++----------------------------+----------------------------------------------------------------------------------------------------------+
+| zebra_remove_rnh_client() | remove the client registration for a recursive nexthop |
++----------------------------+----------------------------------------------------------------------------------------------------------+
+| zebra_evaluate_rnh_table() | (re)evaluate the recursive nexthop table (most probably because the main routing table has changed). |
++----------------------------+----------------------------------------------------------------------------------------------------------+
+| zebra_cleanup_rnh_client() | Cleanup a client from the "rnh" module data structures (most probably because the client is going away). |
++----------------------------+----------------------------------------------------------------------------------------------------------+
+
+4.2. Control flow
+
+The next hop registration control flow is the following:
+
+::
+
+ <==== BGP Process ====>|<==== Zebra Process ====>
+ |
+ receive module nht module | zserv module rnh module
+ ----------------------------------------------------------------------
+ | | |
+ bgp_update_ | | |
+ main() | bgp_find_or_add_ | |
+ | nexthop() | |
+ | | |
+ | | zserv_nexthop_ |
+ | | register() |
+ | | | zebra_add_rnh()
+ | | |
+
+
+The next hop notification control flow is the following:
+
+::
+
+ <==== Zebra Process ====>|<==== BGP Process ====>
+ |
+ rib module rnh module | zebra module nht module
+ ----------------------------------------------------------------------
+ | | |
+ meta_queue_ | | |
+ process() | zebra_evaluate_ | |
+ | rnh_table() | |
+ | | |
+ | | bgp_read_nexthop_ |
+ | | update() |
+ | | | bgp_parse_
+ | | | nexthop_update()
+ | | |
+
+
+zclient message format
+~~~~~~~~~~~~~~~~~~~~~~
+
+ZEBRA_NEXTHOP_REGISTER and ZEBRA_NEXTHOP_UNREGISTER messages are
+encoded in the following way:
+
+::
+
+ . 0 1 2 3
+ 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | AF | prefix len |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ . Nexthop prefix .
+ . .
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ . .
+ . .
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | AF | prefix len |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ . Nexthop prefix .
+ . .
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+
+``ZEBRA_NEXTHOP_UPDATE`` message is encoded as follows:
+
+::
+
+ . 0 1 2 3
+ 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | AF | prefix len |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ . Nexthop prefix getting resolved .
+ . .
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | metric |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | #nexthops |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | nexthop type |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ . resolving Nexthop details .
+ . .
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ . .
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ | nexthop type |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ . resolving Nexthop details .
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+
+BGP data structure
+~~~~~~~~~~~~~~~~~~
+Legend:
+
+::
+
+ /\ struct bgp_node: a BGP destination/route/prefix
+ \/
+
+ [ ] struct bgp_info: a BGP path (e.g. route received from a peer)
+
+ _
+ (_) struct bgp_nexthop_cache: a BGP nexthop
+
+ /\ NULL
+ \/--+ ^
+ | :
+ +--[ ]--[ ]--[ ]--> NULL
+ /\ :
+ \/--+ :
+ | :
+ +--[ ]--[ ]--> NULL
+ :
+ _ :
+ (_)...........
+
+
+Zebra data structure
+~~~~~~~~~~~~~~~~~~~~
+
+RNH table::
+
+ O
+ / \
+ O O
+ / \
+ O O
+
+ struct rnh
+ {
+ u_char flags;
+ struct route_entry *state;
+ struct list *client_list;
+ struct route_node *node;
+ };
+
+User interface changes
+~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ frr# show ip nht
+ 3.3.3.3
+ resolved via kernel
+ via 11.0.0.6, swp1
+ Client list: bgp(fd 12)
+ 11.0.0.10
+ resolved via connected
+ is directly connected, swp2
+ Client list: bgp(fd 12)
+ 11.0.0.18
+ resolved via connected
+ is directly connected, swp4
+ Client list: bgp(fd 12)
+ 11.11.11.11
+ resolved via kernel
+ via 10.0.1.2, eth0
+ Client list: bgp(fd 12)
+
+ frr# show ip bgp nexthop
+ Current BGP nexthop cache:
+ 3.3.3.3 valid [IGP metric 0], #paths 3
+ Last update: Wed Oct 16 04:43:49 2013
+
+ 11.0.0.10 valid [IGP metric 1], #paths 1
+ Last update: Wed Oct 16 04:43:51 2013
+
+ 11.0.0.18 valid [IGP metric 1], #paths 2
+ Last update: Wed Oct 16 04:43:47 2013
+
+ 11.11.11.11 valid [IGP metric 0], #paths 1
+ Last update: Wed Oct 16 04:43:47 2013
+
+ frr# show ipv6 nht
+ frr# show ip bgp nexthop detail
+
+ frr# debug bgp nht
+ frr# debug zebra nht
+
+ 6. Sample test cases
+
+ r2----r3
+ / \ /
+ r1----r4
+
+ - Verify that a change in IGP cost triggers NHT
+ + shutdown the r1-r4 and r2-r4 links
+ + no shut the r1-r4 and r2-r4 links and wait for OSPF to come back
+ up
+ + We should be back to the original nexthop via r4 now
+ - Verify that a NH becoming unreachable triggers NHT
+ + Shutdown all links to r4
+ - Verify that a NH becoming reachable triggers NHT
+ + no shut all links to r4
+
+Future work
+~~~~~~~~~~~
+
+- route-policy for next hop validation (e.g. ignore default route)
+- damping for rapid next hop changes
+- prioritized handling of nexthop changes ((un)reachability vs. metric
+ changes)
+- handling recursion loop, e.g::
+
+ 11.11.11.11/32 -> 12.12.12.12
+ 12.12.12.12/32 -> 11.11.11.11
+ 11.0.0.0/8 -> <interface>
+- better statistics
+++ /dev/null
-0. Introduction
-
-This is the design specification for next hop tracking feature in
-Frr.
-
-1. Background
-
-Recursive routes are of the form:
-
- p/m --> n
- [Ex: 1.1.0.0/16 --> 2.2.2.2]
-
-where 'n' itself is resolved through another route as follows:
-
- p2/m --> h, interface
- [Ex: 2.2.2.0/24 --> 3.3.3.3, eth0]
-
-Usually, BGP routes are recursive in nature and BGP nexthops get
-resolved through an IGP route. IGP usually adds its routes pointing to
-an interface (these are called non-recursive routes).
-
-When BGP receives a recursive route from a peer, it needs to validate
-the nexthop. The path is marked valid or invalid based on the
-reachability status of the nexthop. Nexthop validation is also
-important for BGP decision process as the metric to reach the nexthop
-is a parameter to best path selection process.
-
-As it goes with routing, this is a dynamic process. Route to the
-nexthop can change. The nexthop can become unreachable or
-reachable. In the current BGP implementation, the nexthop validation
-is done periodically in the scanner run. The default scanner run
-interval is one minute. Every minute, the scanner task walks the
-entire BGP table. It checks the validity of each nexthop with Zebra
-(the routing table manager) through a request and response message
-exchange between BGP and Zebra process. BGP process is blocked for
-that duration. The mechanism has two major drawbacks:
-
-(1) The scanner task runs to completion. That can potentially starve
- the other tasks for long periods of time, based on the BGP table
- size and number of nexthops.
-
-(2) Convergence around routing changes that affect the nexthops can be
- long (around a minute with the default intervals). The interval
- can be shortened to achieve faster reaction time, but it makes the
- first problem worse, with the scanner task consuming most of the
- CPU resources.
-
-"Next hop tracking" feature makes this process event-driven. It
-eliminates periodic nexthop validation and introduces an asynchronous
-communication path between BGP and Zebra for route change notifications
-that can then be acted upon.
-
-2. Goal
-
-Stating the obvious, the main goal is to remove the two limitations we
-discussed in the previous section. The goals, in a constructive tone,
-are the following:
-
-- fairness: the scanner run should not consume an unjustly high amount
- of CPU time. This should give an overall good performance and
- response time to other events (route changes, session events,
- IO/user interface).
-
-- convergence: BGP must react to nexthop changes instantly and provide
- sub-second convergence. This may involve diverting the routes from
- one nexthop to another.
-
-3. Overview of the changes
-
-The changes are in both BGP and Zebra modules. The short summary is
-the following:
-
-- Zebra implements a registration mechanism by which clients can
- register for next hop notification. Consequently, it maintains a
- separate table, per (VRF, AF) pair, of next hops and interested
- client-list per next hop.
-
-- When the main routing table changes in Zebra, it evaluates the next
- hop table: for each next hop, it checks if the route table
- modifications have changed its state. If so, it notifies the
- interested clients.
-
-- BGP is one such client. It registers the next hops corresponding to
- all of its received routes/paths. It also threads the paths against
- each nexthop structure.
-
-- When BGP receives a next hop notification from Zebra, it walks the
- corresponding path list. It makes them valid or invalid depending
- on the next hop notification. It then re-computes best path for the
- corresponding destination. This may result in re-announcing those
- destinations to peers.
-
-4. Design
-
-4.1. Modules
-
-The core design introduces an "nht" (next hop tracking) module in BGP
-and "rnh" (recursive nexthop) module in Zebra. The "nht" module
-provides the following APIs:
-
-bgp_find_or_add_nexthop() : find or add a nexthop in BGP nexthop table
-bgp_find_nexthop() : find a nexthop in BGP nexthop table
-bgp_parse_nexthop_update() : parse a nexthop update message coming
- from zebra
-
-The "rnh" module provides the following APIs:
-
-zebra_add_rnh() : add a recursive nexthop
-zebra_delete_rnh() : delete a recursive nexthop
-zebra_lookup_rnh() : lookup a recursive nexthop
-
-zebra_add_rnh_client() : register a client for nexthop notifications
- against a recursive nexthop
-
-zebra_remove_rnh_client(): remove the client registration for a
- recursive nexthop
-
-zebra_evaluate_rnh_table(): (re)evaluate the recursive nexthop table
- (most probably because the main routing
- table has changed).
-
-zebra_cleanup_rnh_client(): Cleanup a client from the "rnh" module
- data structures (most probably because the
- client is going away).
-
-4.2. Control flow
-
-The next hop registration control flow is the following:
-
-<==== BGP Process ====>|<==== Zebra Process ====>
- |
-receive module nht module | zserv module rnh module
-----------------------------------------------------------------------
- | | |
-bgp_update_ | | |
- main() | bgp_find_or_add_ | |
- | nexthop() | |
- | | |
- | | zserv_nexthop_ |
- | | register() |
- | | | zebra_add_rnh()
- | | |
-
-
-The next hop notification control flow is the following:
-
-<==== Zebra Process ====>|<==== BGP Process ====>
- |
-rib module rnh module | zebra module nht module
-----------------------------------------------------------------------
- | | |
-meta_queue_ | | |
- process() | zebra_evaluate_ | |
- | rnh_table() | |
- | | |
- | | bgp_read_nexthop_ |
- | | update() |
- | | | bgp_parse_
- | | | nexthop_update()
- | | |
-
-
-4.3. zclient message format
-
-ZEBRA_NEXTHOP_REGISTER and ZEBRA_NEXTHOP_UNREGISTER messages are
-encoded in the following way:
-
-/*
- * 0 1 2 3
- * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * | AF | prefix len |
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * . Nexthop prefix .
- * . .
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * . .
- * . .
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * | AF | prefix len |
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * . Nexthop prefix .
- * . .
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- */
-
-ZEBRA_NEXTHOP_UPDATE message is encoded as follows:
-
-/*
- * 0 1 2 3
- * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * | AF | prefix len |
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * . Nexthop prefix getting resolved .
- * . .
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * | metric |
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * | #nexthops |
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * | nexthop type |
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * . resolving Nexthop details .
- * . .
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * . .
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * | nexthop type |
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- * . resolving Nexthop details .
- * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- */
-
-4.4. BGP data structure
-
-Legend:
-
-/\ struct bgp_node: a BGP destination/route/prefix
-\/
-
-[ ] struct bgp_info: a BGP path (e.g. route received from a peer)
-
- _
-(_) struct bgp_nexthop_cache: a BGP nexthop
-
-
-
- /\ NULL
- \/--+ ^
- | :
- +--[ ]--[ ]--[ ]--> NULL
- /\ :
- \/--+ :
- | :
- +--[ ]--[ ]--> NULL
- :
- _ :
- (_).............
-
-
-4.5. Zebra data structure
-
-rnh table:
-
- O
- / \
- O O
- / \
- O O
-
- struct rnh
- {
- u_char flags;
- struct route_entry *state;
- struct list *client_list;
- struct route_node *node;
- };
-
-5. User interface changes
-
-frr# show ip nht
-3.3.3.3
- resolved via kernel
- via 11.0.0.6, swp1
- Client list: bgp(fd 12)
-11.0.0.10
- resolved via connected
- is directly connected, swp2
- Client list: bgp(fd 12)
-11.0.0.18
- resolved via connected
- is directly connected, swp4
- Client list: bgp(fd 12)
-11.11.11.11
- resolved via kernel
- via 10.0.1.2, eth0
- Client list: bgp(fd 12)
-
-frr# show ip bgp nexthop
-Current BGP nexthop cache:
- 3.3.3.3 valid [IGP metric 0], #paths 3
- Last update: Wed Oct 16 04:43:49 2013
-
- 11.0.0.10 valid [IGP metric 1], #paths 1
- Last update: Wed Oct 16 04:43:51 2013
-
- 11.0.0.18 valid [IGP metric 1], #paths 2
- Last update: Wed Oct 16 04:43:47 2013
-
- 11.11.11.11 valid [IGP metric 0], #paths 1
- Last update: Wed Oct 16 04:43:47 2013
-
-frr# show ipv6 nht
-frr# show ip bgp nexthop detail
-
-frr# debug bgp nht
-frr# debug zebra nht
-
-6. Sample test cases
-
- r2----r3
- / \ /
- r1----r4
-
-- Verify that a change in IGP cost triggers NHT
- + shutdown the r1-r4 and r2-r4 links
- + no shut the r1-r4 and r2-r4 links and wait for OSPF to come back
- up
- + We should be back to the original nexthop via r4 now
-- Verify that a NH becoming unreachable triggers NHT
- + Shutdown all links to r4
-- Verify that a NH becoming reachable triggers NHT
- + no shut all links to r4
-
-7. Future work
-
-- route-policy for next hop validation (e.g. ignore default route)
-- damping for rapid next hop changes
-- prioritized handling of nexthop changes ((un)reachability vs. metric
- changes)
-- handling recursion loop, e.g.
- 11.11.11.11/32 -> 12.12.12.12
- 12.12.12.12/32 -> 11.11.11.11
- 11.0.0.0/8 -> <interface>
-- better statistics
-Developing for FRRouting
+Process & Workflow
========================
-General note on this document
------------------------------
-
-This document is "descriptive/post-factual" in that it documents
-pratices that are in use; it is not "definitive/pre-factual" in
-prescribing practices.
+FRR is a large project developed by many different groups. This section
+documents standards for code style & quality, commit messages, pull requests
+and best practices that all contributors are asked to follow.
-This means that when a procedure changes, it is agreed upon, then put
-into practice, and then documented here. If this document doesn't match
-reality, it's the document that needs to be updated, not reality.
+This section is "descriptive/post-factual" in that it documents pratices that
+are in use; it is not "definitive/pre-factual" in prescribing practices. This
+means that when a procedure changes, it is agreed upon, then put into practice,
+and then documented here. If this document doesn't match reality, it's the
+document that needs to be updated, not reality.
Git Structure
-------------
-The master Git for FRRouting resides on Github at
-`https://github.com/frrouting/frr <https://github.com/FRRouting/frr>`__
-
-.. figure:: git_branches.svg
- :alt: git branches continually merging to the left from 3 lanes;
- float-right
-
- git branches continually merging to the left from 3 lanes;
- float-right
+The master Git for FRR resides on `Github
+<https://github.com/frrouting/frr>`__.
-There is one main branch for development and a release branch for each
-major release.
+.. figure:: git_branches.png
-New contributions are done against the head of the master branch. The CI
-systems will pick up the Github Pull Requests or the new patch from
-Patchwork, run some basic build and functional tests.
-
-For each major release (1.0, 1.1 etc) a new release branch is created
-based on the master.
-
-There was an attempt to use a "develop" branch automatically maintained
-by the CI system. This is not currently in active use, though the system
-is operational. If the "develop" branch is in active use and this
-paragraph is still here, this document obviously wasn't updated.
+There is one main branch for development, ``master``. For each major release
+(2.0, 3.0 etc) a new release branch is created based on the master. Subsequent
+point releases based on a major branch are marked by tagging.
Programming language, Tools and Libraries
-----------------------------------------
-The core of FRRouting is written in C (gcc or clang supported) and makes
+The core of FRR is written in C (gcc or clang supported) and makes
use of GNU compiler extensions. A few non-essential scripts are
-implemented in Perl and Python. FRRouting requires the following tools
+implemented in Perl and Python. FRR requires the following tools
to build distribution packages: automake, autoconf, texinfo, libtool and
gawk and various libraries (i.e. libpam and libjson-c).
If your contribution requires a new library or other tool, then please
highlight this in your description of the change. Also make sure it’s
-supported by all FRRouting platform OSes or provide a way to build
+supported by all FRR platform OSes or provide a way to build
without the library (potentially without the new feature) on the other
platforms.
-Documentation should be written in Tex (.texi) or Markdown (.md) format
-with a preference for Markdown.
+Documentation should be written in reStructuredText. Sphinx extensions may be
+utilized but pure ReST is preferred where possible. See `Documentation
+<#documentation>`__.
Mailing lists
-------------
-Italicized lists are private.
+The FRR development group maintains multiple mailing lists for use by the
+community. Italicized lists are private.
+----------------------------------+--------------------------------+
| Topic | List |
Changelog
~~~~~~~~~
-The changelog will be the base for the release notes. A changelog entry
-for your changes is usually not required and will be added based on your
-commit messages by the maintainers. However, you are free to include an
-update to the changelog with some better description. The changelog will
-be the base for the release notes.
+The changelog will be the base for the release notes. A changelog entry for
+your changes is usually not required and will be added based on your commit
+messages by the maintainers. However, you are free to include an update to the
+changelog with some better description.
Submitting Patches and Enhancements
-----------------------------------
+FRR accepts patches from two sources:
+
+- Email (git format-patch)
+- Github pull request
+
+Contributors are highly encouraged to use Github's fork-and-pr workflow. It is
+easier for us to review it, test it, try it and discuss it on Github than it is
+via email, thus your patch will get more attention more quickly on Github.
+
+The base branch for new contributions and non-critical bug fixes should be
+``master``. Please ensure your pull request is based on this branch when you
+submit it.
+
Pre-submission Checklist
~~~~~~~~~~~~~~~~~~~~~~~~
-- Format code (see `Developer's Guidelines <#developers-guidelines>`__)
+- Format code (see `Code Formatting <#developers-guidelines>`__)
- Verify and acknowledge license (see `License for
contributions <#license-for-contributions>`__)
- Ensure you have properly signed off (see `Signing
License for contributions
~~~~~~~~~~~~~~~~~~~~~~~~~
-FRRouting is under a “GPLv2 or later” license. Any code submitted must
+FRR is under a “GPLv2 or later” license. Any code submitted must
be released under the same license (preferred) or any license which
allows redistribution under this GPLv2 license (eg MIT License).
Signing Off
~~~~~~~~~~~
-Code submitted to FRRouting must be signed off. We have the same
+Code submitted to FRR must be signed off. We have the same
requirements for using the signed-off-by process as the Linux kernel. In
short, you must include a signed-off-by tag in every patch.
In short, when you sign off on a commit, you assert your agreement to
all of the following:
+::
+
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
We've documented where we would like to have the different fixes applied
at
-https://github.com/FRRouting/frr/wiki/Where-Do-I-create-a-Pull-Request-against%3F
+https://github.com/FRR/frr/wiki/Where-Do-I-create-a-Pull-Request-against%3F
If you are unsure where your submission goes, look at that document or
ask a project maintainer.
- You should automatically receive an email with the test results
within less than 2 hrs of the submission. If you don’t get the
- email, then check status on the github pull request (if submitted
- by pull request) or on Patchwork at
- https://patchwork.frrouting.org (if submitted as patch to mailing
- list).
+ email, then check status on the Github pull request.
- Please notify the development mailing list if you think something
- doesn’t work.
+ doesn't work.
- If the tests failed:
community members.
- Your submission is done once it is merged to the master branch.
-Developer's Guidelines
-----------------------
+Coding Practices & Style
+------------------------
Commit messages
~~~~~~~~~~~~~~~
Documentation
~~~~~~~~~~~~~
-FRRouting is a large and complex software project developed by many
-different people over a long period of time. Without adequate
-documentation, it can be exceedingly difficult to understand code
-segments, APIs and other interfaces. In the interest of keeping the
-project healthy and maintainable, you should make every effort to
-document your code so that other people can understand what it does
-without needing to closely read the code itself.
+FRR is a large and complex software project developed by many different people
+over a long period of time. Without adequate documentation, it can be
+exceedingly difficult to understand code segments, APIs and other interfaces.
+In the interest of keeping the project healthy and maintainable, you should
+make every effort to document your code so that other people can understand
+what it does without needing to closely read the code itself.
Some specific guidelines that contributors should follow are:
- **For new code in ``lib/``, these guidelines are hard requirements.**
If you are contributing code that adds significant user-visible
-functionality or introduces a new API, please document it in ``doc/``.
-Markdown and LaTeX are acceptable formats, although Markdown is
-currently preferred for new documentation. This may change in the near
-future.
+functionality please document it in ``doc/``. If you make significant changes
+to portions of the codebase covered in the Developer's Manual, please
+update the relevant sections. If you add a major feature or introduce a new
+API, please document the architecture and API to the best of your abilities in
+the Developer's Manual.
+
+Documentation should be in reStructuredText.
Finally, if you come across some code that is undocumented and feel like
going above and beyond, document it! We absolutely appreciate and accept
--- /dev/null
+## Topology
+
+The goal of this test is to verify that the all the basic functionality
+of ldpd is working as expected, be it running on Linux or OpenBSD. In
+addition to that, more advanced features are also tested, like LDP
+sessions over IPv6, MD5 authentication and pseudowire signaling.
+
+In the topology below there are 3 PE routers, 3 CE routers and one P
+router (not attached to any consumer site).
+
+All routers have IPv4 addresses and OSPF is used as the IGP. The
+three routers from the bottom of the picture, P, PE2 and PE3, are also
+configured for IPv6 (dual-stack) and static IPv6 routes are used to
+provide connectivity among them.
+
+The three CEs share the same VPLS membership. LDP is used to set up the
+LSPs among the PEs and to signal the pseudowires. MD5 authentication is
+used to protect all LDP sessions.
+
+```
+ CE1 172.16.1.1/24
+ +
+ |
+ +---+---+
+ | PE1 |
+ | IOS XE|
+ | |
+ +---+---+
+ |
+ | 10.0.1.0/24
+ |
+ +---+---+
+ | P |
+ +------+ IOS XR+------+
+ | | | |
+ | +-------+ |
+ 10.0.2.0/24 | | 10.0.3.0/24
+2001:db8:2::/64 | | 2001:db8:3::/64
+ | |
+ +---+---+ +---+---+
+ | PE2 | | PE3 |
+ |OpenBSD+-------------+ Linux |
+ | | | |
+ +---+---+ 10.0.4.0/24 +---+---+
+ | 2001:db8:4::/64 |
+ + +
+ 172.16.1.2/24 CE2 CE3 172.16.1.3/24
+```
+
+## Configuration
+
+#### Linux
+1 - Enable IPv4/v6 forwarding:
+```
+# sysctl -w net.ipv4.ip_forward=1
+# sysctl -w net.ipv6.conf.all.forwarding=1
+```
+
+2 - Enable MPLS forwarding:
+```
+# modprobe mpls-router
+# modprobe mpls-iptunnel
+# echo 100000 > /proc/sys/net/mpls/platform_labels
+# echo 1 > /proc/sys/net/mpls/conf/eth1/input
+# echo 1 > /proc/sys/net/mpls/conf/eth2/input
+```
+
+3 - Set up the interfaces:
+```
+# ip link add name lo1 type dummy
+# ip link set dev lo1 up
+# ip addr add 4.4.4.4/32 dev lo1
+# ip -6 addr add 4:4:4::4/128 dev lo1
+# ip link set dev eth1 up
+# ip addr add 10.0.4.4/24 dev eth1
+# ip -6 addr add 2001:db8:4::4/64 dev eth1
+# ip link set dev eth2 up
+# ip addr add 10.0.3.4/24 dev eth2
+# ip -6 addr add 2001:db8:3::4/64 dev eth2
+```
+
+4 - Set up the bridge and pseudowire interfaces:
+```
+# ip link add type bridge
+# ip link set dev bridge0 up
+# ip link set dev eth0 up
+# ip link set dev eth0 master bridge0
+# ip link add name mpw0 type dummy
+# ip link set dev mpw0 up
+# ip link set dev mpw0 master bridge0
+# ip link add name mpw1 type dummy
+# ip link set dev mpw1 up
+# ip link set dev mpw1 master bridge0
+```
+
+> NOTE: MPLS support in the Linux kernel is very recent and it still
+doesn't support pseudowire interfaces. We are using here dummy interfaces
+just to show how the VPLS configuration should look like in the future.
+
+5 - Add static IPv6 routes for the remote loopbacks:
+```
+# ip -6 route add 2:2:2::2/128 via 2001:db8:3::2
+# ip -6 route add 3:3:3::3/128 via 2001:db8:4::3
+```
+
+6 - Edit /etc/frr/ospfd.conf:
+```
+router ospf
+ network 4.4.4.4/32 area 0.0.0.0
+ network 10.0.3.4/24 area 0.0.0.0
+ network 10.0.4.4/24 area 0.0.0.0
+!
+```
+
+7 - Edit /etc/frr/ldpd.conf:
+```
+debug mpls ldp messages recv
+debug mpls ldp messages sent
+debug mpls ldp zebra
+!
+mpls ldp
+ router-id 4.4.4.4
+ dual-stack cisco-interop
+ neighbor 1.1.1.1 password opensourcerouting
+ neighbor 2.2.2.2 password opensourcerouting
+ neighbor 3.3.3.3 password opensourcerouting
+ !
+ address-family ipv4
+ discovery transport-address 4.4.4.4
+ label local advertise explicit-null
+ !
+ interface eth2
+ !
+ interface eth1
+ !
+ !
+ address-family ipv6
+ discovery transport-address 4:4:4::4
+ ttl-security disable
+ !
+ interface eth2
+ !
+ interface eth1
+ !
+ !
+!
+l2vpn ENG type vpls
+ bridge br0
+ member interface eth0
+ !
+ member pseudowire mpw0
+ neighbor lsr-id 1.1.1.1
+ pw-id 100
+ !
+ member pseudowire mpw1
+ neighbor lsr-id 3.3.3.3
+ neighbor address 3:3:3::3
+ pw-id 100
+ !
+!
+```
+
+> NOTE: We have to disable ttl-security under the ipv6 address-family
+in order to interoperate with the IOS-XR router. GTSM is mandatory for
+LDPv6 but the IOS-XR implementation is not RFC compliant in this regard.
+
+8 - Run zebra, ospfd and ldpd.
+
+#### OpenBSD
+1 - Enable IPv4/v6 forwarding:
+```
+# sysctl net.inet.ip.forwarding=1
+# sysctl net.inet6.ip6.forwarding=1
+```
+
+2 - Enable MPLS forwarding:
+```
+# ifconfig em2 10.0.2.3/24 mpls
+# ifconfig em3 10.0.4.3/24 mpls
+```
+
+3 - Set up the interfaces:
+```
+# ifconfig lo1 alias 3.3.3.3 netmask 255.255.255.255
+# ifconfig lo1 inet6 3:3:3::3/128
+# ifconfig em2 inet6 2001:db8:2::3/64
+# ifconfig em3 inet6 2001:db8:4::3/64
+```
+
+4 - Set up the bridge and pseudowire interfaces:
+```
+# ifconfig bridge0 create
+# ifconfig bridge0 up
+# ifconfig em1 up
+# ifconfig bridge0 add em1
+# ifconfig mpw0 create
+# ifconfig mpw0 up
+# ifconfig bridge0 add mpw0
+# ifconfig mpw1 create
+# ifconfig mpw1 up
+# ifconfig bridge0 add mpw1
+```
+
+5 - Add static IPv6 routes for the remote loopbacks:
+```
+# route -n add 4:4:4::4/128 2001:db8:4::4
+# route -n add 2:2:2::2/128 2001:db8:2::2
+```
+
+6 - Edit /etc/frr/ospfd.conf:
+```
+router ospf
+ network 10.0.2.3/24 area 0
+ network 10.0.4.3/24 area 0
+ network 3.3.3.3/32 area 0
+!
+```
+
+7 - Edit /etc/frr/ldpd.conf:
+```
+debug mpls ldp messages recv
+debug mpls ldp messages sent
+debug mpls ldp zebra
+!
+mpls ldp
+ router-id 3.3.3.3
+ dual-stack cisco-interop
+ neighbor 1.1.1.1 password opensourcerouting
+ neighbor 2.2.2.2 password opensourcerouting
+ neighbor 4.4.4.4 password opensourcerouting
+ !
+ address-family ipv4
+ discovery transport-address 3.3.3.3
+ label local advertise explicit-null
+ !
+ interface em3
+ !
+ interface em2
+ !
+ !
+ address-family ipv6
+ discovery transport-address 3:3:3::3
+ ttl-security disable
+ !
+ interface em3
+ !
+ interface em2
+ !
+ !
+!
+l2vpn ENG type vpls
+ bridge br0
+ member interface em1
+ !
+ member pseudowire mpw0
+ neighbor lsr-id 1.1.1.1
+ pw-id 100
+ !
+ member pseudowire mpw1
+ neighbor lsr-id 4.4.4.4
+ neighbor address 4:4:4::4
+ pw-id 100
+ !
+!
+```
+
+8 - Run zebra, ospfd and ldpd.
+
+#### Cisco routers
+CE1 (IOS):
+```
+interface FastEthernet0/0
+ ip address 172.16.1.1 255.255.255.0
+ !
+!
+```
+
+CE2 (IOS):
+```
+interface FastEthernet0/0
+ ip address 172.16.1.2 255.255.255.0
+ !
+!
+```
+
+CE3 (IOS):
+```
+interface FastEthernet0/0
+ ip address 172.16.1.3 255.255.255.0
+ !
+!
+```
+
+PE1 - IOS-XE (1):
+```
+mpls ldp neighbor 2.2.2.2 password opensourcerouting
+mpls ldp neighbor 3.3.3.3 password opensourcerouting
+mpls ldp neighbor 4.4.4.4 password opensourcerouting
+!
+l2vpn vfi context VFI
+ vpn id 1
+ member pseudowire2
+ member pseudowire1
+!
+bridge-domain 1
+ member GigabitEthernet1 service-instance 1
+ member vfi VFI
+!
+interface Loopback1
+ ip address 1.1.1.1 255.255.255.255
+!
+interface pseudowire1
+ encapsulation mpls
+ neighbor 3.3.3.3 100
+!
+interface pseudowire2
+ encapsulation mpls
+ neighbor 4.4.4.4 100
+!
+interface GigabitEthernet3
+ ip address 10.0.1.1 255.255.255.0
+ mpls ip
+!
+router ospf 1
+ network 0.0.0.0 255.255.255.255 area 0
+!
+```
+
+P - IOS-XR (2):
+```
+interface Loopback1
+ ipv4 address 2.2.2.2 255.255.255.255
+ ipv6 address 2:2:2::2/128
+!
+interface GigabitEthernet0/0/0/0
+ ipv4 address 10.0.1.2 255.255.255.0
+!
+interface GigabitEthernet0/0/0/1
+ ipv4 address 10.0.2.2 255.255.255.0
+ ipv6 address 2001:db8:2::2/64
+ ipv6 enable
+!
+interface GigabitEthernet0/0/0/2
+ ipv4 address 10.0.3.2 255.255.255.0
+ ipv6 address 2001:db8:3::2/64
+ ipv6 enable
+!
+router static
+ address-family ipv6 unicast
+ 3:3:3::3/128 2001:db8:2::3
+ 4:4:4::4/128 2001:db8:3::4
+ !
+!
+router ospf 1
+ router-id 2.2.2.2
+ address-family ipv4 unicast
+ area 0
+ interface Loopback1
+ !
+ interface GigabitEthernet0/0/0/0
+ !
+ interface GigabitEthernet0/0/0/1
+ !
+ interface GigabitEthernet0/0/0/2
+ !
+ !
+!
+mpls ldp
+ router-id 2.2.2.2
+ neighbor
+ 1.1.1.1:0 password clear opensourcerouting
+ 3.3.3.3:0 password clear opensourcerouting
+ 4.4.4.4:0 password clear opensourcerouting
+ !
+ address-family ipv4
+ !
+ address-family ipv6
+ discovery transport-address 2:2:2::2
+ !
+ interface GigabitEthernet0/0/0/0
+ address-family ipv4
+ !
+ !
+ interface GigabitEthernet0/0/0/1
+ address-family ipv4
+ !
+ address-family ipv6
+ !
+ !
+ interface GigabitEthernet0/0/0/2
+ address-family ipv4
+ !
+ address-family ipv6
+ !
+ !
+!
+```
+
+## Verification - Control Plane
+
+Using the CLI on the Linux box, the goal is to ensure that everything
+is working as expected.
+
+First, verify that all the required adjacencies and neighborships sessions
+were established:
+
+```
+linux# show mpls ldp discovery
+Local LDP Identifier: 4.4.4.4:0
+Discovery Sources:
+ Interfaces:
+ eth1: xmit/recv
+ LDP Id: 3.3.3.3:0, Transport address: 3.3.3.3
+ Hold time: 15 sec
+ LDP Id: 3.3.3.3:0, Transport address: 3:3:3::3
+ Hold time: 15 sec
+ eth2: xmit/recv
+ LDP Id: 2.2.2.2:0, Transport address: 2.2.2.2
+ Hold time: 15 sec
+ LDP Id: 2.2.2.2:0, Transport address: 2:2:2::2
+ Hold time: 15 sec
+ Targeted Hellos:
+ 4.4.4.4 -> 1.1.1.1: xmit/recv
+ LDP Id: 1.1.1.1:0, Transport address: 1.1.1.1
+ Hold time: 45 sec
+ 4:4:4::4 -> 3:3:3::3: xmit/recv
+ LDP Id: 3.3.3.3:0, Transport address: 3:3:3::3
+ Hold time: 45 sec
+
+linux# show mpls ldp neighbor
+Peer LDP Identifier: 1.1.1.1:0
+ TCP connection: 4.4.4.4:40921 - 1.1.1.1:646
+ Session Holdtime: 180 sec
+ State: OPERATIONAL; Downstream-Unsolicited
+ Up time: 00:06:02
+ LDP Discovery Sources:
+ IPv4:
+ Targeted Hello: 1.1.1.1
+
+Peer LDP Identifier: 2.2.2.2:0
+ TCP connection: 4:4:4::4:52286 - 2:2:2::2:646
+ Session Holdtime: 180 sec
+ State: OPERATIONAL; Downstream-Unsolicited
+ Up time: 00:06:02
+ LDP Discovery Sources:
+ IPv4:
+ Interface: eth2
+ IPv6:
+ Interface: eth2
+
+Peer LDP Identifier: 3.3.3.3:0
+ TCP connection: 4:4:4::4:60575 - 3:3:3::3:646
+ Session Holdtime: 180 sec
+ State: OPERATIONAL; Downstream-Unsolicited
+ Up time: 00:05:57
+ LDP Discovery Sources:
+ IPv4:
+ Interface: eth1
+ IPv6:
+ Targeted Hello: 3:3:3::3
+ Interface: eth1
+```
+
+Note that the neighborships with the P and PE2 routers were established
+over IPv6, since this is the default behavior for dual-stack LSRs, as
+specified in RFC 7552. If desired, the **dual-stack transport-connection
+prefer ipv4** command can be used to establish these sessions over IPv4
+(the command should be applied an all routers).
+
+Now, verify that there's a remote label for each PE address:
+```
+linux# show mpls ldp binding
+1.1.1.1/32
+ Local binding: label: 20
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 1.1.1.1 imp-null
+ 2.2.2.2 24000
+ 3.3.3.3 20
+2.2.2.2/32
+ Local binding: label: 21
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 1.1.1.1 18
+ 2.2.2.2 imp-null
+ 3.3.3.3 21
+3.3.3.3/32
+ Local binding: label: 22
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 1.1.1.1 21
+ 2.2.2.2 24003
+ 3.3.3.3 imp-null
+4.4.4.4/32
+ Local binding: label: imp-null
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 1.1.1.1 22
+ 2.2.2.2 24001
+ 3.3.3.3 22
+10.0.1.0/24
+ Local binding: label: 23
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 1.1.1.1 imp-null
+ 2.2.2.2 imp-null
+ 3.3.3.3 23
+10.0.2.0/24
+ Local binding: label: 24
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 1.1.1.1 20
+ 2.2.2.2 imp-null
+ 3.3.3.3 imp-null
+10.0.3.0/24
+ Local binding: label: imp-null
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 1.1.1.1 19
+ 2.2.2.2 imp-null
+ 3.3.3.3 24
+10.0.4.0/24
+ Local binding: label: imp-null
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 1.1.1.1 23
+ 2.2.2.2 24002
+ 3.3.3.3 imp-null
+2:2:2::2/128
+ Local binding: label: 18
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 2.2.2.2 imp-null
+ 3.3.3.3 18
+3:3:3::3/128
+ Local binding: label: 19
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 2.2.2.2 24007
+4:4:4::4/128
+ Local binding: label: imp-null
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 2.2.2.2 24006
+ 3.3.3.3 19
+2001:db8:2::/64
+ Local binding: label: -
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 2.2.2.2 imp-null
+ 3.3.3.3 imp-null
+2001:db8:3::/64
+ Local binding: label: imp-null
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 2.2.2.2 imp-null
+2001:db8:4::/64
+ Local binding: label: imp-null
+ Remote bindings:
+ Peer Label
+ ----------------- ---------
+ 3.3.3.3 imp-null
+```
+
+Check if the pseudowires are up:
+```
+linux# show l2vpn atom vc
+Interface Peer ID VC ID Name Status
+--------- --------------- ---------- ---------------- ----------
+mpw1 3.3.3.3 100 ENG UP
+mpw0 1.1.1.1 100 ENG UP
+```
+
+Check the label bindings of the pseudowires:
+```
+linux# show l2vpn atom binding
+ Destination Address: 1.1.1.1, VC ID: 100
+ Local Label: 25
+ Cbit: 1, VC Type: Ethernet, GroupID: 0
+ MTU: 1500
+ Remote Label: 16
+ Cbit: 1, VC Type: Ethernet, GroupID: 0
+ MTU: 1500
+ Destination Address: 3.3.3.3, VC ID: 100
+ Local Label: 26
+ Cbit: 1, VC Type: Ethernet, GroupID: 0
+ MTU: 1500
+ Remote Label: 26
+ Cbit: 1, VC Type: Ethernet, GroupID: 0
+ MTU: 1500
+```
+
+## Verification - Data Plane
+
+Verify that all the exchanged label mappings were installed in zebra:
+```
+linux# show mpls table
+ Inbound Outbound
+ Label Type Nexthop Label
+-------- ------- --------------- --------
+ 17 LDP 2001:db8:3::2 3
+ 19 LDP 2001:db8:3::2 24005
+ 20 LDP 10.0.3.2 24000
+ 21 LDP 10.0.3.2 3
+ 22 LDP 10.0.3.2 24001
+ 23 LDP 10.0.3.2 3
+ 24 LDP 10.0.3.2 3
+ 25 LDP 10.0.3.2 3
+
+linux# show ip route ldp
+Codes: K - kernel route, C - connected, S - static, R - RIP,
+ O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel, L - LDP,
+ > - selected route, * - FIB route
+
+L>* 1.1.1.1/32 [0/0] via 10.0.3.2, eth2 label 24000
+L>* 3.3.3.3/32 [0/0] via 10.0.3.2, eth2 label 24001
+```
+
+Verify that all the exchanged label mappings were installed in the kernel:
+```
+$ ip -M ro
+17 via inet6 2001:db8:3::2 dev eth2 proto zebra
+19 as to 24005 via inet6 2001:db8:3::2 dev eth2 proto zebra
+20 as to 24000 via inet 10.0.3.2 dev eth2 proto zebra
+21 via inet 10.0.3.2 dev eth2 proto zebra
+22 as to 24001 via inet 10.0.3.2 dev eth2 proto zebra
+23 via inet 10.0.3.2 dev eth2 proto zebra
+24 via inet 10.0.3.2 dev eth2 proto zebra
+25 via inet 10.0.3.2 dev eth2 proto zebra
+$
+$ ip route | grep mpls
+1.1.1.1 encap mpls 24000 via 10.0.3.2 dev eth2 proto zebra metric 20
+3.3.3.3 encap mpls 24001 via 10.0.3.2 dev eth2 proto zebra metric 20
+```
+
+Now ping PE1's loopback using lo1's address as a source address:
+```
+$ ping -c 5 -I 4.4.4.4 1.1.1.1
+PING 1.1.1.1 (1.1.1.1) from 4.4.4.4 : 56(84) bytes of data.
+64 bytes from 1.1.1.1: icmp_seq=1 ttl=253 time=3.02 ms
+64 bytes from 1.1.1.1: icmp_seq=2 ttl=253 time=3.13 ms
+64 bytes from 1.1.1.1: icmp_seq=3 ttl=253 time=3.19 ms
+64 bytes from 1.1.1.1: icmp_seq=4 ttl=253 time=3.07 ms
+64 bytes from 1.1.1.1: icmp_seq=5 ttl=253 time=3.27 ms
+
+--- 1.1.1.1 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4005ms
+rtt min/avg/max/mdev = 3.022/3.140/3.278/0.096 ms
+```
+
+Verify that the ICMP echo request packets are leaving with the MPLS
+label advertised by the P router. Also, verify that the ICMP echo reply
+packets are arriving with an explicit-null MPLS label:
+```
+# tcpdump -n -i eth2 mpls and icmp
+tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
+listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes
+10:01:40.758771 MPLS (label 24000, exp 0, [S], ttl 64) IP 4.4.4.4 > 1.1.1.1: ICMP echo request, id 13370, seq 1, length 64
+10:01:40.761777 MPLS (label 0, exp 0, [S], ttl 254) IP 1.1.1.1 > 4.4.4.4: ICMP echo reply, id 13370, seq 1, length 64
+10:01:41.760343 MPLS (label 24000, exp 0, [S], ttl 64) IP 4.4.4.4 > 1.1.1.1: ICMP echo request, id 13370, seq 2, length 64
+10:01:41.763448 MPLS (label 0, exp 0, [S], ttl 254) IP 1.1.1.1 > 4.4.4.4: ICMP echo reply, id 13370, seq 2, length 64
+10:01:42.761758 MPLS (label 24000, exp 0, [S], ttl 64) IP 4.4.4.4 > 1.1.1.1: ICMP echo request, id 13370, seq 3, length 64
+10:01:42.764924 MPLS (label 0, exp 0, [S], ttl 254) IP 1.1.1.1 > 4.4.4.4: ICMP echo reply, id 13370, seq 3, length 64
+10:01:43.763193 MPLS (label 24000, exp 0, [S], ttl 64) IP 4.4.4.4 > 1.1.1.1: ICMP echo request, id 13370, seq 4, length 64
+10:01:43.766237 MPLS (label 0, exp 0, [S], ttl 254) IP 1.1.1.1 > 4.4.4.4: ICMP echo reply, id 13370, seq 4, length 64
+10:01:44.764552 MPLS (label 24000, exp 0, [S], ttl 64) IP 4.4.4.4 > 1.1.1.1: ICMP echo request, id 13370, seq 5, length 64
+10:01:44.767803 MPLS (label 0, exp 0, [S], ttl 254) IP 1.1.1.1 > 4.4.4.4: ICMP echo reply, id 13370, seq 5, length 64
+```