From 9d7361338862a5102c03a7e8072061c927c6607e Mon Sep 17 00:00:00 2001 From: Rafael Zalamena Date: Thu, 16 Apr 2020 16:26:51 -0300 Subject: [PATCH] doc: update FPM documentation Tell users about new FPM implementation and add more documentation about it. Signed-off-by: Rafael Zalamena --- doc/developer/fpm.rst | 103 ++++++++++++++++++++++++ doc/developer/index.rst | 1 + doc/user/zebra.rst | 168 ++++++++++++++++++++++++++++++++-------- 3 files changed, 238 insertions(+), 34 deletions(-) create mode 100644 doc/developer/fpm.rst diff --git a/doc/developer/fpm.rst b/doc/developer/fpm.rst new file mode 100644 index 0000000000..9849869133 --- /dev/null +++ b/doc/developer/fpm.rst @@ -0,0 +1,103 @@ +FPM +=== + +FPM stands for Forwarding Plane Manager and it's a module for use with Zebra. + +The encapsulation header for the messages exchanged with the FPM is +defined by the file :file:`fpm/fpm.h` in the frr tree. The routes +themselves are encoded in Netlink or protobuf format, with Netlink +being the default. + +Netlink is standard format for encoding messages to talk with kernel space +in Linux and it is also the name of the socket type used by it. +The FPM netlink usage differs from Linux's in: + +- Linux netlink sockets use datagrams in a multicast fashion, FPM uses + as a stream and it is unicast. +- FPM netlink messages might have more or less information than a normal + Linux netlink socket message (example: RTM_NEWROUTE might add an extra + route attribute to signalize VxLAN encapsulation). + +Protobuf is one of a number of new serialization formats wherein the +message schema is expressed in a purpose-built language. Code for +encoding/decoding to/from the wire format is generated from the +schema. Protobuf messages can be extended easily while maintaining +backward-compatibility with older code. Protobuf has the following +advantages over Netlink: + +- Code for serialization/deserialization is generated automatically. This + reduces the likelihood of bugs, allows third-party programs to be integrated + quickly, and makes it easy to add fields. +- The message format is not tied to an OS (Linux), and can be evolved + independently. + +.. note:: + + Currently there are two FPM modules in ``zebra``: + + * ``fpm`` + * ``dplane_fpm_nl`` + +fpm +^^^ + +The first FPM implementation that was built using hooks in ``zebra`` route +handling functions. It uses its own netlink/protobuf encoding functions to +translate ``zebra`` route data structures into formatted binary data. + + +dplane_fpm_nl +^^^^^^^^^^^^^ + +The newer FPM implementation that was built using ``zebra``'s data plane +framework as a plugin. It only supports netlink and it shares ``zebra``'s +netlink functions to translate route event snapshots into formatted binary +data. + + +Protocol Specification +---------------------- + +FPM (in any mode) uses a TCP connection to talk with external applications. +It operates as TCP client and uses the CLI configured address/port to connect +to the FPM server (defaults to port ``2620``). + +FPM frames all data with a header to help the external reader figure how +many bytes it has to read in order to read the full message (this helps +simulates datagrams like in the original netlink Linux kernel usage). + +Frame header: + +:: + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +---------------+---------------+-------------------------------+ + | Version | Message type | Message length | + +---------------+---------------+-------------------------------+ + | Data... | + +---------------------------------------------------------------+ + + +Version +^^^^^^^ + +Currently there is only one version, so it should be always ``1``. + + +Message Type +^^^^^^^^^^^^ + +Defines what underlining protocol we are using: netlink (``1``) or protobuf (``2``). + + +Message Length +^^^^^^^^^^^^^^ + +Amount of data in this frame in network byte order. + + +Data +^^^^ + +The netlink or protobuf message payload. diff --git a/doc/developer/index.rst b/doc/developer/index.rst index 3a33d9a5ec..26b590c876 100644 --- a/doc/developer/index.rst +++ b/doc/developer/index.rst @@ -11,6 +11,7 @@ FRRouting Developer's Guide library testing bgpd + fpm ospf zebra vtysh diff --git a/doc/user/zebra.rst b/doc/user/zebra.rst index 520080e83a..3629b47877 100644 --- a/doc/user/zebra.rst +++ b/doc/user/zebra.rst @@ -736,43 +736,30 @@ these cases, the FIB needs to be maintained reliably in the fast path as well. We refer to the component that programs the forwarding plane (directly or indirectly) as the Forwarding Plane Manager or FPM. -The FIB push interface comprises of a TCP connection between zebra and -the FPM. The connection is initiated by zebra -- that is, the FPM acts -as the TCP server. - .. program:: configure The relevant zebra code kicks in when zebra is configured with the -:option:`--enable-fpm` flag. Zebra periodically attempts to connect to -the well-known FPM port. Once the connection is up, zebra starts -sending messages containing routes over the socket to the FPM. Zebra -sends a complete copy of the forwarding table to the FPM, including -routes that it may have picked up from the kernel. The existing -interaction of zebra with the kernel remains unchanged -- that is, the -kernel continues to receive FIB updates as before. - -The encapsulation header for the messages exchanged with the FPM is -defined by the file :file:`fpm/fpm.h` in the frr tree. The routes -themselves are encoded in Netlink or protobuf format, with Netlink -being the default. - -Protobuf is one of a number of new serialization formats wherein the -message schema is expressed in a purpose-built language. Code for -encoding/decoding to/from the wire format is generated from the -schema. Protobuf messages can be extended easily while maintaining -backward-compatibility with older code. Protobuf has the following -advantages over Netlink: - -- Code for serialization/deserialization is generated automatically. This - reduces the likelihood of bugs, allows third-party programs to be integrated - quickly, and makes it easy to add fields. -- The message format is not tied to an OS (Linux), and can be evolved - independently. - -As mentioned before, zebra encodes routes sent to the FPM in Netlink -format by default. The format can be controlled via the FPM module's -load-time option to zebra, which currently takes the values `Netlink` -and `protobuf`. +:option:`--enable-fpm` flag and started with the module (``-M fpm`` +or ``-M dplane_fpm_nl``). + +.. note:: + + The ``fpm`` implementation attempts to connect to ``127.0.0.1`` port ``2620`` + by default without configurations. The ``dplane_fpm_nl`` only attempts to + connect to a server if configured. + +Zebra periodically attempts to connect to the well-known FPM port (``2620``). +Once the connection is up, zebra starts sending messages containing routes +over the socket to the FPM. Zebra sends a complete copy of the forwarding +table to the FPM, including routes that it may have picked up from the kernel. +The existing interaction of zebra with the kernel remains unchanged -- that +is, the kernel continues to receive FIB updates as before. + +The default FPM message format is netlink, however it can be controlled +with the module load-time option. The modules accept the following options: + +- ``fpm``: ``netlink`` and ``protobuf``. +- ``dplane_fpm_nl``: none, it only implements netlink. The zebra FPM interface uses replace semantics. That is, if a 'route add' message for a prefix is followed by another 'route add' message, @@ -782,6 +769,119 @@ replaces the information sent in the first message. If the connection to the FPM goes down for some reason, zebra sends the FPM a complete copy of the forwarding table(s) when it reconnects. +For more details on the implementation, please read the developer's manual FPM +section. + +FPM Commands +============ + +``fpm`` implementation +---------------------- + +.. index:: fpm connection ip A.B.C.D port (1-65535) +.. clicmd:: fpm connection ip A.B.C.D port (1-65535) + + Configure ``zebra`` to connect to a different FPM server than + ``127.0.0.1`` port ``2620``. + + +.. index:: no fpm connection ip A.B.C.D port (1-65535) +.. clicmd:: no fpm connection ip A.B.C.D port (1-65535) + + Configure ``zebra`` to connect to the default FPM server at ``127.0.0.1`` + port ``2620``. + + +.. index:: show zebra fpm stats +.. clicmd:: show zebra fpm stats + + Shows the FPM statistics. + + Sample output: + + :: + + Counter Total Last 10 secs + + connect_calls 3 2 + connect_no_sock 0 0 + read_cb_calls 2 2 + write_cb_calls 2 0 + write_calls 1 0 + partial_writes 0 0 + max_writes_hit 0 0 + t_write_yields 0 0 + nop_deletes_skipped 6 0 + route_adds 5 0 + route_dels 0 0 + updates_triggered 11 0 + redundant_triggers 0 0 + dests_del_after_update 0 0 + t_conn_down_starts 0 0 + t_conn_down_dests_processed 0 0 + t_conn_down_yields 0 0 + t_conn_down_finishes 0 0 + t_conn_up_starts 1 0 + t_conn_up_dests_processed 11 0 + t_conn_up_yields 0 0 + t_conn_up_aborts 0 0 + t_conn_up_finishes 1 0 + + +.. index:: clear zebra fpm stats +.. clicmd:: clear zebra fpm stats + + Resets all FPM counters. + + +``dplane_fpm_nl`` implementation +-------------------------------- + +.. index:: fpm address [port (1-65535)] +.. clicmd:: fpm address [port (1-65535)] + + Configures the FPM server address. Once configured ``zebra`` will attempt + to connect to it immediately. + + +.. index:: no fpm address [ [port (1-65535)]] +.. clicmd:: no fpm address [ [port (1-65535)]] + + Disables FPM entirely. ``zebra`` will close any current connections and + will not attempt to connect to it anymore. + + +.. index:: show fpm counters [json] +.. clicmd:: show fpm counters [json] + + Show the FPM statistics (plain text or JSON formatted). + + Sample output: + + :: + + FPM counters + ============ + Input bytes: 0 + Output bytes: 308 + Output buffer current size: 0 + Output buffer peak size: 308 + Connection closes: 0 + Connection errors: 0 + Data plane items processed: 0 + Data plane items enqueued: 0 + Data plane items queue peak: 0 + Buffer full hits: 0 + User FPM configurations: 1 + User FPM disable requests: 0 + + +.. index:: clear fpm counters +.. clicmd:: clear fpm counters + + Resets all FPM counters. + + .. _zebra-dplane: Dataplane Commands -- 2.39.5