diff options
Diffstat (limited to 'doc/developer')
| -rw-r--r-- | doc/developer/building.rst | 1 | ||||
| -rw-r--r-- | doc/developer/cross-compiling.rst | 326 | ||||
| -rw-r--r-- | doc/developer/library.rst | 1 | ||||
| -rw-r--r-- | doc/developer/subdir.am | 2 | ||||
| -rw-r--r-- | doc/developer/topotests.rst | 2 | ||||
| -rw-r--r-- | doc/developer/xrefs.rst | 170 |
6 files changed, 502 insertions, 0 deletions
diff --git a/doc/developer/building.rst b/doc/developer/building.rst index a6cd545872..c687ba8dc8 100644 --- a/doc/developer/building.rst +++ b/doc/developer/building.rst @@ -29,3 +29,4 @@ Building FRR building-frr-for-ubuntu2004 building-frr-for-archlinux building-docker + cross-compiling diff --git a/doc/developer/cross-compiling.rst b/doc/developer/cross-compiling.rst new file mode 100644 index 0000000000..339e00c921 --- /dev/null +++ b/doc/developer/cross-compiling.rst @@ -0,0 +1,326 @@ +Cross-Compiling +=============== + +FRR is capable of being cross-compiled to a number of different architectures. +With an adequate toolchain this process is fairly straightforward, though one +must exercise caution to validate this toolchain's correctness before attempting +to compile FRR or its dependencies; small oversights in the construction of the +build tools may lead to problems which quickly become difficult to diagnose. + +Toolchain Preliminary +--------------------- + +The first step to cross-compiling any program is to identify the system which +the program (FRR) will run on. From here on this will be called the "host" +machine, following autotools' convention, while the machine building FRR will be +called the "build" machine. The toolchain will of course be installed onto the +build machine and be leveraged to build FRR for the host machine to run. + +.. note:: + + The build machine used while writing this guide was ``x86_64-pc-linux-gnu`` + and the target machine was ``arm-linux-gnueabihf`` (a Raspberry Pi 3B+). + Replace this with your targeted tuple below if you plan on running the + commands from this guide: + + .. code-block:: shell + + export HOST_ARCH="arm-linux-gnueabihf" + + For your given target, the build system's OS may have some support for + building cross compilers natively, or may even offer binary toolchains built + upstream for the target architecture. Check your package manager or OS + documentation before committing to building a toolchain from scratch. + +This guide will not detail *how* to build a cross-compiling toolchain but +will instead assume one already exists and is installed on the build system. +The methods for building the toolchain itself may differ between operating +systems so consult the OS documentation for any particulars regarding +cross-compilers. The OSDev wiki has a `pleasant tutorial`_ on cross-compiling in +the context of operating system development which bootstraps from only the +native GCC and binutils on the build machine. This may be useful if the build +machine's OS does not offer existing tools to build a cross-compiler targeting +the host. + +.. _pleasant tutorial: https://wiki.osdev.org/GCC_Cross-Compiler + +This guide will also not demonstrate how to build all of FRR's dependencies for the +target architecture. Instead, general instructions for using a cross-compiling +toolchain to compile packages using CMake, Autotools, and Makefiles are +provided; these three cases apply to almost all FRR dependencies. + +.. _glibc mismatch: + +.. warning:: + + Ensure the versions and implementations of the C standard library (glibc or + what have you) match on the host and the build toolchain. ``ldd --version`` + will help you here. Upgrade one or the other if the they do not match. + +Testing the Toolchain +--------------------- + +Before any cross-compilation begins it would be prudent to test the new +toolchain by writing, compiling and linking a simple program. + +.. code-block:: shell + + # A small program + cat > nothing.c <<EOF + int main() { return 0; } + EOF + + # Build and link with the cross-compiler + ${HOST_ARCH}-gcc -o nothing nothing.c + + # Inspect the resulting binary, results may vary + file ./nothing + + # nothing: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), + # dynamically linked, interpreter /lib/ld-linux-armhf.so.3, + # for GNU/Linux 3.2.0, not stripped + +If this produced no errors then the installed toolchain is probably ready to +start compiling the build dependencies and eventually FRR itself. There still +may be lurking issues but fundamentally the toolchain can produce binaries and +that's good enough to start working with it. + +.. warning:: + + If any errors occurred during the previous functional test please look back + and address them before moving on; this indicates your cross-compiling + toolchain is *not* in a position to build FRR or its dependencies. Even if + everything was fine, keep in mind that many errors from here on *may still + be related* to your toolchain (e.g. libstdc++.so or other components) and this + small test is not a guarantee of complete toolchain coherence. + +Cross-compiling Dependencies +---------------------------- + +When compiling FRR it is necessary to compile some of its dependencies alongside +it on the build machine. This is so symbols from the shared libraries (which +will be loaded at run-time on the host machine) can be linked to the FRR +binaries at compile time; additionally, headers for these libraries are needed +during the compile stage for a successful build. + +Sysroot Overview +^^^^^^^^^^^^^^^^ + +All build dependencies should be installed into a "root" directory on the build +computer, hereafter called the "sysroot". This directory will be prefixed to +paths while searching for requisite libraries and headers during the build +process. Often this may be set via a ``--prefix`` flag when building the +dependent packages, meaning a ``make install`` will copy compiled libraries into +(e.g.) ``/usr/${HOST_ARCH}/usr``. + +If the toolchain was built on the build machine then there is likely already a +sysroot where those tools and standard libraries were installed; it may be +helpful to use that directory as the sysroot for this build as well. + +Basic Workflow +^^^^^^^^^^^^^^ + +Before compiling or building any dependencies, make note of which daemons are +being targeted and which libraries will be needed. Not all dependencies are +necessary if only building with a subset of the daemons. + +The following workflow will compile and install any libraries which can be built +with Autotools. The resultant library will be installed into the sysroot +``/usr/${HOST_ARCH}``. + +.. code-block:: shell + + ./configure \ + CC=${HOST_ARCH}-gcc \ + CXX=${HOST_ARCH}-g++ \ + --build=${HOST_ARCH} \ + --prefix=/usr/${HOST_ARCH} + make + make install + +Some libraries like ``json-c`` and ``libyang`` are packaged with CMake and can +be built and installed generally like: + +.. code-block:: shell + + mkdir build + cd build + CC=${HOST_ARCH}-gcc \ + CXX=${HOST_ARCH}-g++ \ + cmake \ + -DCMAKE_INSTALL_PREFIX=/usr/${HOST_ARCH} \ + .. + make + make install + +For programs with only a Makefile (e.g. ``libcap``) the process may look still a +little different: + +.. code-block:: shell + + CC=${HOST_ARCH}-gcc make + make install DESTDIR=/usr/${HOST_ARCH} + +These three workflows should handle the bulk of building and installing the +build-time dependencies for FRR. Verify that the installed files are being +placed correctly into the sysroot and were actually built using the +cross-compile toolchain, not by the native toolchain by accident. + +Dependency Notes +^^^^^^^^^^^^^^^^ + +There are a lot of things that can go wrong during a cross-compilation. Some of +the more common errors and a few special considerations are collected below for +reference. + +libyang +""""""" + +``-DENABLE_LYD_PRIV=ON`` should be provided during the CMake step. + +Ensure also that the version of ``libyang`` being installed corresponds to the +version required by the targeted FRR version. + +gRPC +"""" + +This piece is requisite only if the ``--enable-grpc`` flag will be passed +later on to FRR. One may get burned when compiling gRPC if the ``protoc`` +version on the build machine differs from the version of ``protoc`` being linked +to during a gRPC build. The error messages from this defect look like: + +.. code-block:: terminal + + gens/src/proto/grpc/channelz/channelz.pb.h: In member function ‘void grpc::channelz::v1::ServerRef::set_name(const char*, size_t)’: + gens/src/proto/grpc/channelz/channelz.pb.h:9127:64: error: ‘EmptyDefault’ is not a member of ‘google::protobuf::internal::ArenaStringPtr’ + 9127 | name_.Set(::PROTOBUF_NAMESPACE_ID::internal::ArenaStringPtr::EmptyDefault{}, ::std::string( + +This happens because protocol buffer code generation uses ``protoc`` to create +classes with different getters and setters corresponding to the protobuf data +defined by the source tree's ``.proto`` files. Clearly the cross-compiled +``protoc`` cannot be used for this code generation because that binary is built +for a different CPU. + +The solution is to install matching versions of native and cross-compiled +protocol buffers; this way the native binary will generate code and the +cross-compiled library will be linked to by gRPC and these versions will not +disagree. + +---- + +The ``-latomic`` linker flag may also be necessary here if using ``libstdc++`` +since GCC's C++11 implementation makes library calls in certain cases for +``<atomic>`` so ``-latomic`` cannot be assumed. + +Cross-compiling FRR Itself +-------------------------- + +With all the necessary libraries cross-compiled and installed into the sysroot, +the last thing to actually build is FRR itself: + +.. code-block:: shell + + # Clone and bootstrap the build + git clone 'https://github.com/FRRouting/frr.git' + # (e.g.) git checkout stable/7.5 + ./bootstrap.sh + + # Build clippy using the native toolchain + mkdir build-clippy + cd build-clippy + ../configure --enable-clippy-only + make clippy-only + cd .. + + # Next, configure FRR and use the clippy we just built + ./configure \ + CC=${HOST_ARCH}-gcc \ + CXX=${HOST_ARCH}-g++ \ + --host=${HOST_ARCH} \ + --with-sysroot=/usr/${HOST_ARCH} \ + --with-clippy=./build-clippy/lib/clippy \ + --sysconfdir=/etc/frr \ + --sbindir="\${prefix}/lib/frr" \ + --localstatedir=/var/run/frr \ + --prefix=/usr \ + --enable-user=frr \ + --enable-group=frr \ + --enable-vty-group=frrvty \ + --disable-doc \ + --enable-grpc + + # Send it + make + +Installation to Host Machine +---------------------------- + +If no errors were observed during the previous steps it is safe to ``make +install`` FRR into its own directory. + +.. code-block:: shell + + # Install FRR its own "sysroot" + make install DESTDIR=/some/path/to/sysroot + +After running the above command, FRR binaries, modules and example configuration +files will be installed into some path on the build machine. The directory +will have folders like ``/usr`` and ``/etc``; this "root" should now be copied +to the host and installed on top of the root directory there. + +.. code-block:: shell + + # Tar this sysroot (preserving permissions) + tar -C /some/path/to/sysroot -cpvf frr-${HOST_ARCH}.tar . + + # Transfer tar file to host machine + scp frr-${HOST_ARCH}.tar me@host-machine: + + # Overlay the tarred sysroot on top of the host machine's root + ssh me@host-machine <<-EOF + # May need to elevate permissions here + tar -C / -xpvf frr-${HOST_ARCH}.tar.gz . + EOF + +Now FRR should be installed just as if ``make install`` had been run on the host +machine. Create configuration files and assign permissions as needed. Lastly, +ensure the correct users and groups exist for FRR on the host machine. + +Troubleshooting +--------------- + +Even when every precaution has been taken some things may still go wrong! This +section details some common runtime problems. + +Mismatched Libraries +^^^^^^^^^^^^^^^^^^^^ + +If you see something like this after installing on the host: + +.. code-block:: console + + /usr/lib/frr/zebra: error while loading shared libraries: libyang.so.1: cannot open shared object file: No such file or directory + +... at least one of FRR's dependencies which was linked to the binary earlier is +not available on the host OS. Even if it has been installed the host +repository's version may lag what is needed for more recent FRR builds (this is +likely to happen with YANG at the moment). + +If the matching library is not available from the host OS package manager it may +be possible to compile them using the same toolchain used to compile FRR. The +library may have already been built earlier when compiling FRR on the build +machine, in which case it may be as simple as following the same workflow laid +out during the `Installation to Host Machine`_. + +Mismatched Glibc Versions +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The version and implementation of the C standard library must match on both the +host and build toolchain. The error corresponding to this misconfiguration will +look like: + +.. code-block:: console + + /usr/lib/frr/zebra: /lib/${HOST_ARCH}/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/libfrr.so.0) + +See the earlier warning about preventing a `glibc mismatch`_. diff --git a/doc/developer/library.rst b/doc/developer/library.rst index 1bfe5df2f0..2e36c253ea 100644 --- a/doc/developer/library.rst +++ b/doc/developer/library.rst @@ -11,6 +11,7 @@ Library Facilities (libfrr) rcu lists logging + xrefs locking hooks cli diff --git a/doc/developer/subdir.am b/doc/developer/subdir.am index 07a25886d0..f7e4486ef0 100644 --- a/doc/developer/subdir.am +++ b/doc/developer/subdir.am @@ -27,6 +27,7 @@ dev_RSTFILES = \ doc/developer/building.rst \ doc/developer/cli.rst \ doc/developer/conf.py \ + doc/developer/cross-compiling.rst \ doc/developer/frr-release-procedure.rst \ doc/developer/grpc.rst \ doc/developer/hooks.rst \ @@ -58,6 +59,7 @@ dev_RSTFILES = \ doc/developer/topotests-snippets.rst \ doc/developer/topotests.rst \ doc/developer/workflow.rst \ + doc/developer/xrefs.rst \ doc/developer/zebra.rst \ # end diff --git a/doc/developer/topotests.rst b/doc/developer/topotests.rst index 3e8987f126..93d81548b2 100644 --- a/doc/developer/topotests.rst +++ b/doc/developer/topotests.rst @@ -21,8 +21,10 @@ Installing Mininet Infrastructure apt-get install mininet apt-get install python-pip apt-get install iproute + apt-get install iperf pip install ipaddr pip install "pytest<5" + pip install "scapy>=2.4.2" pip install exabgp==3.4.17 (Newer 4.0 version of exabgp is not yet supported) useradd -d /var/run/exabgp/ -s /bin/false exabgp diff --git a/doc/developer/xrefs.rst b/doc/developer/xrefs.rst new file mode 100644 index 0000000000..6a0794d41b --- /dev/null +++ b/doc/developer/xrefs.rst @@ -0,0 +1,170 @@ +.. _xrefs: + +Introspection (xrefs) +===================== + +The FRR library provides an introspection facility called "xrefs." The intent +is to provide structured access to annotated entities in the compiled binary, +such as log messages and thread scheduling calls. + +Enabling and use +---------------- + +Support for emitting an xref is included in the macros for the specific +entities, e.g. :c:func:`zlog_info` contains the relevant statements. The only +requirement for the system to work is a GNU compatible linker that supports +section start/end symbols. (The only known linker on any system FRR supports +that does not do this is the Solaris linker.) + +To verify xrefs have been included in a binary or dynamic library, run +``readelf -n binary``. For individual object files, it's +``readelf -S object.o | grep xref_array`` instead. + +An extraction tool will be added in a future commit. + +Structure and contents +---------------------- + +As a slight improvement to security and fault detection, xrefs are divided into +a ``const struct xref *`` and an optional ``struct xrefdata *``. The required +const part contains: + +.. c:member:: enum xref_type xref.type + + Identifies what kind of object the xref points to. + +.. c:member:: int line +.. c:member:: const char *xref.file +.. c:member:: const char *xref.func + + Source code location of the xref. ``func`` will be ``<global>`` for + xrefs outside of a function. + +.. c:member:: struct xrefdata *xref.xrefdata + + The optional writable part of the xref. NULL if no non-const part exists. + +The optional non-const part has: + +.. c:member:: const struct xref *xrefdata.xref + + Pointer back to the constant part. Since circular pointers are close to + impossible to emit from inside a function body's static variables, this + is initialized at startup. + +.. c:member:: char xrefdata.uid[16] + + Unique identifier, see below. + +.. c:member:: const char *xrefdata.hashstr +.. c:member:: uint32_t xrefdata.hashu32[2] + + Input to unique identifier calculation. These should encompass all + details needed to make an xref unique. If more than one string should + be considered, use string concatenation for the initializer. + +Both structures can be extended by embedding them in a larger type-specific +struct, e.g. ``struct xref_logmsg *``. + +Unique identifiers +------------------ + +All xrefs that have a writable ``struct xrefdata *`` part are assigned an +unique identifier, which is formed as base32 (crockford) SHA256 on: + +- the source filename +- the ``hashstr`` field +- the ``hashu32`` fields + +.. note:: + + Function names and line numbers are intentionally not included to allow + moving items within a file without affecting the identifier. + +For running executables, this hash is calculated once at startup. When +directly reading from an ELF file with external tooling, the value must be +calculated when necessary. + +The identifiers have the form ``AXXXX-XXXXX`` where ``X`` is +``0-9, A-Z except I,L,O,U`` and ``A`` is ``G-Z except I,L,O,U`` (i.e. the +identifiers always start with a letter.) When reading identifiers from user +input, ``I`` and ``L`` should be replaced with ``1`` and ``O`` should be +replaced with ``0``. There are 49 bits of entropy in this identifier. + +Underlying machinery +-------------------- + +Xrefs are nothing other than global variables with some extra glue to make +them possible to find from the outside by looking at the binary. The first +non-obvious part is that they can occur inside of functions, since they're +defined as ``static``. They don't have a visible name -- they don't need one. + +To make finding these variables possible, another global variable, a pointer +to the first one, is created in the same way. However, it is put in a special +ELF section through ``__attribute__((section("xref_array")))``. This is the +section you can see with readelf. + +Finally, on the level of a whole executable or library, the linker will stuff +the individual pointers consecutive to each other since they're in the same +section — hence the array. Start and end of this array is given by the +linker-autogenerated ``__start_xref_array`` and ``__stop_xref_array`` symbols. +Using these, both a constructor to run at startup as well as an ELF note are +created. + +The ELF note is the entrypoint for externally retrieving xrefs from a binary +without having to run it. It can be found by walking through the ELF data +structures even if the binary has been fully stripped of debug and section +information. SystemTap's SDT probes & LTTng's trace points work in the same +way (though they emit 1 note for each probe, while xrefs only emit one note +in total which refers to the array.) Using xrefs does not impact SystemTap +or LTTng, the notes have identifiers they can be distinguished by. + +The ELF structure of a linked binary (library or executable) will look like +this:: + + $ readelf --wide -l -n lib/.libs/libfrr.so + + Elf file type is DYN (Shared object file) + Entry point 0x67d21 + There are 12 program headers, starting at offset 64 + + Program Headers: + Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align + PHDR 0x000040 0x0000000000000040 0x0000000000000040 0x0002a0 0x0002a0 R 0x8 + INTERP 0x125560 0x0000000000125560 0x0000000000125560 0x00001c 0x00001c R 0x10 + [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2] + LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x02aff0 0x02aff0 R 0x1000 + LOAD 0x02b000 0x000000000002b000 0x000000000002b000 0x0b2889 0x0b2889 R E 0x1000 + LOAD 0x0de000 0x00000000000de000 0x00000000000de000 0x070048 0x070048 R 0x1000 + LOAD 0x14e428 0x000000000014f428 0x000000000014f428 0x00fb70 0x01a2b8 RW 0x1000 + DYNAMIC 0x157a40 0x0000000000158a40 0x0000000000158a40 0x000270 0x000270 RW 0x8 + NOTE 0x0002e0 0x00000000000002e0 0x00000000000002e0 0x00004c 0x00004c R 0x4 + TLS 0x14e428 0x000000000014f428 0x000000000014f428 0x000000 0x000008 R 0x8 + GNU_EH_FRAME 0x12557c 0x000000000012557c 0x000000000012557c 0x00819c 0x00819c R 0x4 + GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW 0x10 + GNU_RELRO 0x14e428 0x000000000014f428 0x000000000014f428 0x009bd8 0x009bd8 R 0x1 + + (...) + + Displaying notes found in: .note.gnu.build-id + Owner Data size Description + GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: 6a1f66be38b523095ebd6ec13cc15820cede903d + + Displaying notes found in: .note.FRR + Owner Data size Description + FRRouting 0x00000010 Unknown note type: (0x46455258) description data: 6c eb 15 00 00 00 00 00 74 ec 15 00 00 00 00 00 + +Where 0x15eb6c…0x15ec74 are the offsets (relative to the note itself) where +the xref array is in the file. Also note the owner is clearly marked as +"FRRouting" and the type is "XREF" in hex. + +For SystemTap's use of ELF notes, refer to +https://libstapsdt.readthedocs.io/en/latest/how-it-works/internals.html as an +entry point. + +.. note:: + + Due to GCC bug 41091, the "xref_array" section is not correctly generated + for C++ code when compiled by GCC. A workaround is present for runtime + functionality, but to extract the xrefs from a C++ source file, it needs + to be built with clang (or a future fixed version of GCC) instead. |