From 8cc2aca441fbd4153b605a4de94276fe4893a1dd Mon Sep 17 00:00:00 2001
From: Wesley Coakley <wcoakley@nvidia.com>
Date: Tue, 2 Feb 2021 18:16:51 -0500
Subject: [PATCH] doc: cross compilation guide

Wrote a little guide for cross-compiling FRR, gleaned from notes I took
while compiling for a RPi 3B+ on a Gentoo x86_64 system.

Care was taken to keep this documentation as generic as possible so
these steps could be applied to any cross-compile targeting a supported
architecture.

Signed-off-by: Wesley Coakley <wcoakley@nvidia.com>
---
 doc/developer/building.rst        |   1 +
 doc/developer/cross-compiling.rst | 326 ++++++++++++++++++++++++++++++
 doc/developer/subdir.am           |   1 +
 3 files changed, 328 insertions(+)
 create mode 100644 doc/developer/cross-compiling.rst

diff --git a/doc/developer/building.rst b/doc/developer/building.rst
index a6cd545872..c687ba8dc8 100644
--- a/doc/developer/building.rst
+++ b/doc/developer/building.rst
@@ -29,3 +29,4 @@ Building FRR
    building-frr-for-ubuntu2004
    building-frr-for-archlinux
    building-docker
+   cross-compiling
diff --git a/doc/developer/cross-compiling.rst b/doc/developer/cross-compiling.rst
new file mode 100644
index 0000000000..339e00c921
--- /dev/null
+++ b/doc/developer/cross-compiling.rst
@@ -0,0 +1,326 @@
+Cross-Compiling
+===============
+
+FRR is capable of being cross-compiled to a number of different architectures.
+With an adequate toolchain this process is fairly straightforward, though one
+must exercise caution to validate this toolchain's correctness before attempting
+to compile FRR or its dependencies; small oversights in the construction of the
+build tools may lead to problems which quickly become difficult to diagnose.
+
+Toolchain Preliminary
+---------------------
+
+The first step to cross-compiling any program is to identify the system which
+the program (FRR) will run on. From here on this will be called the "host"
+machine, following autotools' convention, while the machine building FRR will be
+called the "build" machine. The toolchain will of course be installed onto the
+build machine and be leveraged to build FRR for the host machine to run.
+
+.. note::
+
+   The build machine used while writing this guide was ``x86_64-pc-linux-gnu``
+   and the target machine was ``arm-linux-gnueabihf`` (a Raspberry Pi 3B+).
+   Replace this with your targeted tuple below if you plan on running the
+   commands from this guide:
+
+      .. code-block:: shell
+
+      export HOST_ARCH="arm-linux-gnueabihf"
+
+   For your given target, the build system's OS may have some support for
+   building cross compilers natively, or may even offer binary toolchains built
+   upstream for the target architecture. Check your package manager or OS
+   documentation before committing to building a toolchain from scratch.
+
+This guide will not detail *how* to build a cross-compiling toolchain but
+will instead assume one already exists and is installed on the build system.
+The methods for building the toolchain itself may differ between operating
+systems so consult the OS documentation for any particulars regarding
+cross-compilers. The OSDev wiki has a `pleasant tutorial`_ on cross-compiling in
+the context of operating system development which bootstraps from only the
+native GCC and binutils on the build machine. This may be useful if the build
+machine's OS does not offer existing tools to build a cross-compiler targeting
+the host.
+
+.. _pleasant tutorial: https://wiki.osdev.org/GCC_Cross-Compiler
+
+This guide will also not demonstrate how to build all of FRR's dependencies for the
+target architecture. Instead, general instructions for using a cross-compiling
+toolchain to compile packages using CMake, Autotools, and Makefiles are
+provided; these three cases apply to almost all FRR dependencies.
+
+.. _glibc mismatch:
+
+.. warning::
+
+   Ensure the versions and implementations of the C standard library (glibc or
+   what have you) match on the host and the build toolchain. ``ldd --version``
+   will help you here. Upgrade one or the other if the they do not match.
+
+Testing the Toolchain
+---------------------
+
+Before any cross-compilation begins it would be prudent to test the new
+toolchain by writing, compiling and linking a simple program.
+
+.. code-block:: shell
+
+   # A small program
+   cat > nothing.c <<EOF
+   int main() { return 0; }
+   EOF
+
+   # Build and link with the cross-compiler
+   ${HOST_ARCH}-gcc -o nothing nothing.c
+
+   # Inspect the resulting binary, results may vary
+   file ./nothing
+
+   # nothing: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV),
+   # dynamically linked, interpreter /lib/ld-linux-armhf.so.3,
+   # for GNU/Linux 3.2.0, not stripped
+
+If this produced no errors then the installed toolchain is probably ready to
+start compiling the build dependencies and eventually FRR itself. There still
+may be lurking issues but fundamentally the toolchain can produce binaries and
+that's good enough to start working with it.
+
+.. warning::
+
+   If any errors occurred during the previous functional test please look back
+   and address them before moving on; this indicates your cross-compiling
+   toolchain is *not* in a position to build FRR or its dependencies. Even if
+   everything was fine, keep in mind that many errors from here on *may still
+   be related* to your toolchain (e.g. libstdc++.so or other components) and this
+   small test is not a guarantee of complete toolchain coherence.
+
+Cross-compiling Dependencies
+----------------------------
+
+When compiling FRR it is necessary to compile some of its dependencies alongside
+it on the build machine. This is so symbols from the shared libraries (which
+will be loaded at run-time on the host machine) can be linked to the FRR
+binaries at compile time; additionally, headers for these libraries are needed
+during the compile stage for a successful build.
+
+Sysroot Overview
+^^^^^^^^^^^^^^^^
+
+All build dependencies should be installed into a "root" directory on the build
+computer, hereafter called the "sysroot". This directory will be prefixed to
+paths while searching for requisite libraries and headers during the build
+process. Often this may be set via a ``--prefix`` flag when building the
+dependent packages, meaning a ``make install`` will copy compiled libraries into
+(e.g.) ``/usr/${HOST_ARCH}/usr``.
+
+If the toolchain was built on the build machine then there is likely already a
+sysroot where those tools and standard libraries were installed; it may be
+helpful to use that directory as the sysroot for this build as well.
+
+Basic Workflow
+^^^^^^^^^^^^^^
+
+Before compiling or building any dependencies, make note of which daemons are
+being targeted and which libraries will be needed. Not all dependencies are
+necessary if only building with a subset of the daemons.
+
+The following workflow will compile and install any libraries which can be built
+with Autotools. The resultant library will be installed into the sysroot
+``/usr/${HOST_ARCH}``.
+
+.. code-block:: shell
+
+   ./configure \
+      CC=${HOST_ARCH}-gcc \
+      CXX=${HOST_ARCH}-g++ \
+      --build=${HOST_ARCH} \
+      --prefix=/usr/${HOST_ARCH}
+   make
+   make install
+
+Some libraries like ``json-c`` and ``libyang`` are packaged with CMake and can
+be built and installed generally like:
+
+.. code-block:: shell
+
+   mkdir build
+   cd build
+   CC=${HOST_ARCH}-gcc \
+   CXX=${HOST_ARCH}-g++ \
+   cmake \
+       -DCMAKE_INSTALL_PREFIX=/usr/${HOST_ARCH} \
+       ..
+   make
+   make install
+
+For programs with only a Makefile (e.g. ``libcap``) the process may look still a
+little different:
+
+.. code-block:: shell
+
+   CC=${HOST_ARCH}-gcc make
+   make install DESTDIR=/usr/${HOST_ARCH}
+
+These three workflows should handle the bulk of building and installing the
+build-time dependencies for FRR. Verify that the installed files are being
+placed correctly into the sysroot and were actually built using the
+cross-compile toolchain, not by the native toolchain by accident.
+
+Dependency Notes
+^^^^^^^^^^^^^^^^
+
+There are a lot of things that can go wrong during a cross-compilation. Some of
+the more common errors and a few special considerations are collected below for
+reference.
+
+libyang
+"""""""
+
+``-DENABLE_LYD_PRIV=ON`` should be provided during the CMake step.
+
+Ensure also that the version of ``libyang`` being installed corresponds to the
+version required by the targeted FRR version.
+
+gRPC
+""""
+
+This piece is requisite only if the ``--enable-grpc`` flag will be passed
+later on to FRR. One may get burned when compiling gRPC if the ``protoc``
+version on the build machine differs from the version of ``protoc`` being linked
+to during a gRPC build. The error messages from this defect look like:
+
+.. code-block:: terminal
+
+   gens/src/proto/grpc/channelz/channelz.pb.h: In member function ‘void grpc::channelz::v1::ServerRef::set_name(const char*, size_t)’:
+   gens/src/proto/grpc/channelz/channelz.pb.h:9127:64: error: ‘EmptyDefault’ is not a member of ‘google::protobuf::internal::ArenaStringPtr’
+    9127 |   name_.Set(::PROTOBUF_NAMESPACE_ID::internal::ArenaStringPtr::EmptyDefault{}, ::std::string(
+
+This happens because protocol buffer code generation uses ``protoc`` to create
+classes with different getters and setters corresponding to the protobuf data
+defined by the source tree's ``.proto`` files. Clearly the cross-compiled
+``protoc`` cannot be used for this code generation because that binary is built
+for a different CPU.
+
+The solution is to install matching versions of native and cross-compiled
+protocol buffers; this way the native binary will generate code and the
+cross-compiled library will be linked to by gRPC and these versions will not
+disagree.
+
+----
+
+The ``-latomic`` linker flag may also be necessary here if using ``libstdc++``
+since GCC's C++11 implementation makes library calls in certain cases for
+``<atomic>`` so ``-latomic`` cannot be assumed.
+
+Cross-compiling FRR Itself
+--------------------------
+
+With all the necessary libraries cross-compiled and installed into the sysroot,
+the last thing to actually build is FRR itself:
+
+.. code-block:: shell
+
+   # Clone and bootstrap the build
+   git clone 'https://github.com/FRRouting/frr.git'
+   # (e.g.) git checkout stable/7.5
+   ./bootstrap.sh
+
+   # Build clippy using the native toolchain
+   mkdir build-clippy
+   cd build-clippy
+   ../configure --enable-clippy-only
+   make clippy-only
+   cd ..
+
+   # Next, configure FRR and use the clippy we just built
+   ./configure \
+      CC=${HOST_ARCH}-gcc \
+      CXX=${HOST_ARCH}-g++ \
+      --host=${HOST_ARCH} \
+      --with-sysroot=/usr/${HOST_ARCH} \
+      --with-clippy=./build-clippy/lib/clippy \
+      --sysconfdir=/etc/frr \
+      --sbindir="\${prefix}/lib/frr" \
+      --localstatedir=/var/run/frr \
+      --prefix=/usr \
+      --enable-user=frr \
+      --enable-group=frr \
+      --enable-vty-group=frrvty \
+      --disable-doc \
+      --enable-grpc
+
+   # Send it
+   make
+
+Installation to Host Machine
+----------------------------
+
+If no errors were observed during the previous steps it is safe to ``make
+install`` FRR into its own directory.
+
+.. code-block:: shell
+
+   # Install FRR its own "sysroot"
+   make install DESTDIR=/some/path/to/sysroot
+
+After running the above command, FRR binaries, modules and example configuration
+files will be installed into some path on the build machine. The directory
+will have folders like ``/usr`` and ``/etc``; this "root" should now be copied
+to the host and installed on top of the root directory there.
+
+.. code-block:: shell
+
+   # Tar this sysroot (preserving permissions)
+   tar -C /some/path/to/sysroot -cpvf frr-${HOST_ARCH}.tar .
+
+   # Transfer tar file to host machine
+   scp frr-${HOST_ARCH}.tar me@host-machine:
+
+   # Overlay the tarred sysroot on top of the host machine's root
+   ssh me@host-machine <<-EOF
+      # May need to elevate permissions here
+      tar -C / -xpvf frr-${HOST_ARCH}.tar.gz .
+   EOF
+
+Now FRR should be installed just as if ``make install`` had been run on the host
+machine. Create configuration files and assign permissions as needed. Lastly,
+ensure the correct users and groups exist for FRR on the host machine.
+
+Troubleshooting
+---------------
+
+Even when every precaution has been taken some things may still go wrong! This
+section details some common runtime problems.
+
+Mismatched Libraries
+^^^^^^^^^^^^^^^^^^^^
+
+If you see something like this after installing on the host:
+
+.. code-block:: console
+
+   /usr/lib/frr/zebra: error while loading shared libraries: libyang.so.1: cannot open shared object file: No such file or directory
+
+... at least one of FRR's dependencies which was linked to the binary earlier is
+not available on the host OS. Even if it has been installed the host
+repository's version may lag what is needed for more recent FRR builds (this is
+likely to happen with YANG at the moment).
+
+If the matching library is not available from the host OS package manager it may
+be possible to compile them using the same toolchain used to compile FRR. The
+library may have already been built earlier when compiling FRR on the build
+machine, in which case it may be as simple as following the same workflow laid
+out during the `Installation to Host Machine`_.
+
+Mismatched Glibc Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The version and implementation of the C standard library must match on both the
+host and build toolchain. The error corresponding to this misconfiguration will
+look like:
+
+.. code-block:: console
+
+   /usr/lib/frr/zebra: /lib/${HOST_ARCH}/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/libfrr.so.0)
+
+See the earlier warning about preventing a `glibc mismatch`_.
diff --git a/doc/developer/subdir.am b/doc/developer/subdir.am
index 07a25886d0..51e806c785 100644
--- a/doc/developer/subdir.am
+++ b/doc/developer/subdir.am
@@ -27,6 +27,7 @@ dev_RSTFILES = \
 	doc/developer/building.rst \
 	doc/developer/cli.rst \
 	doc/developer/conf.py \
+	doc/developer/cross-compiling.rst \
 	doc/developer/frr-release-procedure.rst \
 	doc/developer/grpc.rst \
 	doc/developer/hooks.rst \
-- 
2.39.5