unit tests: support code coverage instrumentation and reports

Currently, make check runs the unit tests and reports pass/fail,
but we have no way to guage how much of the code is covered by
these tests.  gcov provides those statistics on a per source
file basis, but requires special CFLAGS and LDFLAGS.  Here, we
add the --enable-gcov configure option to setup those options
correctly.  We also add a make target called check-coverage,
which runs the unit tests, runs gcov and uploads the data to
the codecov.io cloud service for display.

Finally, we include a Dockerfile-coverage which creates a
container image in alpine linux to run the tests.  To create
the image:

$ docker build \
	--build-arg commit=`git rev-parse HEAD` \
	--build-arg token=<upload token from codecov.io> \
	-t frr-gcov:latest \
	-f docker/alpine/Dockerfile-coverage .

and to create and upload the report:

$ docker run -it --rm frr-gcov:latest

Testing done:

Created and uploaded a report from my fork using alpine linux 3.7.
Non-coverage alpine 3.7 build still works.

Issue: https://github.com/FRRouting/frr/issues/2442
Signed-off-by: Arthur Jones <arthur.jones@riverbed.com>

diff --git a/Makefile.am b/Makefile.am
index 5dc80b4983d57969e9f1f4d1042ed9dc1c406b25..ed22c60e7c748704a0a9bd4e4a4e30a2925e41d2 100644 (file)
--- a/Makefile.am
+++ b/Makefile.am
@@ -115,3 +115,23 @@ noinst_HEADERS += defaults.h
 
 indent:
        tools/indent.py `find sharpd bgpd eigrpd include isisd lib nhrpd ospf6d ospfd pimd qpb ripd vtysh zebra -name '*.[ch]' | grep -v include/linux`
+
+if HAVE_GCOV
+
+coverage: check
+       @ find . -name '*.o' -exec gcov {} \;
+
+yorn:
+       @ echo "OK to upload coverage to https://coverage.io [y/N]:"
+       @ read yn; test "$$yn" = "y"
+
+upload-check-coverage:
+       @ if [ "x${COMMIT}" = "x" ]; then echo "COMMIT required"; exit 1; fi
+       @ if [ "x${TOKEN}" = "x" ]; then echo "TOKEN required"; exit 1; fi
+       curl -s https://codecov.io/bash | bash -s - -C ${COMMIT} -t ${TOKEN}
+
+force-check-coverage: coverage upload-check-coverage
+
+check-coverage: coverage yorn upload-check-coverage
+
+endif

diff --git a/configure.ac b/configure.ac
index 3ec29cc38fd8637e8b30d33a7a01c673fd60602f..8846fcdf717c3d031e5ca2c4c826f6529faee1a5 100755 (executable)
--- a/configure.ac
+++ b/configure.ac
@@ -226,7 +226,14 @@ AC_ARG_ENABLE([memory-sanitizer], AS_HELP_STRING([--enable-memory-sanitizer], \
               ])
 
 dnl if the user has specified any CFLAGS, override our settings
-if test "x${enable_dev_build}" = "xyes"; then
+if test "x${enable_gcov}" = "xyes"; then
+   if test "z$orig_cflags" = "z"; then
+      AC_C_FLAG([-coverage])
+      AC_C_FLAG([-O0])
+   fi
+
+   LDFLAGS="${LDFLAGS} -lgcov"
+elif test "x${enable_dev_build}" = "xyes"; then
    AC_DEFINE(DEV_BUILD,,Build for development)
    if test "z$orig_cflags" = "z"; then
       AC_C_FLAG([-g3])
@@ -441,6 +448,8 @@ AC_ARG_ENABLE([clippy-only],
   AS_HELP_STRING([--enable-clippy-only], [Only build clippy]))
 AC_ARG_ENABLE([numeric_version],
   AS_HELP_STRING([--enable-numeric-version], [Only numeric digits allowed in version (for Alpine)]))
+AC_ARG_ENABLE([gcov],
+  AS_HELP_STRING([--enable-gcov], [Add code coverage information]))
 
 AS_IF([test "${enable_clippy_only}" != "yes"], [
 AC_CHECK_HEADERS(json-c/json.h)
@@ -692,6 +701,11 @@ AC_DEFINE_UNQUOTED(MULTIPATH_NUM, $MPATH_NUM, Maximum number of paths for a rout
 
 AC_DEFINE_UNQUOTED(VTYSH_PAGER, "$VTYSH_PAGER", [What pager to use])
 
+dnl --------------------
+dnl Enable code coverage
+dnl --------------------
+AM_CONDITIONAL([HAVE_GCOV],[test '!' "$enable_gcov" = no])
+
 dnl ------------------------------------
 dnl Alpine only accepts numeric versions
 dnl ------------------------------------

diff --git a/doc/user/installation.rst b/doc/user/installation.rst
index 26d30f1e10e71ca00ab934a16243a647b3a71412..158e2c8595243418b726194b38d873d8a682e8ad 100644 (file)
--- a/doc/user/installation.rst
+++ b/doc/user/installation.rst
@@ -205,6 +205,15 @@ options from the list below.
    hardcoded arrays that FRR builds towards, so we need to know how big to
    make these arrays at build time.
 
+.. option:: --enable-gcov
+
+   Code coverage reports from gcov require adjustments to the C and LD flags.
+   With this option, gcov instrumentation is added to the build and coverage
+   reports are created during execution.  The check-coverage make target is
+   also created to ease report uploading to codecov.io.  The upload requires
+   the COMMIT (git hash) and TOKEN (codecov upload token) environment variables
+   be set.
+
 You may specify any combination of the above options to the configure
 script. By default, the executables are placed in :file:`/usr/local/sbin`
 and the configuration files in :file:`/usr/local/etc`. The :file:`/usr/local/`

diff --git a/docker/alpine/Dockerfile-coverage b/docker/alpine/Dockerfile-coverage
new file mode 100644 (file)
index 0000000..5fdb117
--- /dev/null
+++ b/docker/alpine/Dockerfile-coverage
@@ -0,0 +1,12 @@
+FROM alpine:3.7
+ARG commit
+ARG token
+ENV COMMIT=${commit}
+ENV TOKEN=${token}
+ADD . /src
+RUN cd /src && \
+       source alpine/APKBUILD.in && \
+       apk add --no-cache alpine-sdk $makedepends $checkdepends && \
+       ./bootstrap.sh && \
+       ./configure --enable-gcov
+ENTRYPOINT [ "/bin/sh", "-c", "cd /src && make && make -j 1 check-coverage" ]