When FRR intentionally asserts currently, the assertion
stops program execution and any debug logs currently
in play may just be lost completely.
Attempt to grab the abort and cleanup the log file, maybe we'll have
something useful. New behavior:
zebra: lib/plist.c:562: void trie_install_fn(struct prefix_list_entry *, struct prefix_list_entry **): Assertion `object->next_best == *updptr || !*updptr' failed.
ZEBRA: Received signal 6 at
1611269027 (si_addr 0x7700138569, PC 0x7fdb210cec81); aborting...
ZEBRA: zlog_signal+0xb3
7fdb2140aa73 7ffdd8f67c90 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: core_handler+0xd8
7fdb21433e38 7ffdd8f67d90 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: funlockfile+0x50
7fdb2126c140 7ffdd8f67f00 /lib/x86_64-linux-gnu/libpthread.so.0 (mapped at 0x7fdb21258000)
ZEBRA: ---- signal ----
ZEBRA: gsignal+0x141
7fdb210cec81 7ffdd8f684b0 /lib/x86_64-linux-gnu/libc.so.6 (mapped at 0x7fdb21093000)
ZEBRA: abort+0x123
7fdb210b8537 7ffdd8f685d0 /lib/x86_64-linux-gnu/libc.so.6 (mapped at 0x7fdb21093000)
ZEBRA: ?
7fdb210b840f 7ffdd8f68700 /lib/x86_64-linux-gnu/libc.so.6 (mapped at 0x7fdb21093000)
ZEBRA: __assert_fail+0x42
7fdb210c7602 7ffdd8f68750 /lib/x86_64-linux-gnu/libc.so.6 (mapped at 0x7fdb21093000)
ZEBRA: trie_install_fn+0x131
7fdb214200c1 7ffdd8f68780 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: trie_walk_affected+0x44
7fdb2141fe14 7ffdd8f687a0 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: prefix_list_trie_add+0x12f
7fdb2141e8ff 7ffdd8f687e0 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: prefix_list_entry_update_finish+0x192
7fdb2141e752 7ffdd8f68830 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: lib_prefix_list_entry_prefix_modify+0xa4
7fdb213faea4 7ffdd8f68860 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: lib_prefix_list_entry_ipv4_prefix_modify+0xf
7fdb213fa6af 7ffdd8f688c0 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: nb_callback_configuration+0x422
7fdb214175c2 7ffdd8f688d0 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: nb_candidate_commit_apply+0x66
7fdb21414d86 7ffdd8f68d60 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: nb_candidate_commit+0x66
7fdb21415146 7ffdd8f691f0 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: nb_cli_classic_commit+0x5f
7fdb2141784f 7ffdd8f69230 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: nb_cli_apply_changes+0x4ec
7fdb21417e8c 7ffdd8f6b270 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: ip_prefix_list+0x552
7fdb213f7bb2 7ffdd8f6d780 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: cmd_execute_command_real+0x14c
7fdb213e554c 7ffdd8f6e0f0 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: cmd_execute_command+0x5d
7fdb213e52bd 7ffdd8f6e130 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: cmd_execute+0xa1
7fdb213e5651 7ffdd8f6e180 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: vty_execute+0x253
7fdb2144b643 7ffdd8f6e1d0 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: vtysh_read+0xf9
7fdb214494d9 7ffdd8f70210 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: thread_call+0x8f
7fdb214419ef 7ffdd8f70450 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: frr_run+0x298
7fdb214091a8 7ffdd8f705d0 /lib/libfrr.so.0 (mapped at 0x7fdb213af000)
ZEBRA: main+0x300 42e760
7ffdd8f70740 /usr/lib/frr/zebra (mapped at 0x400000)
ZEBRA: __libc_start_main+0xea
7fdb210b9d0a 7ffdd8f70820 /lib/x86_64-linux-gnu/libc.so.6 (mapped at 0x7fdb21093000)
ZEBRA: _start+0x2a 42022a
7ffdd8f708f0 /usr/lib/frr/zebra (mapped at 0x400000)
ZEBRA: in thread vtysh_read scheduled from lib/vty.c:2688
core_handler: showing active allocations in memory group libfrr
core_handler: memstats: Buffer : 2 * 24
core_handler: memstats: Host config : 3 * (variably sized)
core_handler: memstats: Command Tokens : 4117 * 72
core_handler: memstats: Command Token Text : 2964 * (variably sized)
core_handler: memstats: Command Token Help : 2964 * (variably sized)
core_handler: memstats: Command Argument : 7 * (variably sized)
core_handler: memstats: Command Argument Name : 1083 * (variably sized)
core_handler: memstats: RCU thread : 2 * 128
core_handler: memstats: FRR POSIX Thread : 4 * (variably sized)
core_handler: memstats: POSIX sync primitives : 4 * (variably sized)
core_handler: memstats: Graph : 25 * 8
core_handler: memstats: Graph Node : 4795 * 32
core_handler: memstats: Hash : 104 * (variably sized)
core_handler: memstats: Hash Bucket : 33272 * 32
core_handler: memstats: Hash Index : 52 * (variably sized)
core_handler: memstats: Interface : 11 * 272
core_handler: memstats: Connected : 28 * 48
core_handler: memstats: Link List : 83 * 40
core_handler: memstats: Link Node : 127 * 24
core_handler: memstats: Temporary memory : 36903 * (variably sized)
core_handler: memstats: Module loading name : 1 * 13
core_handler: memstats: Nexthop : 9 * 136
core_handler: memstats: NetNS Context : 2 * (variably sized)
core_handler: memstats: NetNS Name : 1 * 18
core_handler: memstats: Northbound Node : 640 * 1168
core_handler: memstats: Northbound Configuration : 2 * 16
core_handler: memstats: Northbound Configuration Entry: 32398 * 1032
core_handler: memstats: Prefix List : 1 * 80
core_handler: memstats: Prefix List Str : 1 * 26
core_handler: memstats: Prefix List Entry : 32397 * 128
core_handler: memstats: Prefix List Trie Table : 196 * 4096
core_handler: memstats: Prefix : 28 * 48
core_handler: memstats: Privilege information : 3 * (variably sized)
core_handler: memstats: Stream FIFO : 1 * 64
core_handler: memstats: Route table : 22 * 56
core_handler: memstats: Route node : 54 * (variably sized)
core_handler: memstats: Thread : 25 * 168
core_handler: memstats: Thread master : 12 * (variably sized)
core_handler: memstats: Thread Poll Info : 6 * 8192
core_handler: memstats: Thread stats : 16 * 72
core_handler: memstats: Typed-hash bucket : 15 * (variably sized)
core_handler: memstats: Typed-heap array : 1 * 576
core_handler: memstats: Vector : 9646 * 16
core_handler: memstats: Vector index : 9646 * (variably sized)
core_handler: memstats: VRF : 1 * 200
core_handler: memstats: VTY : 2 * (variably sized)
core_handler: memstats: Work queue : 3 * (variably sized)
core_handler: memstats: Work queue name string : 2 * (variably sized)
core_handler: memstats: YANG module : 6 * 48
core_handler: memstats: log thread-local buffer : 3 * 9752
core_handler: showing active allocations in memory group logging subsystem
core_handler: memstats: log file target : 1 * 88
core_handler: showing active allocations in memory group Label Manager
core_handler: showing active allocations in memory group Table Manager
core_handler: showing active allocations in memory group zebra
core_handler: memstats: Zebra Interface Information : 11 * 488
core_handler: memstats: Router Advertisement Prefix : 2 * 48
core_handler: memstats: Zebra DPlane Provider : 1 * 232
core_handler: memstats: Route Entry : 24 * 88
core_handler: memstats: RIB destination : 20 * 88
core_handler: memstats: Nexthop Group Entry : 9 * 88
core_handler: memstats: Nexthop Group Connected : 9 * 40
core_handler: memstats: Zebra Name Space : 5 * (variably sized)
core_handler: memstats: RIB table info : 4 * 16
core_handler: memstats: ZEBRA VRF : 1 * 4744
core_handler: memstats: MH global info : 1 * 128
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x0 updptr: 0x11b4ea0 *updptr: 0x11b5e10
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11b62b0 updptr: 0x11b4eb0 *updptr: 0x0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x0 updptr: 0x11b5818 *updptr: 0x11b7090
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x0 updptr: 0x11b5888 *updptr: 0x0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x0 updptr: 0x11b5910 *updptr: 0x0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x0 updptr: 0x11b59a0 *updptr: 0x11b87b0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x0 updptr: 0x11b7100 *updptr: 0x11b87b0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x0 updptr: 0x11b75a0 *updptr: 0x11b87b0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x0 updptr: 0x11b7a40 *updptr: 0x11b87b0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x0 updptr: 0x11b7ee0 *updptr: 0x11b87b0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11b8c50 updptr: 0x11b8380 *updptr: 0x11b8c50
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11b90f0 updptr: 0x11b8820 *updptr: 0x11b90f0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11b9590 updptr: 0x11b8cc0 *updptr: 0x11b9590
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11b9a30 updptr: 0x11b9160 *updptr: 0x11b9a30
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11b9ed0 updptr: 0x11b9600 *updptr: 0x11b9ed0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11ba370 updptr: 0x11b9aa0 *updptr: 0x11ba370
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11ba810 updptr: 0x11b9f40 *updptr: 0x11ba810
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11bacb0 updptr: 0x11ba3e0 *updptr: 0x11bacb0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11bb150 updptr: 0x11b4e20 *updptr: 0x0
2021/01/21 17:43:47 ZEBRA: object->next_best: 0x11bb5f0 updptr: 0x11c2560 *updptr: 0x11c1710
fish: Job 2, “sudo /usr/lib/frr/zebra --log s…” terminated by signal SIGABRT (Abort)
Signed-off-by: Donald Sharp <sharpd@nvidia.com>