lib: Fix to optimize the time taken while batching huge configs

Issue: When the incoming config has say 30K entries of a prefix-lists,
current implementation is to schedule the configs to be batched and
only after batching the entire config, the processing of the configs
take place. As part of batching this config, we perform string
concatenation to save all the configs in the buffer which over time
results in taking longer time.

Ex: Imagine each line of config is 50 chars. With a delimiter of ‘- ‘ we end
up adding 52 chars to buffer for each command i.e. 52*30000 = 156K of chars.
Strlcat is an expensive operation and every time we strlcat, we have to
traverse at end of string to append new char.
Because of this, we end up adding extra 6-8 secs for accepting the config.

Fix: The idea here is to bring back something similar to the backoff
count implemented as part of 20e9a402 (lib: introduce configuration
back-off timer for YANG-modeled commands).

Essentially we keep a cap of 5000 per batch. So once 5000k config
commands are batched, we process them, clear the buffer, set the count
to 0 and then continue processing the rest of the config.

option1 file has 30K entries of prefix-list
Without Fix:
root@mlx-3700-20:mgmt:/var/log/raja/frr# time sudo vtysh -f option1
<SNIP>..............
Waiting for children to finish applying config...
[25191|staticd] done
[25189|watchfrr] done
[25178|ospfd] done
[25190|pbrd] done
[25181|bgpd] done
[25175|zebra] done

real    0m20.123s
user    0m9.384s
sys     0m2.403s

With Fix:
root@mlx-3700-20:mgmt:/var/log/raja/frr# time sudo vtysh -f option1
<SNIP>..............
Waiting for children to finish applying config...
[19887|staticd] done
[19885|watchfrr] done
[19886|pbrd] done
[19874|ospfd] done
[19877|bgpd] done
[19871|zebra] done

real	0m12.168s
user	0m7.511s
sys	0m1.981s

Issue: 3589101

Ticket# 3589101

Signed-off-by: Rajasekar Raja <rajasekarr@nvidia.com>

3 files changed, 16 insertions, 1 deletions

diff --git a/lib/northbound_cli.c b/lib/northbound_cli.c
index f9794bee3c..b199dd61f8 100644
--- a/lib/northbound_cli.c
+++ b/lib/northbound_cli.c
@@ -83,6 +83,7 @@ static int nb_cli_classic_commit(struct vty *vty)
 static void nb_cli_pending_commit_clear(struct vty *vty)
 {
 	vty->pending_commit = 0;
+	vty->buffer_cmd_count = 0;
 	XFREE(MTYPE_TMP, vty->pending_cmds_buf);
 	vty->pending_cmds_buflen = 0;
 	vty->pending_cmds_bufpos = 0;
@@ -102,12 +103,19 @@ int nb_cli_pending_commit_check(struct vty *vty)
 
 static int nb_cli_schedule_command(struct vty *vty)
 {
-	/* Append command to dynamically sized buffer of scheduled commands. */
+	/* Append command to dynamically sized buffer of scheduled commands.
+	 * vty->buf -Incoming config
+	 * vty->pending_cmds_buf - Pending buffer where incoming configs are
+	 *                         accumulated for later processing
+	 * vty->pending_cmds_bufpos - length of the pending buffer
+	 *
+	 */
 	if (!vty->pending_cmds_buf) {
 		vty->pending_cmds_buflen = 4096;
 		vty->pending_cmds_buf =
 			XCALLOC(MTYPE_TMP, vty->pending_cmds_buflen);
 	}
+
 	if ((strlen(vty->buf) + 3)
 	    > (vty->pending_cmds_buflen - vty->pending_cmds_bufpos)) {
 		vty->pending_cmds_buflen *= 2;
@@ -121,6 +129,9 @@ static int nb_cli_schedule_command(struct vty *vty)
 
 	/* Schedule the commit operation. */
 	vty->pending_commit = 1;
+	vty->buffer_cmd_count++;
+	if (vty->buffer_cmd_count == NB_CMD_BATCH_SIZE)
+		nb_cli_pending_commit_check(vty);
 
 	return CMD_SUCCESS;
 }
diff --git a/lib/northbound_cli.h b/lib/northbound_cli.h
index 4c8dc50bd2..43c40f49e1 100644
--- a/lib/northbound_cli.h
+++ b/lib/northbound_cli.h
@@ -20,6 +20,9 @@ enum nb_cfg_format {
 	NB_CFG_FMT_XML,
 };
 
+/* Maximum config commands in a batch*/
+#define NB_CMD_BATCH_SIZE 5000
+
 extern struct nb_config *vty_shared_candidate_config;
 
 /*
diff --git a/lib/vty.h b/lib/vty.h
index be54159aa9..c6f9f5a3a7 100644
--- a/lib/vty.h
+++ b/lib/vty.h
@@ -149,6 +149,7 @@ struct vty {
 	struct nb_config *candidate_config_base;
 
 	/* Dynamic transaction information. */
+	size_t buffer_cmd_count;
 	bool pending_allowed;
 	bool pending_commit;
 	char *pending_cmds_buf;