| Age | Commit message (Collapse) | Author |
|---|
|
Make code changes in fpm_read to create a list of ctx and send it to
zebra for processing rather than sending individual ctx
Signed-off-by: Krishnasamy <krishnasamyr@nvidia.com>
|
|
Signed-off-by: Donna Sharp <dksharp5@gmail.com>
|
|
Coverity 2024 new hotness
|
|
When looping through the dplane providers, the worklist was
being populated with items from the last provider and then
the event system was checked to see if we should stop processing.
If the event system says `yes` then the dplane code would stop
and send the worklist to the master zebra pthread for collection.
This obviously skipped the next dplane provider on the list
which is double plus not good.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
The dg_update_list access is controlled by the dg_mutex in all
other locations. Let's just add a mutex usage around the initialization
of the dg_update_list even if it's part of the startup, just to keep
things consistent.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Remove a route api that's no longer used after refactoring the
netlink and FPM code.
Signed-off-by: Mark Stapp <mjs@cisco.com>
|
|
Add several additional route attribute data and accessors
to the dplane module.
Signed-off-by: Mark Stapp <mjs@cisco.com>
|
|
clang-19's SA complains about unused initializers for this kind of
"switch (enum) { return string }" kind of code. Use direct string
return values to avoid the issue.
Signed-off-by: David Lamparter <equinox@opensourcerouting.org>
|
|
zebra: vlan to dplane
|
|
Currently FRR is limiting the nexthop count to a uint8_t not a
uint16_t. This leads to issues when the nexthop count is 256
which results in the count to overflow to 0 causing problems
in the code.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Trigger: Zebra core seen when we convert l2vni to l3vni and back
BackTrace:
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(_zlog_assert_failed+0xe9) [0x7f4af96989d9]
/usr/lib/frr/zebra(zebra_vxlan_if_vni_up+0x250) [0x5561022ae030]
/usr/lib/frr/zebra(netlink_vlan_change+0x2f4) [0x5561021fd354]
/usr/lib/frr/zebra(netlink_parse_info+0xff) [0x55610220d37f]
/usr/lib/frr/zebra(+0xc264a) [0x55610220d64a]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x7d) [0x7f4af967e96d]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7f4af9637588]
/usr/lib/frr/zebra(main+0x402) [0x5561021f4d32]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f4af932624a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f4af9326305]
/usr/lib/frr/zebra(_start+0x21) [0x5561021f72f1]
Root Cause:
In working case,
- We get a RTM_NEWLINK whose ctx is enqueued by zebra dplane and
dequeued by zebra main and processed i.e.
(102000 is deleted from vxlan99) before we handle RTM_NEWVLAN.
- So in handling of NEWVLAN (vxlan99) we bail out since find with
vlan id 703 does not exist.
root@leaf2:mgmt:/var/log/frr# cat ~/raja_logs/working/nocras.log | grep "RTM_NEWLINK\|QUEUED\|vxlan99\|in thread"
2024/07/18 23:09:33.741105 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=616, seq=0, pid=0
2024/07/18 23:09:33.744061 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f2244000cf0, op INTF_INSTALL, ifindex (65), result QUEUED
2024/07/18 23:09:33.767240 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=508, seq=0, pid=0
2024/07/18 23:09:33.767380 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f2244000cf0, op INTF_INSTALL, ifindex (73), result QUEUED
2024/07/18 23:09:33.767389 ZEBRA: [NVFT0-HS1EX] INTF_INSTALL for vxlan99(73)
2024/07/18 23:09:33.767404 ZEBRA: [TQR2A-H2RFY] Vlan-Vni(1186:1186-6000002:6000002) update for VxLAN IF vxlan99(73)
2024/07/18 23:09:33.767422 ZEBRA: [TP4VP-XZ627] Del L2-VNI 102000 intf vxlan99(73)
2024/07/18 23:09:33.767858 ZEBRA: [QYXB9-6RNNK] RTM_NEWVLAN bridge IF vxlan99 NS 0
2024/07/18 23:09:33.767866 ZEBRA: [KKZGZ-8PCDW] Cannot find VNI for VID (703) IF vxlan99 for vlan state update >>>>BAIL OUT
In failure case,
- The NEWVLAN is received first even before processing RTM_NEWLINK.
- Since the vxlan id 102000 is not removed from the vxlan99,
the find with vlan id 703 returns the 102000 one and we invoke
zebra_vxlan_if_vni_up where the interfaces don't match and assert.
root@leaf2:mgmt:/var/log/frr# cat ~/raja_logs/noworking/crash.log | grep "RTM_NEWLINK\|QUEUED\|vxlan99\|in thread"
2024/07/18 22:26:43.829370 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=616, seq=0, pid=0
2024/07/18 22:26:43.829646 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7fe07c026d30, op INTF_INSTALL, ifindex (65), result QUEUED
2024/07/18 22:26:43.853930 ZEBRA: [QYXB9-6RNNK] RTM_NEWVLAN bridge IF vxlan99 NS 0
2024/07/18 22:26:43.853949 ZEBRA: [K61WJ-XQQ3X] Intf vxlan99(73) L2-VNI 102000 is UP >>> VLAN PROCESSED BEFORE INTF EVENT
2024/07/18 22:26:43.853951 ZEBRA: [SPV50-BX2RP] RAJA zevpn_vxlanif vxlan48 and ifp vxlan99
2024/07/18 22:26:43.854005 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=508, seq=0, pid=0
2024/07/18 22:26:43.854241 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=516, seq=0, pid=0
2024/07/18 22:26:43.854251 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=544, seq=0, pid=0
ZEBRA: in thread kernel_read scheduled from zebra/kernel_netlink.c:505 kernel_read()
Fix:
Similar to #13396, where link change
handling was offloaded to dplane, same is being done for vlan events.
Note: Prior to this change, zebra main thread was interested in the
RTNLGRP_BRVLAN. So all the kernel events pertaining to vlan was
handled by zebra main.
With this change change as well the handling of vlan events is still
with Zebra main. However we offload it via Dplane thread.
Ticket :#3878175
Signed-off-by: Rajasekar Raja <rajasekarr@nvidia.com>
|
|
The show zebra dplane provider command was ommitting
the input and output queues to the dplane itself.
It would be nice to have this insight as well.
New output:
r1# show zebra dplane providers
dataplane Incoming Queue from Zebra: 100
Zebra dataplane providers:
Kernel (1): in: 6, q: 0, q_max: 3, out: 6, q: 14, q_max: 3
dplane_fpm_nl (2): in: 6, q: 10, q_max: 3, out: 6, q: 0, q_max: 3
dataplane Outgoing Queue to Zebra: 43
r1#
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Currently when the dplane_thread_loop is run, it moves contexts
from the dg_update_list and puts the contexts on the input queue
of the first provider. This provider is given a chance to run
and then the items on the output queue are pulled off and placed
on the input queue of the next provider. Rinse/Repeat down through
the entire list of providers. Now imagine that we have a list
of multiple providers and the last provider is getting backed up.
Contexts will end up sticking in the input Queue of the `slow`
provider. This can grow without bounds. This is a real problem
when you have a situation where an interface is flapping and an
upper level protocol is sending a continous stream of route
updates to reflect the change in ecmp. You can end up with
a very very large backlog of contexts. This is bad because
zebra can easily grow to a very very large memory size and on
restricted systems you can run out of memory. Fortunately
for us, the MetaQ already participates with this process
by not doing more route processing until the dg_update_list
goes below the working limit of dg_updates_per_cycle. Thus
if FRR modifies the behavior of this loop to not move more
contexts onto the input queue if either the input queue
or output queue of the next provider has reached this limit.
FRR will naturaly start auto handling backpressure for the dplane
context system and memory will not go out of control.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
The ctx queue data structures already have a counter
associated with them. Let's just use them instead.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Currently the FRR code will receive both kernel and
connected routes that do not actually have an underlying
nexthop group at all. Zebra turns around and creates
a `matching` nexthop hash entry and installs it.
For connected routes, this will create 2 singleton
nexthops in the dplane per interface (v4 and v6).
For kernel routes it would just create 1 singleton
nexthop that might be used or not.
This is bad because the dplane has a limited amount
of space available for nexthop entries and if you
happen to have a large number of interfaces then
all of a sudden you have 2x(# of interfaces) singleton
nexthops.
Let's modify the code to delay creation of these singleton
nexthops until they have been used by something else in the
system.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
The nexthop updates counter value was never displayed.
Add it.
> # show zebra dplane
> Zebra dataplane:
> Route updates: 7673010
> Route update errors: 0
> Nexthop updates: 1100
> Nexthop update errors: 0
> [..]
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
|
|
The bgp_duplicate_nexthop test installs routes with nexthop's
flags set to both DUPLICATE and FIB: this should not happen.
The DUPLICATE flag of a nexthop indicates this nexthop is already
used in the same nexthop-group, and there is no need to install it
twice in the system; having the FIB flag set indicates that the
nexthop is installed in the system. This is why both flags should
not be set on the same nexthop.
This case happens at installation time, but can also happen
at update time.
- Fix this by not setting the FIB flag value when the DUPLICATE
flag is present.
- Modify the bgp_duplicate_test to check that the FIB flag is not
present on duplicated nexthops.
- Modify the bgp_peer_type_multipath_relax test.
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
|
|
zebra: Prevent starvation in dplane_thread_loop
|
|
When removing a large number of routes, the linux kernel can take the
cpu for an extended amount of time, leaving a situation where FRR
detects a starvation event.
r1# sharp install routes 10.0.0.0 nexthop 192.168.44.33 1000000 repeat 10
2024-06-14 12:55:49.365 [NTFY] sharpd: [M7Q4P-46WDR] vty[5]@# sharp install routes 10.0.0.0 nexthop 192.168.44.33 1000000 repeat 10
2024-06-14 12:55:49.365 [DEBG] sharpd: [YP4TQ-01TYK] Inserting 1000000 routes
2024-06-14 12:55:57.256 [DEBG] sharpd: [TPHKD-3NYSB] Installed All Items 7.890085
2024-06-14 12:55:57.256 [DEBG] sharpd: [YJ486-NX5R1] Removing 1000000 routes
2024-06-14 12:56:07.802 [WARN] zebra: [QH9AB-Y4XMZ][EC 100663314] STARVATION: task dplane_thread_loop (634377bc8f9e) ran for 7078ms (cpu time 220ms)
2024-06-14 12:56:25.039 [DEBG] sharpd: [WTN53-GK9Y5] Removed all Items 27.783668
2024-06-14 12:56:25.039 [DEBG] sharpd: [YP4TQ-01TYK] Inserting 1000000 routes
2024-06-14 12:56:32.783 [DEBG] sharpd: [TPHKD-3NYSB] Installed All Items 7.743524
2024-06-14 12:56:32.783 [DEBG] sharpd: [YJ486-NX5R1] Removing 1000000 routes
2024-06-14 12:56:41.447 [WARN] zebra: [QH9AB-Y4XMZ][EC 100663314] STARVATION: task dplane_thread_loop (634377bc8f9e) ran for 5175ms (cpu time 179ms)
Let's modify the loop in dplane_thread_loop such that after a provider
has been run, check to see if the event should yield, if so, stop
and reschedule this for the future.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Instead of keeping a counter that is independent
of the queue's data structure. Just use the queue's
built-in counter. Ensure that it's pthread safe by
keeping it wrapped inside the mutex for adding/deleting
to the queue.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Add a version value and accessor API for the zebra dataplane;
plugins can test this to detect API changes.
Signed-off-by: Mark Stapp <mjs@cisco.com>
|
|
these changes are for improving the code maintainability and readability
Signed-off-by: sri-mohan1 <sri.mohan@samsung.com>
|
|
Fix the following coverity issue:
*** CID 1575079: Null pointer dereferences (REVERSE_INULL)
/zebra/zebra_dplane.c: 5950 in dplane_srv6_encap_srcaddr_set()
5944 if (ret == AOK)
5945 result = ZEBRA_DPLANE_REQUEST_QUEUED;
5946 else {
5947 atomic_fetch_add_explicit(&zdplane_info
5948 .dg_srv6_encap_srcaddr_set_errors,
5949 1, memory_order_relaxed);
CID 1575079: Null pointer dereferences (REVERSE_INULL)
Null-checking "ctx" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.
5950 if (ctx)
5951 dplane_ctx_free(&ctx);
5952 }
5953 return result;
5954 }
5955
Remove the pointer check for `ctx`. At this point in the
function it has to be non null since we deref'ed it.
Additionally the alloc function that creates it cannot
fail.
Signed-off-by: Carmine Scarpitta <cscarpit@cisco.com>
|
|
Add a bunch of set functions and associated data structure in
zebra_dplane to allow the configuration of the source address for SRv6
encap in the data plane.
Signed-off-by: Carmine Scarpitta <carmine.scarpitta@uniroma2.it>
|
|
The linux kernel can send up a flag that tells us that the
connected address is not a PREFIXROUTE. Add the ability
to note this and pass it up from the data plane.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Zebra final shutdown finally
|
|
Fix a crash because a use-after-free.
> =================================================================
> ==1249835==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000074210 at pc 0x7fa1b42a652c bp 0x7ffc477a2aa0 sp 0x7ffc477a2a98
> READ of size 8 at 0x604000074210 thread T0
> #0 0x7fa1b42a652b in list_delete_all_node git/frr/lib/linklist.c:299:20
> #1 0x7fa1b42a683f in list_delete git/frr/lib/linklist.c:312:2
> #2 0x5ee515 in dplane_ctx_free_internal git/frr/zebra/zebra_dplane.c:858:4
> #3 0x5ee59c in dplane_ctx_free git/frr/zebra/zebra_dplane.c:884:2
> #4 0x5ee544 in dplane_ctx_fini git/frr/zebra/zebra_dplane.c:905:2
> #5 0x7045c0 in rib_process_dplane_results git/frr/zebra/zebra_rib.c:4928:4
> #6 0x7fa1b4434fb2 in event_call git/frr/lib/event.c:1970:2
> #7 0x7fa1b42a0ccf in frr_run git/frr/lib/libfrr.c:1213:3
> #8 0x556808 in main git/frr/zebra/main.c:488:2
> #9 0x7fa1b3d0bd09 in __libc_start_main csu/../csu/libc-start.c:308:16
> #10 0x4453e9 in _start (/usr/lib/frr/zebra+0x4453e9)
>
> 0x604000074210 is located 0 bytes inside of 40-byte region [0x604000074210,0x604000074238)
> freed by thread T0 here:
> #0 0x4bf1dd in free (/usr/lib/frr/zebra+0x4bf1dd)
> #1 0x7fa1b42df0c0 in qfree git/frr/lib/memory.c:130:2
> #2 0x7fa1b42a68ce in list_free_internal git/frr/lib/linklist.c:24:2
> #3 0x7fa1b42a6870 in list_delete git/frr/lib/linklist.c:313:2
> #4 0x5ee515 in dplane_ctx_free_internal git/frr/zebra/zebra_dplane.c:858:4
> #5 0x5ee59c in dplane_ctx_free git/frr/zebra/zebra_dplane.c:884:2
> #6 0x5ee544 in dplane_ctx_fini git/frr/zebra/zebra_dplane.c:905:2
> #7 0x7045c0 in rib_process_dplane_results git/frr/zebra/zebra_rib.c:4928:4
> #8 0x7fa1b4434fb2 in event_call git/frr/lib/event.c:1970:2
> #9 0x7fa1b42a0ccf in frr_run git/frr/lib/libfrr.c:1213:3
> #10 0x556808 in main git/frr/zebra/main.c:488:2
> #11 0x7fa1b3d0bd09 in __libc_start_main csu/../csu/libc-start.c:308:16
>
> previously allocated by thread T0 here:
> #0 0x4bf5d2 in calloc (/usr/lib/frr/zebra+0x4bf5d2)
> #1 0x7fa1b42dee18 in qcalloc git/frr/lib/memory.c:105:27
> #2 0x7fa1b42a3784 in list_new git/frr/lib/linklist.c:18:9
> #3 0x6d165f in pbr_iptable_alloc_intern git/frr/zebra/zebra_pbr.c:1015:29
> #4 0x7fa1b426ad1f in hash_get git/frr/lib/hash.c:147:13
> #5 0x6d15f2 in zebra_pbr_add_iptable git/frr/zebra/zebra_pbr.c:1030:13
> #6 0x5db2a3 in zread_iptable git/frr/zebra/zapi_msg.c:3759:3
> #7 0x5e365d in zserv_handle_commands git/frr/zebra/zapi_msg.c:4039:3
> #8 0x7e09fc in zserv_process_messages git/frr/zebra/zserv.c:520:3
> #9 0x7fa1b4434fb2 in event_call git/frr/lib/event.c:1970:2
> #10 0x7fa1b42a0ccf in frr_run git/frr/lib/libfrr.c:1213:3
> #11 0x556808 in main git/frr/zebra/main.c:488:2
> #12 0x7fa1b3d0bd09 in __libc_start_main csu/../csu/libc-start.c:308:16
Fixes: 1cc380679e ("zebra: Actually free all memory associated ctx->u.iptable.interface_name_list")
Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com>
|
|
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
On shutdown go through and ensure that any contexts the
dplane provider holds are freed.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
INTERFACE_NAMSIZ is just a redefine of IFNAMSIZ and IFNAMSIZ
is the standard for interface name length on all platforms
that FRR currently compiles on.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
The iprule/pbr rule object has a vrf id, and zebra uses
that internally, but the vrf id isn't returned to clients
who install rules and are waiting for results. Include the
vrf_id sent by the client in the zapi result notification
message; update the existing clients so they decode the id.
Signed-off-by: Mark Stapp <mjs@labn.net>
|
|
Add the txqlen attribute to the common interface struct. Capture
the value in zebra, and distribute it through the interface lib
module's zapi messaging.
Signed-off-by: Mark Stapp <mjs@labn.net>
|
|
Upon some internal testing some crashes were found. This fixes
the several crashes and normalizes the code to be closer in
it's execution pre and post changes to use the data plane.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Signed-off-by: G. Paul Ziemba <paulz@labn.net>
|
|
Subset: zebra dataplane
Add new vlan filter fields. No kernel dataplane
implementation yet (linux does not support).
Changes by:
Josh Werner <joshuawerner@mitre.org>
Eli Baum <ebaum@mitre.org>
G. Paul Ziemba <paulz@labn.net>
Signed-off-by: G. Paul Ziemba <paulz@labn.net>
|
|
The function `dplane_ctx_route_init` initializes a dplane route context
from the route object passed as an argument. Let's abstract this
function to allow initializing the dplane route context without actually
copying a route object.
This allows us to use this function for initializing a dplane route
context when we don't have any route to copy in it.
Signed-off-by: Carmine Scarpitta <carmine.scarpitta@uniroma2.it>
|
|
1) Add a bunch of get/set functions and associated data
structure in zebra_dplane to allow the setting and retrieval
of interface netlink data up into the master pthread.
2) Add a bit of code to breakup startup into stages. This is
because FRR currently has a mix of dplane and non dplane interactions
and the code needs to be paused before continuing on.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
There is no need for this functionality and it is
not used.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Signed-off-by: G. Paul Ziemba <paulz@labn.net>
|
|
There is no path in some functions where the ctx
has not already been de-refed. As such no need
to test for it's existence.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
The prov->dp_out_queued counter was never being decremented
when a ctx was pulled off of the list. Let's change it to
accurately reflect real life.
Broken:
janelle.pinkbelly.org# show zebra dplane providers detailed
Zebra dataplane providers:
Kernel (1): in: 330872, q: 0, q_max: 100, out: 330872, q: 330872, q_max: 330872
janelle.pinkbelly.org#
Fixed:
sharpd@janelle:/tmp/topotests$ vtysh -c "show zebra dplane providers detailed"
Zebra dataplane providers:
Kernel (1): in: 221495, q: 0, q_max: 100, out: 221495, q: 0, q_max: 100
sharpd@janelle:/tmp/topotests$
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Prevent this function from leaking the ctx memory.
Also properly record that something has gone wrong.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Having tests for memory allocation success makes no sense
given what happens when frr fails to allocate memory.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Remove the pointer check for ctx. At this point in the
function it has to be non null since we deref'ed it.
Additionally the alloc function that creates it cannot
fail.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
zebra: Mark connected route as installed after interface flap event
|
|
Issue:
After vlan flap, zebra was not marking the selected/best route as installed.
As a result, when a static route was configured with nexthop as directly
connected interface's(vlan) IP, the static route was not being installed
in the kernel since its nexthop was unresolved. The nexthop was marked
unresolved because zebra failed to mark the best route as installed after
interface flap.
This was happening because, in dplane_route_update_internal() if the old and
new context type, and nexthop group id are the same, then zebra doesn't send
down a route replace request to kernel. But, the installed (ROUTE_ENTRY_INSTALLED)
flag is set when zebra receives a response from kernel. Since the
request to kernel was being skipped for the route entry, installed flag
was not being set
Fix:
In dplane_route_update_internal() if the old and new context type, and
nexthop group id are the same, then before returning, installed flag will
be set on the route-entry if it's not set already.
Signed-off-by: Pooja Jagadeesh Doijode <pdoijode@nvidia.com>
|
|
During shutdown, the main pthread stops the dplane pthread
before exiting. Don't try to clean up any events scheduled
to the dplane pthread at that point - just let the thread
exit and clean up.
Signed-off-by: Mark Stapp <mjs@labn.net>
|
|
Let's find a better name for it.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
|
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
|
