Who rated this post

Turning off SMT will probably fix the problem, but not for the reasons you think.

Are you using Dynamic Split?  I don't think you are, because you have 10 real cores trying to execute both SND and Worker functions, if I'm understanding your architecture correctly due to how SMT divides cores.  26000 uses a 2x Intel Xeon Gold 6254 CPU @ 3.10GHz (18C, 36T).  So 2 sockets, but your split appears to be set for only one socket.  With SMT, socket 1 is cores 0-35 (18 real & 18 siblings), socket 2 is cores 36-71 (18 real & 18 siblings).

Your SNDs are cores 0-4 and cores 36-40, which would make sense if there were a single socket, as these would be sibling cores.  But you have two sockets, so cores 0-4 are siblings with cores 18-22 (all workers), and cores 36-40 are siblings with cores 54-58 (all workers).  So you have 10 of your 36 real cores trying to perform both SND and worker functions. 

It's easy to see how the CoreXL queues could overflow if the worker is constantly getting clobbered by SND functions. Since the SNDs run in the kernel for KPPAK mode, the busier the SNDs get, the more the conflicting worker instances get kicked off the sibling core to make way for the kernel, which has ultimate scheduling priority over measly USFW worker processes.  This may also cause significant traffic between NUMA nodes, further impacting performance.  Doing fast_accel's would make this situation much worse on the 10 overlapping real cores, by making the conflicting SNDs even busier than they were before.

Check my work here, please @Bob_Zimmerman.