Kaspars Zibarts

CoreXL gone rogue on VSX after Take 112

Discussion created by Kaspars Zibarts on Aug 3, 2018
Latest reply on Aug 3, 2018 by Dameon Welch-Abernathy

Just wondering if anyone else has seen this. Upgraded our VSX (13800 appliances) from R80.10 Take 42 to 112 yesterday. All went great and no issues during update process but this morning noticed that CPU core utilisation profile has considerable changes - 10 cores that were allocated to a busy VS dropped from 25% ave to nearly zero and some others jumped up, especially two SXL cores.

 

Sample of busy VS

 

Sample of SXL

 

First I thought that we have started accelerating more but that was not the case as total fwk CPU usage on that busy VS was just under 200% (so should have been 20% per core with 10 cores) and SXL stats were as before. So nothing had changed traffic / acceleration wise.

 

Next thing I suspected that affinity settings might have been modified by T112 somehow. But it all looked great from config point of view for VS6 - 10 dedicated cores

 

 

But CoreXL output showed that nearly all 20 cores (2-19) are in use by VS6! What!!??

 

 

Re-run affinity command and bingo - cores allocated as before T112

 

 

CPU usage profile back to normal after I re-run all affinity commands

 

Same happened with another VSX: 23800 appliance cluster, upgrade from Take 35 instead of 42 though.

 

Regular firewalls seem to have avoided any issues with CoreXL.

 

Ouch!

Outcomes