Occasional packet loss due to CPU Spike on R77.30

[Muazzam]

Platform: 23500
Version: R77.30 T216
Hyperthreading: Disabled
Blades: Only FW (No other blade enabled)
CoreXL: 8/12 Split (SND/Workers)
Acceleration (SXL): 98%
Multi Queue is enabled on two high throughput (10G) interfaces. Total of 4 queues.
The RX drops on these 2 interfaces are about 0.00005%.
TX/TX buffer on the 2 interfaces are 512/1024.

Typical throughput is about 1-2Gbps, occasional spikes 4-5Gbps.
Occasional packet loss has been reported and we notice CPU spikes 80-90% at the time of packet loss. The first 4 cores (used for SND/MQ) typically runs around 40-50% but at peak throughput goes to 70-80%.
At the time of packet loss, we also notice some workers cores going above 80%, but this is not always the case. Sometimes at peak throughput all workers are close 100% idle, while the 4 MQ cores are over 80%.

This firewall is doing file transfers. Looking at the "Top Connections" in "CPview" at the time of peak traffic, we noticed some connections using 0.8 to 1.5Gbps.

Not sure what needs to be changed.
Is the upgrade to R80.20 would help?
What about changing the interface buffer size?
What about assigning more cores to MQ?

[Timothy_Hall]

With only 4 queues and 8 SND/IRQ cores, you must be using the igb driver which is limited to 4 cores for Multi-Queue, correct?  (ethtool -i  [interfacename] to check)

Please provide output of netstat -ni, and ethtool -S [interfacename] for the 2 busy 10Gbps interfaces.  Also provide output of sar -n EDEV so we can see how packet loss is occurring over time.

Do you have Aggressive Aging under IPS enabled?  If so try disabling it and see if it helps.

Do not modify the interface buffer sizes yet, as all that is doing is addressing a symptom of the problem.

[Muazzam]

[Expert@FW1]# ethtool -i eth1-0x
driver: ixgbe
version: 3.9.15-NAPI
firmware-version: 0x800000cb
bus-info: 0000:87:00.0
** Same for other interface **

[Expert@FW1]# ethtool -S eth1-0x
NIC statistics:
rx_packets: 279704603154
tx_packets: 452829681289
rx_bytes: 18280692593016
tx_bytes: 133428743654784
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 8824673
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 264
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 279704603154
tx_pkts_nic: 452829681297
rx_bytes_nic: 19399511347401
tx_bytes_nic: 135265222145354
lsc_int: 2
tx_busy: 0
non_eop_descs: 0
broadcast: 319
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 1
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 2
rx_flow_control_xon: 0
tx_flow_control_xoff: 2
rx_flow_control_xoff: 0
rx_csum_offload_errors: 0
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 279705731752
fdir_overflow: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_queue_0_packets: 72384819744
tx_queue_0_bytes: 20277441221321
tx_queue_1_packets: 65835988286
tx_queue_1_bytes: 18062272500146
tx_queue_2_packets: 71829483754
tx_queue_2_bytes: 20336772417796
tx_queue_3_packets: 72910129484
tx_queue_3_bytes: 20823012955252
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_5_packets: 0
tx_queue_5_bytes: 0
tx_queue_6_packets: 0
tx_queue_6_bytes: 0
tx_queue_7_packets: 0
tx_queue_7_bytes: 0
tx_queue_8_packets: 23025878448
tx_queue_8_bytes: 8318408460972
tx_queue_9_packets: 23856857091
tx_queue_9_bytes: 7796498083054
tx_queue_10_packets: 11911454580
tx_queue_10_bytes: 3701370972811
tx_queue_11_packets: 11861998596
tx_queue_11_bytes: 3681795471141
tx_queue_12_packets: 12845436216
tx_queue_12_bytes: 3955018588935
tx_queue_13_packets: 13800068566
tx_queue_13_bytes: 4498401641335
tx_queue_14_packets: 12399794438
tx_queue_14_bytes: 3865538912552
tx_queue_15_packets: 11685521160
tx_queue_15_bytes: 3670300606869
tx_queue_16_packets: 12313500819
tx_queue_16_bytes: 3729440172087
tx_queue_17_packets: 12018337900
tx_queue_17_bytes: 3715034617293
tx_queue_18_packets: 11468714242
tx_queue_18_bytes: 3327370364826
tx_queue_19_packets: 12681697974
tx_queue_19_bytes: 3670066669309
rx_queue_0_packets: 69236200991
rx_queue_0_bytes: 4523872267158
rx_queue_1_packets: 69338772648
rx_queue_1_bytes: 4529408746328
rx_queue_2_packets: 71044133302
rx_queue_2_bytes: 4650491437011
rx_queue_3_packets: 70085496214
rx_queue_3_bytes: 4576920142579

[Expert@FW1]# ethtool -S eth1-0y
NIC statistics:
rx_packets: 453012119134
tx_packets: 279650485293
rx_bytes: 133552514014343
tx_bytes: 16637792517188
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 8822800
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 86417
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 453012119134
tx_pkts_nic: 279650485300
rx_bytes_nic: 135364608026416
tx_bytes_nic: 19394865904195
lsc_int: 2
tx_busy: 0
non_eop_descs: 0
broadcast: 318
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 160
rx_flow_control_xon: 0
tx_flow_control_xoff: 1016
rx_flow_control_xoff: 0
rx_csum_offload_errors: 1966
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 453072257774
fdir_overflow: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_queue_0_packets: 38059433284
tx_queue_0_bytes: 2272177335740
tx_queue_1_packets: 46465181059
tx_queue_1_bytes: 2731746714242
tx_queue_2_packets: 39688419944
tx_queue_2_bytes: 2375284815784
tx_queue_3_packets: 39816008722
tx_queue_3_bytes: 2369247053218
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_5_packets: 0
tx_queue_5_bytes: 0
tx_queue_6_packets: 0
tx_queue_6_bytes: 0
tx_queue_7_packets: 0
tx_queue_7_bytes: 0
tx_queue_8_packets: 9967299559
tx_queue_8_bytes: 585454845061
tx_queue_9_packets: 13888064084
tx_queue_9_bytes: 807454063501
tx_queue_10_packets: 9390683604
tx_queue_10_bytes: 545836699667
tx_queue_11_packets: 9258956063
tx_queue_11_bytes: 553428647074
tx_queue_12_packets: 8802007940
tx_queue_12_bytes: 523286664721
tx_queue_13_packets: 9155441077
tx_queue_13_bytes: 533689263046
tx_queue_14_packets: 9008473465
tx_queue_14_bytes: 542386786095
tx_queue_15_packets: 8055269948
tx_queue_15_bytes: 489386917462
tx_queue_16_packets: 9193862041
tx_queue_16_bytes: 560385281280
tx_queue_17_packets: 9396824724
tx_queue_17_bytes: 560785300262
tx_queue_18_packets: 9128944177
tx_queue_18_bytes: 562883460569
tx_queue_19_packets: 10375615609
tx_queue_19_bytes: 624358669844
rx_queue_0_packets: 115145916331
rx_queue_0_bytes: 34050747734185
rx_queue_1_packets: 109844554640
rx_queue_1_bytes: 32164557433634
rx_queue_2_packets: 112210830906
rx_queue_2_bytes: 32867712997960
rx_queue_3_packets: 115810817257
rx_queue_3_bytes: 34469495848564

[Expert@FW1]# enabled_blades
fw ips

From CPVIEW

Blade status Last update Number Update Time
Application Control disabled N/A N/A
Anti-Virus disabled 1109220741 09Sep2009 9:16:51
Anti-Bot disabled N/A N/A
IPS N/A N/A N/A

The output for sar -n EDEV is going to take too long to sanitize. 

IPS (although showing) in not enabled on the gateway.

[Timothy_Hall]

Your network traffic is getting handled pretty cleanly by your SND/IRQ cores even during high CPU load.

One thing I don't understand is that the ixgbe driver can support up to 16 queues under Gaia kernel 2.6.18, yet with your 8/12 split it only appears to be using 4 RX queues spread across CPUs 0-3 which is where you are seeing the high CPU.  Did you have a 4/16 split at one point and change it to the current 8/12?  If so you must now run cpmq reconfigure and reboot for Multi-Queue to take advantage of all 8 SND/IRQ cores.  Please provide output of cpmq get -v.  Note that the cpmq reconfigure procedure is not required on Gaia kernel 3.10.

Or did you explicitly set MQ for only 4 queues?

[Muazzam]

@Timothy_Hall

Yes, a few months ago we made a change in CoreXL and enabled MQ on the 2 interfaces. We can run cpmq reconfigure and reboot if needed.

[Timothy_Hall]

Yes, doing so will help a lot.

[PhoneBoy]

R77.30 is long end of support and you should upgrade.
Even if you're just doing basic firewall, R80.40 should be an improvement for the following reasons:

• Multiqueue enabled on all supported interfaces by default with more queues available due to the newer Linux kernel.
• Dynamic Split is available (not enabled by default) which will automatically adjust the worker/snd split as needed.

Provided your management is running a recent R80.x JHF, you should be able to manage an R80.40 gateway.