topic Re: core affinity R80.40 - two cores in General Topics

core affinity R80.40 - two cores

D_TK — Mon, 15 Jun 2020 20:05:30 GMT

did the blink R80.40 upgrade on a 4600 appliance cluster that was previously running R80.20. On 80.20, with corexl enabled, this was the distribution, and RX-DRPs were typically close to zero. :

# sim affinity -l
Mgmt : 0
eth1 : 1
eth2 : 1
eth3 : 0
eth5 : 0

After the blink upgrade, the appliances were in USFW mode which seemed strange for a 2 core box. working with TAC, i changed them back to kernel mode and with corexl enabled, the allocation doesn't change from:

sim affinity -l
Mgmt : 0
eth1 : 0
eth2 : 0
eth3 : 0
eth5 : 0

fw ctl affinity -l -r
CPU 0: eth5 eth1 eth2 eth3 Mgmt
fw_1
CPU 1: fw_0
All: in.asessiond mpdaemon in.acapd usrchkd pepd in.geod rad rtmd fwd lpd vpnd pdpd cpd cprid

and RX-DRPs are accumulating.

I've never had to change corexl from "auto" mode - should i even attempt to balance the interfaces? I haven't heard about any user experience issues, but traffic is pretty light right now with most folks still WFH.

thanks

Re: core affinity R80.40 - two cores

_Val_ — Tue, 16 Jun 2020 06:50:26 GMT

Are you sure you only have two cores on your appliance? I think it has to be 4.

With auto-core assignment, and 4 cores, only one of cores serves as SND.

Re: core affinity R80.40 - two cores

D_TK — Tue, 16 Jun 2020 17:00:19 GMT

Yep, the 4600 only has 2 cores.

Re: core affinity R80.40 - two cores

Timothy_Hall — Wed, 17 Jun 2020 00:31:23 GMT

There is no automatic interface affinity in R80.40, at least the way it was implemented in earlier releases. On Gaia 3.10 Multi-Queue is enabled for all interfaces (except management) which spreads the SoftIRQ/SecureXL load across all your SND cores (2) in your 2/2 split. It is possible that the interfaces on the 4600 (a pretty old box) are not capable of Multi-Queue and that is why everything is on Core 0 instead.

What does output of expert mode command mq_mng --show reveal? If that command doesn't work try the clish command show interface (interface name) multi-queue.

Re: core affinity R80.40 - two cores

D_TK — Wed, 17 Jun 2020 01:22:42 GMT

Thanks Tim. No surprise i received this message to that command: "No multiqueue supported interfaces available".

This cluster ran very clean on R80.20 - it has under 50 users behind it, and is fed by only a 50M MPLS link.

I attached a netstat -ni, the strangest part is that the interface with the enormous amount of drops (eth5) has the least amount of traffic - it's just the back-up cable modem we use for ISP redundancy. the only traffic on it is ICMP and tunnel-test.

Wondering if i should cap my 2 core 4000 series boxes at R80.30.

Thanks for any feedback.

Re: core affinity R80.40 - two cores

Timothy_Hall — Wed, 17 Jun 2020 11:39:09 GMT

Please provide output of ethtool -S eth5 and ethtool -S eth1. Although uncommon, sometimes RX-DRPs are caused not by ring buffer misses/overflows, but by incoming frames carrying unknown protocols that have no registered receiver. This reporting behavior seems to have changed in Gaia 3.10 making it more likely than it was for Gaia 2.6.18.

Re: core affinity R80.40 - two cores

D_TK — Wed, 17 Jun 2020 01:54:37 GMT

ethtool -S eth5
NIC statistics:
rx_packets: 124212
tx_packets: 81970
rx_bytes: 12548006
tx_bytes: 9474872
rx_broadcast: 28189
tx_broadcast: 1660
rx_multicast: 9074
tx_multicast: 2
rx_errors: 0
tx_errors: 0
tx_dropped: 0
multicast: 9074
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_csum_offload_good: 65235
rx_csum_offload_errors: 4
rx_header_split: 0
alloc_rx_buff_failed: 0
tx_smbus: 0
rx_smbus: 0
dropped_smbus: 0
rx_dma_failed: 0
tx_dma_failed: 0
rx_hwtstamp_cleared: 0
uncorr_ecc_errors: 0
corr_ecc_errors: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_skipped: 0

ethtool -S eth1
NIC statistics:
rx_packets: 3361399
tx_packets: 3707821
rx_bytes: 1152961595
tx_bytes: 2555983567
rx_broadcast: 195818
tx_broadcast: 26641
rx_multicast: 3054
tx_multicast: 2
rx_errors: 0
tx_errors: 0
tx_dropped: 0
multicast: 3054
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_csum_offload_good: 3150510
rx_csum_offload_errors: 0
rx_header_split: 0
alloc_rx_buff_failed: 0
tx_smbus: 0
rx_smbus: 0
dropped_smbus: 0
rx_dma_failed: 0
tx_dma_failed: 0
rx_hwtstamp_cleared: 0
uncorr_ecc_errors: 0
corr_ecc_errors: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_skipped: 0

Thank You.

Re: core affinity R80.40 - two cores

Timothy_Hall — Wed, 17 Jun 2020 02:07:51 GMT

Yep sure enough these are zero for both interfaces:

rx_no_buffer_count: 0
rx_missed_errors: 0

So there is no problem with ring buffer misses/overruns. This situation was covered in my book:

RX-DRP Culprit 1: Unknown or Undesired Protocol Type

In every Ethernet frame is a header field called “EtherType”. This field specifies the OSI
Layer 3 protocol that the Ethernet frame is carrying. A very common value for this
header field is 0x0800, which indicates that the frame is carrying an Internet Protocol
version 4 (IPv4) packet. Look at this excerpt from Stage 6 of “A Millisecond in the life
of a frame”:

Stage 6: At a later time the CPU begins SoftIRQ processing and looks in the ring
buffer. If a descriptor is present, the CPU retrieves the frame from the associated
receive socket buffer, clears the descriptor referencing the frame in the ring
buffer, and sends the frame to all “registered receivers” which will be the
SecureXL acceleration driver. If a tcpdump capture is currently running,
libpcap will also be a “registered receiver” in that case and get a copy of the
frame as well. The SoftIRQ processing continues until all ring buffers are
completely emptied, or various packet count or time limits have been reached.

During hardware interrupt processing, the NIC driver will examine the EtherType
field and verify there is a “registered receiver” present for the protocol specified in the
frame header. If there is not, the frame is discarded and RX-DRP is incremented.

Example: an Ethernet frame arrives with an EtherType of 0x86dd indicating the
presence of IPv6 in the Ethernet frame. If IPv6 has not been enabled on the firewall (it is
off by default), the frame will be discarded by the NIC driver and RX-DRP incremented.

What other protocols are known to cause this effect in the real world? Let’s take a look
at a brief sampling of other possible rogue EtherTypes you may see, that is by no means
complete:

Appletalk (0x809b)
IPX (0x8137 or 0x8138)
Ethernet Flow Control (0x8808) if NIC flow control is disabled
Jumbo Frames (0x8870) if the firewall is not configured to process jumbo frames

The dropping of these protocols for which there is no “registered receiver” does
cause a very small amount of overhead on the firewall during hardware interrupt
processing, but unless the number of frames discarded in this way exceeds 0.1% of all
inbound packets, you probably shouldn’t worry too much about it. An easy way to
confirm that the lack of a registered receiver is the cause of RX-DRPs is to perform the
following test:

1. In a SSH or terminal window, run watch -d netstat -ni and confirm the constant incrementing of RX-DRP on (interface).

2. In a second SSH session, run tcpdump -ni (interface) host 1.1.1.1

Does the near constant incrementing of RX-DRP on that interface suddenly stop as
long as the tcpdump is still running, and resume when the tcpdump is stopped? If so,
the lack of a registered receiver is indeed the cause of the RX-DRPs. The specified filter
expression (host 1.1.1.1 in our example) does not actually matter, since libpcap will

register to receive all protocols on behalf of the running tcpdump, and then filter the
packets based on the provided tcpdump expression. So as long as the tcpdump is
running, there is essentially a registered received for everything.

But how can we find out what these rogue protocols are, and more importantly figure
out where they are coming from? Run this tcpdump command to show every frame not
carrying IPv4 traffic or ARP traffic based on the EtherType header field:

tcpdump -c100 -eni (iface) not ether proto 0x0800 \
and not ether proto 0x0806 and not stp

(Note the ‘\’ at the end of line 1 of this command is a backslash and allows us to
continue the same command on a new line)

Re: core affinity R80.40 - two cores

D_TK — Wed, 17 Jun 2020 02:40:34 GMT

Thanks Tim. Appreciate your help.