R80.20 gateway showing unusual value of interrupts.

[Muazzam]

Hardware: 13800 with 8/12 split.
OS: R80.20 Take 47.
Traffic type: 95% Web traffic.
Bandwidth: 1Gbps to 5Gbps (at peak times).
Blades Enabled: FW only.
No IPS or IPSec VPN on the box. No https inspection, etc.

The actual issue was high "PSLXL" value that (possibly) causing high CPU for workers.
About 6 weeks ago, besides high PSLXL, there were spikes on the 4 MQ cores. We added one core to MQ and after that the load on 4 cores was divided on 5 (as expected) but no change on workers and PSLXL. Since the last change that involves a reboot, I see unusual number of interrupts. Other firewalls are showing comparable number of interrupts. I have another firewall doing similar function (R77.30 T216) with 95% packets accelerated.

Anyone has seen this behavior and what could be a reason for this? This firewall will be upgraded to take 103 soon but I am curious to find out what is going on?

[Expert@13800]# cpview
|--------------------------------------------------------
| CPVIEW.CPU.Overview
|--------------------------------------------------------
| CPU:
|
CPU User System Idle I/O wait Interrupts
0 0% 43% 57% 0% 168,370
1 0% 50% 50% 0% 9
2 0% 44% 56% 0% 8
3 0% 45% 55% 0% 16
4 0% 44% 56% 0% 11
5 0% 1% 99% 0% 33
6 0% 0% 100% 0% 44
7 26% 38% 36% 0% 50 >> core dedicated to fwd
8 0% 36% 64% 0% 8
9 0% 38% 62% 0% 22
10 0% 48% 52% 0% 30
11 2% 41% 58% 0% 7
12 1% 44% 55% 0% 15
13 2% 42% 56% 0% 25
14 2% 43% 55% 0% 33
15 1% 49% 51% 0% 43
16 2% 41% 57% 0% 24
17 1% 44% 55% 0% 29
18 1% 46% 54% 0% 33
19 1% 43% 56% 0% 3

[Expert@13800]# fwaccel stats -s
Accelerated conns/Total conns : 17353/632756 (2%)
Accelerated pkts/Total pkts : 902351597962/1737655744590 (51%)
F2Fed pkts/Total pkts : 44772565566/1737655744590 (2%)
F2V pkts/Total pkts : 24634357745/1737655744590 (1%)
CPASXL pkts/Total pkts : 0/1737655744590 (0%)
PSLXL pkts/Total pkts : 790531581062/1737655744590 (45%)
QOS inbound pkts/Total pkts : 0/1737655744590 (0%)
QOS outbound pkts/Total pkts : 0/1737655744590 (0%)
Corrected pkts/Total pkts : 0/1737655744590 (0%)

[_Val_]

Anything on IPS? I see passive streaming working, which means you stream something for additional inspection. Could be AVI, AC, IPS, TP.

[Muazzam]

Nothing like these.

Also, note that the PSLXL value of 45% stays the same all the time, even when the traffic volume is very low.

[_Val_]

Which is an additional indication you have something doing streaming. 

Can you post a screenshot of your GW object?

[Timothy_Hall]

To see the attributes of connections that are being specifically handled in PSLXL, run command fwaccel  conns  -f  S.  Are they port 445 (microsoft-ds) connections?

Starting with Gaia kernel 3.10 Multi-Queue is enabled for all interfaces except the management one by default.

[Muazzam]

I ran the command "fwaccel conns -f S" few weeks ago during off-peak hours and it was a over 100MB file with about 900K lines. Now I convert the text file to Excel format and apply filters to check for ports. About 99% of the traffic is either source port 80xx or destination port 80xx (80xx is used for proxy port). This firewall sits between the end user and proxy server. There is was no port 445.

Output of "uname -a" command:

2.6.18-92cpx86_64 #1 SMP Mon Jan 21 09:12:46 IST 2019 x86_64 x86_64 x86_64 GNU/Linux

[Timothy_Hall]

What specific service object is port 80XX matching in your policy, and does it have a protocol type or protocol signature set? 

You may need to create a service object with a "blanked" protocol matching that 80XX traffic, as it sounds like that traffic is matching on a service that is invoking some kind of passive streaming for protocol inspection purposes.  SecureXL can't do passive streaming.

[HeikoAnkenbrand · Accepted solution]

Hi @Muazzam,

I guess that port 80xx is used for a proxy. If this is the case, you can enable fast acceleration. This does not analyze the traffc via PSLXL and the fast path is used.

Medium path (PSLXL) - The SecureXL layer passes the packet to one of the CoreXL FW instances to perform the processing. When Medium Path is available, TCP handshake is fully accelerated with SecureXL. Rulebase match is achieved for the first packet through an existing connection acceleration template. SYN-ACK and ACK packets are also fully accelerated. However, once data starts flowing, to stream it for Content Inspection, the packets will be now handled by a FWK instance. Any packets containing data will be sent to FWK for data extraction to build the data stream. RST, FIN and FIN-ACK packets once again are only handled by SecureXL as they do not contain any data that needs to be streamed. This path is available only when CoreXL is enabled.

Packet flow when the packet is handled by the SecureXL device, except for:

• IPS (some protections)
• VPN (in some configurations)
• Application Control
• Content Awareness
• Anti-Virus
• Anti-Bot
• HTTPS Inspection
• Proxy mode
• Mobile Access
• VoIP
• Web Portals.

Fast Accelerator - The Fast Acceleration (sk156672) feature (green) lets you define trusted connections to allow bypassing deep packet inspection on R80.20 JHF103 and above gateways. This feature significantly improves throughput for these trusted high volume connections and reduces CPU consumption.

The CLI of the gateway can be used to create rules that allow you to bypass the SecureXL PSLXL path to route all connections through the fast path. 

Example Usage:

fw ctl fast_accel add 1.1.1.1 2.2.2.0/24 80 6
fw ctl fast_accel delete 192.168.0.0/16 any 8080 17
fw ctl fast_accel add 255.168.240.0/20 255.0.0.0/8 1503 any
fw ctl fast_accel show_table
fw ctl fast_accel enable
fw ctl fast_accel disable

More to fast acceleration and PSLXL read here: 
- R80.x - Security Gateway Architecture (Logical Packet Flow)
- R80.x - Security Gateway Architecture (Logical Packet Flow) - Update R80.20+
- R80.x - Security Gateway Architecture (Content Inspection)

[Muazzam]

Thank you Tim and HeikoAnkenbrand

Port 80xx is used for web proxy. The object in the firewall has "None" for protocol.
Exactly the same policy is applied to another cluster in different physical location, same hardware, slightly less traffic but it is on version R77.30-T216. That firewall is currently showing 93% accelerated packets, 6% F2F and 0% PSL. Workers are around 12% and interrupts for most cores are similar. I feel that this is an issue with version R80.20-T47. Our next step would be to upgrade this to T103 (or later) and if this does not completely help, we will look into "Fast Accelerator".

[HeikoAnkenbrand]

Hi @Muazzam 

I quite agree with you!
R80.20 HFA 47 is very bugy from my point of view. I would update to a newer HFA.

[Timothy_Hall]

I kind of doubt upgrading to the latest GA Jumbo HFA for R80.20 will solve this particular problem but it is worth a try.  You'll definitely need to do that first, as the fast_accel feature was not added to R80.20 until Jumbo HFA 103. Note that using fast_accel essentially whitelists traffic through the accelerated path, and not all inspection called for in your security policy will occur.  However you are only using the Firewall blade so you should be OK for the most part.

The issue is probably related to the big SecureXL changes introduced in R80.20 but it is tough to know for sure.