R80.20 Managing R80.10 Gateway - CPU Increase

[Jason_Carrillo]

After upgrading our MDS and MLM to R80.20, we are seeing average CPU increases across all of clusters after pushing policy to the gateways. It has become very apparent on one of our firewalls that is licensed for four cores.  The four gateways I am seeing this on are open hardware. I do have a pair of appliance based clusters that do not seem to have any increase in CPU. All gateways running 80.10.

I've opened a ticket with Check Point, but they haven't heard about anything like this and frankly, didn't seem too interested in getting to the bottom of it. 

Has anyone else seen this out there?

[Kaspars_Zibarts]

Very interesting! Keep us updated! That is very important for rest of us think of upgrades..

[Timothy_Hall]

Sounds like you may have some kind of license "core crunch" (as I call it in my book) on your open hardware, where numerous Firewall Worker instances stomp each other competing for the same physical core.  What does "fw ctl affinity -l -r" show?

The other possibility is that with the revamp of SecureXL in R80.20, the SND/IRQ cores that typically handle SecureXL operations may be trying to do much more processing than they were before and are hitting max utilization.  What does the CPU utilization look like on cores designated SND/IRQ vs. Firewall Worker under R80.20?  You may need to adjust your core split...

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[PhoneBoy]

The SecureXL changes in R80.20 shouldn't impact the gateways since they're still on R80.10 (management on R80.20).

[Timothy_Hall]

Missed that point, thanks Dameon.  License core crunch should not be occurring either with only a change in management version.  Would be interested to see output of enabled_blades on the R80.10 firewalls experiencing higher CPU under R80.20 management; can't see how a management version change only would significantly alter how the gateway uses CPU although there are quite a few Threat Prevention enhancements in R80.20.  Seeing what blades are enabled would be a good start as well as the output from the "Super Seven" commands...

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Jason_Carrillo]

Thanks for the input Dameon and Tim.

We are only running FW, VPN, Identity and Monitor blades.

I know it does sound odd, and while I will be the first person to tell someone correlation does not equals causation, the upgrade and subsequently policy pushes matches on all four firewalls.

We have reverted though and while CPU is better, it still seems a bit higher than normal.

What are these seven commands you speak of?

Sent via the Samsung Galaxy S7, an AT&T 4G LTE smartphone

[Timothy_Hall]

See here:

TechTalk: Security Gateway Performance Optimization with Tim Hall 

The "Super Seven" are:

• fwaccel stat
• fwaccel stats -s
• grep -c ^processor /proc/cpuinfo (and maybe /sbin/cpuinfo too)
• fw ctl affinity -l -r
• netstat -ni (and possibly sar -n EDEV)
• fw ctl multik stat
• cpstat os -f multi_cpu -o 1 (and maybe free -m)

With the output of those commands, generally I can determine about 90% of the time where the bottleneck is located; the presentation shares some tips about how to interpret them yourself, as well as some real-world Case Studies.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Jason_Carrillo]

Thanks Timothy. I have attached a text document with the output.

[Timothy_Hall]

I assume these outputs were taken on an R80.10 gateway under R80.20 management with the high CPU issue?  Thanks for the outputs, I'd like to see the output of fwaccel stat and enabled_blades as well.

Given that 81% of the traffic is fully accelerated in SXL and 71% is being templated, I find it a little odd that your two SND/IRQ cores are running about 90% idle, but your six worker cores are in the low 70s for idle time handling PXL/F2F which comprises a mere 17% of traffic.  That CPU load distribution seems a little strange given that path mix of traffic, and I'm wondering if there are a large number of SecureXL violations occurring, so please also provide output of fwaccel stats -p.

Everything else so far looks clean and normal, and I'm still having a really hard time understanding how R80.20 management could be possibly causing this behavior on a R80.10 gateway.  It might come down to:

• Push a policy to this R80.10 gateway from an R80.10 SMS/CMA
• Grab copies of compiled policy files from gateway's $FWDIR/state/__tmp/FW1 directory
• Push the exact same policy to this R80.10 gateway from an R80.20 SMS/CMA
• Grab copies of compiled policy from gateway's $FWDIR/state/__tmp/FW1 directory
• Have a "diff" command party on all the files figuring out the differences

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[RickHoppe]

Perhaps you can use s7pac. It’s based on Tim Hall’s presentation on the Super Seven.

[_Val_]

Very peculiar. The compilation code on R80.20 MGMT for GW on R80.10 should not change. Without any policy changes it is supposed to be exactly the same policy hence the same performance parameters. 

Considering you are saying it is still "higher than normal" event after rollback, could it be just a case of an unlucky coincidence?

[SantiagoPlatero]

Hi all, I'm facing a similar scenario: virtual R80.20 managing a R80.10 (Take 121) open server gateway (a Dell R710 but with only 2 core licensed), but the issue came across when I use the unified policy feature and inline layers (that is, security and application rulebase in one access control rulebase + TP + QoS). The high CPU usage is so problematic that the gateway's LAN interface responds with 2000+ ms delay, when it doesn't respond at all.

So, as the LAN interface packet losses, I've to do a "fw unloadlocal" ('cause SIC fails) and push the old, lookalike pre-R80 rulebase: security rulebase in one layer, application rulebase in another one + TP + QoS. Now the gateway works just fine, with the same rules, same IPS/TP activations, but a different approach on rule matching (ye olde one).

I have almost all blades enabled (only have disabled Policy Server for VPN and DLP), NGTX is enabled but not used on the TP rulebase. Threat Emulation is done in the CP's cloud and only applies for MTA traffic.

In the unified policy scenario I have some rules with standard old fashion TCP/UDP ports and application objects in the service/application column, I think I read some SK regarding this scenario and high CPU usage?

Also, the only rulebases I didn't consolidate in the unified policy where the Mobile Access rulebase (still using the old legacy one) and, of course, the HTTPS inspection rulebase (is activated but bypassed for general internet browsing for functionality issues).

I've opened a ticket to my partner as I've collaborative support contract (and can't contact TAC directly myself), but maybe the OP problem has something to do with my issue?

Thanks dudes.

[Timothy_Hall]

The common thread here seems to be an open hardware gateway that is licensed for less cores than physical cores, which generally can't happen on a Check Point gateway appliance since they are always licensed for the number of actual physical cores.  Don't see how a policy install from a R80.20 SMS could change the number or physical core distribution of Firewall Workers and cause a "core crunch" as described in my book. 

There were quite a few changes in SecureXL for R80.20, but generally the SMS is not really SecureXL-aware so it is hard to see how a policy installation could impact the SND/IRQ side of things on an R80.10 gateway in this situation. Perhaps that has changed in R80.20 but I doubt it, and I would find it even more doubtful that even if something was changed in this area it would end up being pushed/propagated to an older R80.10 gateway by the SMS.

My gut tells me it is probably higher CPU on the Firewall Workers or some kind of load-balancing issue there (Dynamic Dispatcher perhaps?) due to something in the pushed policy.  Santiago's report of ~2000 ms response times for pings would seem to track with this theory, since ICMP is never accelerated by SecureXL and must be handled by the Firewall Workers. 

Until we start seeing some "Super Seven" outputs though I'm not going to speculate further. 

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[SantiagoPlatero]

Hi Timothy, thanks for your feedback. Do you need the "Super Seven" be executed when the issue is in place right? That be, for my scenario, with the unified policy package pushed to the problematic gateway?

Just FYI, I don't have SecureXL enabled (I asked in another topic cause the output of fwaccel stat is there is no license for SecureXL and far as I know there isn't need for a specific license for it), but I have CoreXL enabled for two licensed cores.

[Timothy_Hall]

Yes the issue should be in place before running the Super Seven. 

Not sure why you don't have a license for SecureXL, all reasonably modern licenses should have it (although maybe that only applies to Check Point appliances, can't remember).  Might just need to regenerate your license in the User Center to obtain SecureXL functionality.

Jason Carrillo‌ is SecureXL enabled on your open hardware firewalls?

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Jason_Carrillo]

Yep, though it did turn off on one after the upgrade to management and policy push. Not sure if it flapped on its own or if it was a result of the push though. The spike in CPU definitely correlated with the push though.?

[Timothy_Hall]

There should be a CPU spike when policy is pushed to any gateway, then it settles back down once complete.  I find it interesting that both you and Santiago Platero‌ have SecureXL disabled on an open hardware firewall that is licensed for less cores than  physically present, and are experiencing excessive CPU usage when managing your R80.10 gateways from R80.20.  SecureXL was overhauled significantly in R80.20 gateway, although I don't see how that change would impact a R80.10 gateway being managed by R80.20.

SecureXL can't really be "switched off" completely in R80.20  the way it could be in R80.10 earlier; that is probably just a coincidence here but interesting nonetheless.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[SantiagoPlatero]

Hi Timothy Hall‌, below you will find the super seven's output:

1. fwaccel stat:

# fwaccel stat

Accelerator Status : no license for SecureXL

Accelerator Features : Accounting, NAT, Cryptography, QOS, HasClock,
                       Templates, Synchronous, IdleDetection,
                       Sequencing, TcpStateDetect, AutoExpire,
                       DelayedNotif, TcpStateDetectV2, WireMode,
                       DropTemplates, NatTemplates, Streaming,
                       AntiSpoofing, Nac, ViolationStats, AsychronicNotif,
                       ERDOS, NMR, NMT, NAT64, GTPAcceleration,
                       SCTPAcceleration
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
                        3DES, DES, CAST, CAST-40, AES-128, AES-256,
                        ESP, LinkSelection, DynamicVPN, NatTraversal,
                        EncRouting, AES-XCBC, SHA256‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

2. fwaccel stats -s

# fwaccel stats -s

No statistics given‍‍‍

3.  grep -c ^processor /proc/cpuinfo (and maybe /sbin/cpuinfo too)

# grep -c ^processor /proc/cpuinfo 

4

# sbin/cpuinfo

HyperThreading=disabled‍‍‍‍‍‍‍

4. fw ctl affinity -l -r

# fw ctl affinity -l -r

CPU 0:  eth0 eth1 eth2 eth3
        fw_1
CPU 1:  fw_0
CPU 2:
CPU 3:
All:    pdpd fgd50 pepd in.emaild.mta usrchkd lpd in.acapd fwpushd scrub_cp_file_convertd fwucd rtmd fwd vpnd rad scanengine_b mpdaemon scrubd cp_file_convertd in.emaild.smtp cpd cprid
The current license permits the use of CPUs 0, 1 only.
‍‍‍‍‍‍‍‍‍

5. netstat -ni (and possibly sar -n EDEV)

# netstat -ni 
Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0 153112258    138  23877      0 133773526      0      0      0 BMRU
eth1       1500   0   805181      0      0      0   824074      0      0      0 BMRU
eth2       1500   0 91527294      0   1329      0 96134009      0      0      0 BMRU
eth3       1500   0 45153840      0 149064      0 35913655      0      0      0 BMRU
eth3.2     1500   0        0      0      0      0        0      0      0      0 BMRU
lo        16436   0  2179030      0      0      0  2179030      0      0      0 LRU
‍‍‍‍‍‍‍‍‍

# sar -n EDEV
Linux 2.6.18-92cpx86_64 (<truncated>)        11/05/18

00:00:01        IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
00:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:10:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:10:01         eth3      0.00      0.00      0.00      0.18      0.00      0.00      0.00      0.00      0.00
00:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:20:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:20:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:20:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:20:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:20:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:20:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:30:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:30:01         eth3      0.00      0.00      0.00      0.05      0.00      0.00      0.00      0.00      0.00
00:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:40:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:40:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:40:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:40:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:40:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:40:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:50:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
00:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:00:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:00:01         eth3      0.00      0.00      0.00      0.62      0.00      0.00      0.00      0.00      0.00
01:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:10:01         eth0      0.00      0.00      0.00      0.20      0.00      0.00      0.00      0.00      0.00
01:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:10:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:30:02           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:30:02         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:30:02         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:30:02         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:30:02         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:30:02       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:40:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:40:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:40:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:40:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:40:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:40:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:50:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:00:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:00:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:10:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:10:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:20:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:20:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:20:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:20:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:20:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:20:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:30:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:30:01         eth3      0.00      0.00      0.00      0.37      0.00      0.00      0.00      0.00      0.00
02:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:40:02           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:40:02         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:40:02         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:40:02         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:40:02         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:40:02       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:50:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:00:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:00:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:10:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:10:01         eth3      0.00      0.00      0.00      0.91      0.00      0.00      0.00      0.00      0.00
03:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:20:02           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:20:02         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:20:02         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:20:02         eth2      0.00      0.00      0.00      0.20      0.00      0.00      0.00      0.00      0.00
03:20:02         eth3      0.00      0.00      0.00     11.17      0.00      0.00      0.00      0.00      0.00
03:20:02       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:30:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:30:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:40:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:40:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:40:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:40:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:40:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:40:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:50:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
03:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:00:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:00:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:10:02           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:10:02         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:10:02         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:10:02         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:10:02         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:10:02       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:20:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:20:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:20:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:20:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:20:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:20:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:40:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:40:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:40:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:40:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:40:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:40:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:50:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:00:01         eth0      0.01      0.00      0.00      0.00      0.00      0.00      0.01      0.00      0.00
05:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:00:01         eth3      0.00      0.00      0.00      0.23      0.00      0.00      0.00      0.00      0.00
05:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:10:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:10:01         eth3      0.00      0.00      0.00      0.33      0.00      0.00      0.00      0.00      0.00
05:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:20:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:20:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:20:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:20:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:20:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:20:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:30:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:30:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:40:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:40:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:40:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:40:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:40:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:40:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:50:01         eth3      0.00      0.00      0.00      0.04      0.00      0.00      0.00      0.00      0.00
05:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:00:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:00:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:10:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:10:01         eth3      0.00      0.00      0.00      0.03      0.00      0.00      0.00      0.00      0.00
06:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:20:03           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:20:03         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:20:03         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:20:03         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:20:03         eth3      0.00      0.00      0.00      1.06      0.00      0.00      0.00      0.00      0.00
06:20:03       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:30:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:30:01         eth3      0.00      0.00      0.00      0.56      0.00      0.00      0.00      0.00      0.00
06:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:40:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:40:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:40:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:40:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:40:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:40:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:50:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:00:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:00:01         eth3      0.00      0.00      0.00      0.18      0.00      0.00      0.00      0.00      0.00
07:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:10:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:10:01         eth3      0.00      0.00      0.00      0.19      0.00      0.00      0.00      0.00      0.00
07:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:20:05           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:20:05         eth0      0.00      0.00      0.00      0.02      0.00      0.00      0.00      0.00      0.00
07:20:05         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:20:05         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:20:05         eth3      0.00      0.00      0.00      0.16      0.00      0.00      0.00      0.00      0.00
07:20:05       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:30:01         eth0      0.00      0.00      0.00      0.24      0.00      0.00      0.00      0.00      0.00
07:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:30:01         eth3      0.00      0.00      0.00      0.06      0.00      0.00      0.00      0.00      0.00
07:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:40:02           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:40:02         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:40:02         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:40:02         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:40:02         eth3      0.00      0.00      0.00      0.56      0.00      0.00      0.00      0.00      0.00
07:40:02       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:50:03           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:50:03         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:50:03         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:50:03         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:50:03         eth3      0.00      0.00      0.00      0.15      0.00      0.00      0.00      0.00      0.00
07:50:03       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:00:01         eth0      0.00      0.00      0.00      0.16      0.00      0.00      0.00      0.00      0.00
08:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:00:01         eth3      0.00      0.00      0.00      0.88      0.00      0.00      0.00      0.00      0.00
08:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:10:50           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:10:50         eth0      0.00      0.00      0.00      4.67      0.00      0.00      0.00      0.00      0.00
08:10:50         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:10:50         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:10:50         eth3      0.00      0.00      0.00      6.54      0.00      0.00      0.00      0.00      0.00
08:10:50       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:20:57           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:20:57         eth0      0.00      0.00      0.00      9.09      0.00      0.00      0.00      0.00      0.00
08:20:57         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:20:57         eth2      0.00      0.00      0.00      1.25      0.00      0.00      0.00      0.00      0.00
08:20:57         eth3      0.00      0.00      0.00     10.78      0.00      0.00      0.00      0.00      0.00
08:20:57       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:30:10           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:30:10         eth0      0.00      0.00      0.00      6.06      0.00      0.00      0.00      0.00      0.00
08:30:10         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:30:10         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:30:10         eth3      0.00      0.00      0.00      6.43      0.00      0.00      0.00      0.00      0.00
08:30:10       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:50:01         eth0      0.00      0.00      0.00      2.12      0.00      0.00      0.00      0.00      0.00
08:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:50:01         eth3      0.00      0.00      0.00      2.13      0.00      0.00      0.00      0.00      0.00
08:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:00:01         eth0      0.00      0.00      0.00      9.56      0.00      0.00      0.00      0.00      0.00
09:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:00:01         eth2      0.00      0.00      0.00      0.49      0.00      0.00      0.00      0.00      0.00
09:00:01         eth3      0.00      0.00      0.00     11.05      0.00      0.00      0.00      0.00      0.00
09:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:10:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:10:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:20:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:20:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:20:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:20:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:20:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:20:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:30:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:30:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:40:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:40:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:40:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:40:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:40:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:40:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:50:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:00:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:00:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:10:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:10:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:20:02           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:20:02         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:20:02         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:20:02         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:20:02         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:20:02       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:30:01         eth0      0.01      0.00      0.00      0.00      0.00      0.00      0.01      0.00      0.00
10:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:30:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:40:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:40:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:40:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:40:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:40:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:40:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:50:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:00:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:00:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:10:01         eth0      0.00      0.00      0.00      0.09      0.00      0.00      0.00      0.00      0.00
11:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:10:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:20:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:20:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:20:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:20:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:20:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:20:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:30:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:30:01         eth3      0.00      0.00      0.00      0.37      0.00      0.00      0.00      0.00      0.00
11:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:40:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:40:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:40:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:40:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:40:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:40:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:50:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:00:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:00:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:20:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:20:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:20:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:20:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:20:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:20:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:30:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:30:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:40:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:40:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:40:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:40:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:40:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:40:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:50:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:50:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:50:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:50:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:50:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:50:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:00:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:00:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:00:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:00:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:00:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:00:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:10:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:10:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:10:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:10:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:10:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:10:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:20:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:20:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:20:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:20:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:20:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:20:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:30:01           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:30:01         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:30:01         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:30:01         eth2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:30:01         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:30:01       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:40:52           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:40:52         eth0      0.00      0.00      0.00      3.25      0.00      0.00      0.00      0.00      0.00
13:40:52         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
13:40:52         eth2      0.00      0.00      0.00      0.08      0.00      0.00      0.00      0.00      0.00
13:40:52         eth3      0.00      0.00      0.00      2.52      0.00      0.00      0.00      0.00      0.00
13:40:52       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth0      0.00      0.00      0.00      0.46      0.00      0.00      0.00      0.00      0.00
Average:         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth2      0.00      0.00      0.00      0.02      0.00      0.00      0.00      0.00      0.00
Average:         eth3      0.00      0.00      0.00      0.73      0.00      0.00      0.00      0.00      0.00
Average:       eth3.2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

6. fw ctl multik stat

# fw ctl multik stat
ID | Active  | CPU    | Connections | Peak
----------------------------------------------
 0 | Yes     | 1      |        2078 |     2893
 1 | Yes     | 0      |        1966 |     2979‍‍‍‍‍

Aaaand 7. cpstat os -f multi_cpu -o 1 (and maybe free -m)

# cpstat os -f multi_cpu -o 1 



Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
|   1|           1|            93|           6|      94|        ?|          1627|
|   2|           0|            93|           7|      93|        ?|          1627|
|   3|           2|            77|          21|      79|        ?|          1627|
|   4|           4|            41|          55|      45|        ?|          1627|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
|   1|           1|            93|           6|      94|        ?|          5219|
|   2|           1|            55|          44|      56|        ?|          5220|
|   3|           7|            27|          66|      34|        ?|          5220|
|   4|           6|             2|          92|       8|        ?|          5221|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
|   1|           0|           100|           0|     100|        ?|          2131|
|   2|           0|            61|          39|      61|        ?|          2131|
|   3|           1|            45|          54|      46|        ?|          2131|
|   4|           2|            24|          74|      26|        ?|          2131|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
|   1|           0|           100|           0|     100|        ?|          2131|
|   2|           0|            61|          39|      61|        ?|          2131|
|   3|           1|            45|          54|      46|        ?|          2131|
|   4|           2|            24|          74|      26|        ?|          2131|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
|   1|           0|           100|           0|     100|        ?|          1251|
|   2|           0|            98|           2|      98|        ?|          1251|
|   3|           1|            87|          11|      89|        ?|          1251|
|   4|           0|            50|          49|      51|        ?|          1251|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
|   1|           0|           100|           0|     100|        ?|          1251|
|   2|           0|            98|           2|      98|        ?|          1251|
|   3|           1|            87|          11|      89|        ?|          1251|
|   4|           0|            50|          49|      51|        ?|          1251|
---------------------------------------------------------------------------------‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

# free -m

             total       used       free     shared    buffers     cached
Mem:      16289576    8228732    8060844          0     402668    4027332
-/+ buffers/cache:    3798732   12490844
Swap:     17575100          0   17575100
‍‍‍‍‍‍

Hope you find it useful, thanks a lot!

[Timothy_Hall]

Based on those outputs: due to the licensing restrictions there is no SecureXL; both cores are serving both SND/IRQ and Firewall Worker functions while core 1 is just a Firewall Worker (I think -  core 1 may also be trying to act as a SND/IRQ too but without automatic interface affinity provided by SecureXL no interfaces can ever be assigned to it for handling).  Both of them are getting hammered about the same as far as low idle times, which would seem to indicate that it is indeed an issue on the Firewall Worker side of things as I was guessing earlier.  Hard to tell though because there is no SecureXL here and everything is forced F2F (which causes more CPU utilization), and interface SoftIRQs can only be handled on core 0 at all times (no automatic interface affinity) which is causing 0.33% RX-DRPs on eth3.  Dynamic Dispatcher seems to be working OK and your firewall has plenty of RAM.

Would be interesting to apply an eval license and then enable SecureXL (with no reboot), but still keep CoreXL with 2 instances on 2 cores and see if things improve substantially.  If not then try setting 3 CoreXL instances and rebooting the system into the default 1/3 split for a 4-core system (with the eval license and SecureXL turned on of course), then test again.  The lack of SecureXL is really muddying the water here.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[SantiagoPlatero]

Thanks a lot! I also believe is something regarding SecureXL, but the weird thing about it is the problem is presented only when I push a classic unified policy. I think (or maybe read it somewhere, dunno) the UP feature maybe works closely with SecureXL and CoreXL ones.

In other thread, regarding the SecureXL licensing issue, I asked this and also I want to ask it here (in case I've no answer there): does anyone knows from which blade/license/subscription is SecureXL tied to? I can't find documentation regarding to this. I believe it must be ADN (Advanced Networking), which I have licensed in this gateway ('cause if I haven't it I won't be able to use QoS) so, theoretically, I should be able to enable SecureXL.

I'm creating an evaluation license to enable SecureXL and give it a try as you suggest, keep you on the loop.

Thanks again!

[Ilya_Yusupov]

Hi all,

We checked the SXL license issue in our lab and indeed once we have Open Server with License that limit cores SXL is disabled with No license.

i'm taking it with local RnD and update once we will have a fix.

[SantiagoPlatero]

Thanks a lot Ilya, looking forward for the resolution!

[18568]

Is there a SK to follow up on this critical issue? We were planning R80.20 management upgrade in the next few days but we have only open servers (some of them with licenses limiting the cores) so we'll have to postpone it.

Edit: Just noticed the issue was only figured out today so I guess no SK yet but it would still be good to quickly add it to the known R80.20 limitations as I was lucky to see this thread before the migration but other people might not.

[PhoneBoy]

I'm sure we will create an SK on this in the near future.

It probably won't be added to the Known Limitations as we do plan to fix this before the next version. 

[Ilya_Yusupov]

Hi,

After i digging into this issue i found out that for SXL you will need one the following sku's:

1. cpsb-adnc

2. cpsb-accl

:this is also written in ClusterXL admin guide

if i will find more documents that mention it i will share.

[Ilya_Yusupov]

Hi Jason,

We would like to assist here but we will need you to provide the following:

- in clish output of show asset all

- cat /proc/interrupts

- output of fw ctl multik stat

- output of fwaccel stats -s

- output of fw ctl affinity -l -r

You may send me an email with all those outputs to iliay@checkpoint.com

[Jason_Carrillo]

Hey guys, sorry for going dark on this. I've been working pretty much only on this and haven't had much a chance to update what is going on. 

We are currently working with Diamond TAC on this.

We reverted back to R80.10 on management, but the problem has not subsided. Here is what we see all the time now after the upgrade:

I can go back using CPVIEW (amazing tool btw) and look at the day before the upgrade and CMI applications rarely gets above 0.1% and SXL notifications may get up to 1% or so. Below is what is typical before the upgrade on the current firewall and after we added some cores to our previously four core firewall:

Spikes in these two components seems to correspond to FW_worker CPU increases, sometimes pushing our CPU to above 80%.

Right now, I have three out of four firewalls that just aren't having a very good time of it. There don't appear to be any services impacted at this time, but I don't want to let this sit as is. Something is very wrong with our gateways and everything points at the upgrade + policy push. 

[Jason_Carrillo]

I forgot to mention that 1 out of our 4 gateways is fine after we installed an 8 core license and rebooted. And when I mean fine, I mean when I made the change, I had to double check that there was actual traffic going through the wall. That is how low the CPU was.

We had rebooted the gateway before and that didn't help, but this reboot did help, which both TAC and I found very odd. We are planning a reboot for all the other gateways, probably tonight.

[Timothy_Hall]

Thanks for the update Jason, I posted my reply to the Super Seven outputs above before seeing this and noted strangely elevated CPU usage on the Firewall Worker cores given the relatively small amount of traffic they were handling.  Your findings here seem to track with that analysis and some prior speculation in a Nov 5th post I wrote.  Very odd.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Jason_Carrillo]

TAC was equally surprised about what we are seeing. We just finished a scheduled reboot of our two 8-core gateways and the issue did not resolve. We are still seeing abnormally high CPU on our workers even while only passing 150Mbps of traffic. My home router doesn't work as hard as these gateways are for no apparent reason.

We are a little bit behind on our patch level, so that is the next step for us. Hoping that knocks something loose.

?