FTP causes 100% CPU on one core

[Diego_dg]

Hello, we have noticed that uploading some kind of files from the inside network to the internet using FTP causes one of the cores of the appliance to reach 100% during the file transfer. Using cpview (PM-Stats) we see that PM: 183_CMI_ADVANCED_PATTERNS_DATA_ seems to be responsible for this high CPU use, but if we disable the IPS (ips off) the issue remains. The same FTP goes through other checkpoint firewalls that don't show this behaviour. Please, have somebody experienced this issue?

Thanks in advance.

[G_W_Albrecht]

Your issue sounds like sk105411 When transferring a large file via FTP, fw_worker process consumes 100% CPU     

[Diego_dg]

Thanks for your help! yes, the behaviour is very similar, but SYN attack protection is not enabled (and we have a Take newer than the one that fixed the issue)... 

I forgot to especify that the issue only happens with certain kind of files, usually with: Material eXchange Format (MXF) (professional audio and video files)

[G_W_Albrecht]

Then i would compare the configuration of the GWs that do not show the issue with the one that does - maybe differences in TP config ?

[Diego_dg]

Yes, the main difference between both GWs are the blades: the problematic GW has IPS, APP & URLF and TE, the other GW that doesn't show this behaviour has only AV & AB (we moved AV&AB from the other GW to check if this could fix, the issue but it didn't)

We suspect IPS could be the cause, but disabling it (through "ips off" command) doesn't fix the issue

[G_W_Albrecht]

Disabling IPS is not such a good idea - but i would exclude FTP to internal server from all TP (as it is not needed at all) to see if that resolves the issue.

[Diego_dg]

Yes, I agree, we only did it for testing. We already had created an exception for this traffic on IPS and also excluded it on TP and it didn't fix the issue.

[Timothy_Hall]

I'm assuming the core hitting 100% is a Firewall Worker core (kernel instance) and not an SND/IRQ core.

After running "ips off" did you start a completely brand new FTP connection?  If the FTP connection is already established and passing data, running "ips off" then will not have any effect as it is only applied to new connections.

As a test, try disabling APCL/URLF, installing policy and run the FTP transfer again.  If it substantially improves (and I suspect it will) you need to rework your APCL/URLF policy to ensure FTP traffic does not match any rule invoking APCL/URLF.  In an ordered layer this means the FTP traffic "falls off" the end of the APCL/URLF layer without matching any APCL/URLF rule at all.  There is no way to define an explicit APCL/URLF rule to achieve this same effect.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Diego_dg]

Thanks! it seems that the APCL/URLF could be the cause of the issue: we disabled all the rules and the issue didn't appear, we are moving the rules to the other firewall and check the behaviour for a longer period of time.

[Diego_dg]

We moved the APCL/URLF rules to the other GW and disabled the blade on the first one and now the issue is on the second GW, so it seems that this especific format of file and the APCL/URLF blade doesn't deal well together... we would need to find out what APCL/URLF rule is hitting this traffic... 

[G_W_Albrecht]

I would exclude this traffic from TP completely - as you can assume that it is not infected anyway and not downloaded but uploaded, this could make much sense...

ALso you should see the rule hit in logs...

[Diego_dg]

Yes, we have excluded it from TP, but the problem is with some Application Control rule that we still have not identified... and there is no way to configure exceptions on Application Control, as far as I know...

[Vladimir]

Give this a shot:

[Diego_dg]

Thanks! but I think this screen is from R80.10 and we are running R77.30, there is a similar screen on R77.30 but is for IPS exceptions, not for APCTL/URLF

[Diego_dg]

Hi again, we are still making tests (we should not impact on the service, so testing is a slow process...), we have confirmed that the APCTL/URLF is the cause of the issue, but the behaviour is a bit odd: the issue appears even with no rules on the APCTL/URLF policy. I mean, we enabled again the blades with all rules disabled and we noticed that since then, some traffic causes again very high use of CPU on some cores (around 80%)... 

[Diego_dg]

Hi! the issue is fixed (thanks to checkpoint support). Putting the FTP negated service on all the APCTL/URLF rules that this traffic could hit did the trick.

[Timothy_Hall]

So you exposed the Service field of the R77.30 APCL/URLF policy, added FTP to a rule then negated that service, essentially causing a condition where no APCL/URLF rule was matched for FTP connections, correct?

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Diego_dg]

Yes, it's correct. After doing this, ftp transfers don't cause 100% CPU use anymore.