Low bandwidth when checkpoint is connected

[Sangisha_Daka_K]

Hello checkmates,

I have issue with checkpoint firewall R80.10 gaia Os ,we receive 12 mb from the ISP but when you connect checkpoint 

firewall its reducing to 3.75 mb we tried to disable some blades but still didnt resolve the issue and we also engaged and had 3 hours session with checkpoint support engineers  but that didnt solve any thing as they mentioned that we need to enable secure xl but again when we enable that we loose internet therefore what could be an issue and below is the report from the checkpoint support team.

As I promised, here are my detailed notes on what we did today, including environment, troubleshooting steps and next possible steps.

╔═════════════╗
║ ENVIRONMENT
╚═════════════╝
 One firewall and one management Server, running R80.10
╔═════════════════╗
║ BUSINESS IMPACT / SEVERITY
╚═════════════════╝
 Medium
╔═══════╗
║ ISSUE
╚═══════╝
 Network speed drags when CheckPoint is introduced to the network.
╔═════════════════╗
║ TROUBLESHOOTING
╚═════════════════╝
Very low bandwith, when the CheckPoint is connected.

Before the CheckPoint, the bandwith was 10.86 and after adding the CheckPoint, the bandwith became 3.57.
Currently, the speed is 3.72.
The device is a 5000, running R80.10.
The enabled blades are General, Application Control, URL Filter, IPS and Anti-Bot.

Entered the firewall via Putty session.
Ran the command fw ctl zdebug drop | grep 172.16.0.87
No drops were observed.

Entered SmartDashboard.
Disabled the IPS blade as a test, since it has been known to cause problems with traffic in the past.

Attempted to push policy.
The Application Control and URL Filter have expired contratcs, that can not be fetched.
This is a possible issue.

Speed test after disabling the IPS blade is 4.34

Returned to SmartDashboard.
Opened the Policy and then the threat Prevention Policy.
Created an exception, which disabled Anti-Bot and IPS on the test PC (172.16.0.87).
The speed test was 4.79.

Disabled the Application Control and URL Filter as another test.
Installation of policy failed because Rule 9 contained Application Control.
Disbaled Rule 9.
Pushed policy.

Speed was 4.75.

Returned to SmartDashboard.
Disabled all blades but the General.
Pushed policy.
The speed remained 4.79

Returned to the Putty session.
Ran the command fw monitor -T -p all -e "accept host (172.16.0.87);"
No blade holds onto the packets for too long.

Could this be an interface issue?

The interface, leading to the internet is eth3.
Ethtool eth3 shows the interface is full duplex and with speed of 1000Mb/s.
Auto negociation is on.
Turned it off for testing purposes with the command ethtool -s eth3 autoneg off.

Speed test was 2.19.

Returned to the Putty session.
Ran the command top.
There is nothing unusual - the memory works well and the 3 CPUs are idle most of the time.

Ran the command cpinfo -y all.
The firewall is on Take 112.
There are newer takes but we would only upgrade if absolutely necessery.

Ran the command fwaccel stat.
SecureXL is DISABLED.
Enabled it with the command fwaccel on.

Enabling SecureXL stops the Internet.
Customer stopped SecureXL with the command fwaccel off.
Ran the command cpconfig.
SecureXL is enabled.
Ran the command fw accel stat.
Here, SecureXL is disabled.

Entered SmartDashboard.
Opened Logs and Monitoring.
There is no dropped traffic from the time of the issue.

Ran the command fw ctl affinity -l -r -v -a.
We see the following output:
CPU 0 at eth1,
CPU 1 at fw_2
CPU 2 at fw_1
CPU 3 at fw_0

A good next step to do is to install the newest Jumbo Hotfix Accumulator.

Provided the customer with Take 161, which we downloaded from the Support Center.

Entered the Gaia WebUI.
Opened CPUSE and then Status and Action.
Clicked on Upload package and uploaded the provided jumbo - Take 161.
Take 161 was successfully uploaded.

Right-clicked the Take 161 and tried to install the Take.
The Take can not install since we can not UNinstall Take 121 because of the SMACK Take.
Uninstalled the Smack Take.
Successfully managed to install Take 161.

However, on top of the Gaia WebUI's CPUSE, there is a message: "Your currently installed license is not entitled to receive udpates from Check Point Download Center."

Entered the firewall via Putty session.
Ran the command cplic print -x.
All the contracts on the box have expired.

Entered SmartDashboard.
Opened Manage licenses and packages.
Chose File - Licenses and Contracts - Contracts - Update contracts from the User Center.
Customer put in his username and password.

Internet is still down.
Entered the firewall via Putty session.
Entered cpconfig.
Disabled SecureXL.

Internet is back up but the speed is around 4.10.
╔════════════╗
║ NEXT STEPS
╚════════════╝
There is some trouble, since SecureXL brings the Internet down.
Secure XL should NOT be interfering with the Internet, it should be increasing speed.

Possible next steps include:
- Start a remote session.
- Note the time of the session.
- Enabled SecureXL. Be preapred for Internet to go down.
- Ask customer to run the command fw ctl zdebug drop | grep [IP], as well as top and tcpdump on eth3.
- All of this would tell us why exactly Secure XL is bringing the Internet down.

- This is a SecureXL issue.

[Chris_Atkinson]

Working with TAC is definitely the best way to progress troubleshooting the issue described here.

Please ensure they're aware of the type of internet connection (PPPoE etc) and verify speed/duplex of connections to any downstream switches or network devices.

[Jerry]

disable QoS rules  and do not overload your box with blades like IPS and URLF or APPC. Then check the speed.

what are the resources you've allocated to that VM or ... is it an Appliance? What model?

[Chris_Atkinson]

5000 appliance without QoS enabled according to the above details provided.

[Jerry]

oh sorry overlooked details below, indeed.

of PPoE I'd definitely take a good care of the MTU firest

[Alessandro_Marr]

Install take 142 and review your SND and multiqueue.

[Vladimir]

Please post the results of the "show interface eth#" statistics here for both, internal and external interfaces.

Rerun the speedtest and note if the error counters incremented and let us know the outcome.

What kind of equipment is connected to the internal and external interfaces of the gateway?

[Alessandro_Marr]

Please, run a "top" too and post. Your MTU settings is default? 1500

[Timothy_Hall]

Low performance here is almost certainly an interface issue, please post output of netstat -ni.  Also as advised earlier, make sure MTU is consistent on all interfaces.

Latency vs. Loss: Running a continuous ping during a speed test, is there packet loss or high latency?  If the former, probably an interface issue that will be revealed by netstat -ni.

SecureXL breaking Internet connectivity is generally indicative of something seriously broken in your network setup/config, and can also be caused by use of PPPoE or MTU issues, see:

sk61221: Issues requiring adjustment of the Maximum Segment Size (MSS) of TCP SYN and TCP SYN-ACK pa...

Also wondering if the firewall hardware itself has a problem, please provide output of cpstat -f sensors os

--

CheckMates Break Out Sessions Speaker

CPX 2019 Las Vegas & Vienna - Tuesday@13:30

[Jerry]

 told him, MTU on ext. interface is most of the time the thing we do not pay enough attention to when connecting PPPoE to the WAN uplinks.

1500 is overkill for PPPoE therefore decreasing it to 1460 may or may not help, unless some other sensors as you've mentioned Tim interfere with the device circumstances.

[Jerry]

here is an example from my 5600 CXL (2018 Appliance):

Temperature Sensors
----------------------------------------------
|Name       |Value|Unit   |Type       |Status|
----------------------------------------------
|Intake Temp|40.00|Celsius|Temperature|     0|
|Outlet Temp|42.00|Celsius|Temperature|     0|
|CPU Temp   |43.00|Celsius|Temperature|     0|
----------------------------------------------

Fan Speed Sensors
--------------------------------------
|Name       |Value  |Unit|Type|Status|
--------------------------------------
|System Fan1|6250.00|RPM |Fan |     0|
|System Fan2|5818.50|RPM |Fan |     0|
|System Fan3|5443.50|RPM |Fan |     0|
--------------------------------------

Voltage Sensors
---------------------------------
|Name |Value|Unit|Type   |Status|
---------------------------------
|VCore|1.74 |Volt|Voltage|     0|
|+12V |11.98|Volt|Voltage|     0|
|3.3V |3.31 |Volt|Voltage|     0|
|VDIMM|1.50 |Volt|Voltage|     0|
|+5V  |5.07 |Volt|Voltage|     0|
|VBAT |3.12 |Volt|Voltage|     0|
---------------------------------

***

compare with yours

ifconfig of the WAN interface please and show us your MTU on it.

UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:113156077 errors:0 dropped:0 overruns:0 frame:0
            TX packets:93581556 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:84691993709 (78.8 GiB)  TX bytes:32822951109 (30.5 GiB)

[Timothy_Hall]

Yep in regards to the cpstat -f sensors os command, 0=good, anything else=bad.  The classic one from a performance perspective I've seen is a failed CPU fan at 0 rpm, and subsequent massive CPU downclocking to keep the processor from literally bursting into flames.  Very difficult to figure out why things are so darn slow in that case.  🙂

--

CheckMates Break Out Sessions Speaker

CPX 2019 Las Vegas & Vienna - Tuesday@13:30

[Sangisha_Daka_K]

[Alex_Rozhko]

Besides MTU mentioned earlier did you consider hard-coding speed/duplex config on interfaces and verify same on adjacent routers/switches ? Can you run speed test through GW via interface not connected to ISP?

[Alex_Rozhko]

What was inline before CP appliance? who's the ISP provider? Is there requirements from ISP for modem/router models? what is termination point?

[Van-N]

Hello,

I know that this thread has not had any activity in some time, but I was wondering if this issue was ever resolved. If so, what was the root cause?

I have an environment that seems to be experiencing the same issue. There are no interface errors and MTU is 1500 everywhere and there are no interface errors. We have applied Jumbo Hotfix and still experience the issue. Any assistance is appreciated. Thanks!

[Vladimir]

@Van-N , please describe the equipment seating between your gateway's external interface and the ISP.

Make, model, OS.

I have seen, in smaller environments, specifically with cable modems, occasional need to power-cycle those to regain nominal performance.

As I did not have access to those devices and they are managed by ISPs, no reasonable explanation to this behavior was ever found.

Bandwidth was definitely affected and has immediately recovered to full rated capacity after reboot of the ISP equipment.

Just for kicks, when you are experiencing problems, try to connect a host directly to ISP equipment with the same IP as your gateway and run the tests from it.

Regards,

Vladimir 

[Van-N]

Two Checkpoint 5100 appliances running Gaia R80.20 in cluster.  Connected to Cisco ISR 4300 running 16.6.5, which is connected to ISP device.

Have not had a chance to run checks on the ISP device (power cycle or direct connect) as things are in production now and I am remote so cannot do the physical changes to check those.

[Tim_McColgan]

This is still a major issue with Checkpoint to this day, and is rearing it's ugly head during Covid. Checkpoint is doing something to drastically decrease the bandwidth when remote access clients connect with VPN client. I am on version e83.20 and have experience this for many years. TAC still can't solve it.