1500 problems with firmware R81.10.08_996001608

[BorisL]

Since we updated this past week to R81.10.08_996001608, we were unable to connect to Checkpoint services. Our Harmony endpoints lost connection and we could not log in to portal.checkpoint.com. All other internet access seemed completely normal. Debugging web page access we saw that some checkpoint sites were reachable and others timed out. We thought it was the internet provider and performed all kinds of tests. Also, we were not able to connect to FW using watchtower. Opened a service request with Checkpoint and they told us "it is your internet provider, not our problem". 

We reverted the 1500 to 996000575. Problem solved. 

Our connection to internet is fiber PPoE with Telefonica. Only particular about this connection (bad but it has been there for a long time) is that first  ip address in traceroute, the fiber default router, is 192.168.x.x. I do not know if this is related to the problem or not. But definitely the firewall was blocking or not routing https responses.

FW has always worked and now works again with 996000575

Has anybody had problems with R81.10.08_996001608?

[G_W_Albrecht]

I have not had any issues with R81.10.08 996001608 and know of no customer that had - so better escalate this with TAC and if needed, also talk to your local CP SE. Can not be that it does work with 996000575 but not with 996001608 !

[BorisL]

Hi. Thanks for response, but we tried everything for days. We had to revert.
No other changes other than the FW version. In my opinion, there is some additional filtering in 996001608 which, in our scenario, blocks some https responses, including blocking watchtower connections. It might be, or not, related to the ISP using some private addresses in their routing. 
I am sorry not to have the time or resources to deal with TAC and will stay with 996000575 for longer, until hopefully the problem pops up elsewhere.

[Tom_Hinoue]

Is your appliance locally managed?

If so, I assume there is a possibility that it can be caused by the Smart Accel feature added since R81.10.05.
I think we saw some issues when the CP1500 WAN internet connection is PPPoE and in combination with Smart Accel, where some services like iCloud is affected and cannot login. We currently have a case open to TAC for this issue, pending their updates.

So you might want to disable Smart Accel (or just disable Check Point services in Smart Accel) if you have a chance to see if it resolves your issue as well.

[BorisL]

Hi Tom.

Yes, locally managed and PPPoE internet connection.

In our R81.10 996000575 version  Smart Accel is completely disabled. I doubt that the upgrade would have changed that setting. 

[Tom_Hinoue]

Hi Boris,

Then I assume the culprit is not Smart Accel then... but did you by any chance disable SecureXL as well?
Though SecureXL for PPPoE is supported in Gaia Embedded (apart from maintrain Gaia), it caused many issues in the past for us.

If SecureXL nor PPPoE (MTU) settings does not resolve your case, then I think your issue needs to be investigated by TAC, as I haven't experienced any issues so far with CP services with PPPoE in R81.10.07/R81.10.08.

[BorisL]

I do not think I have enabled or disabled SecureXL. I do not know where that would be done. If you tell me how to test, I can try off hours. The MTU for our PPoE connection is 1500

The case is that we work normally in current version and have problems when updating firmware.  So there is definitely a change in behaviour of the new version, which is most probably related to our specific environment.

As said before, I have no time or resources to spend with TAC. I have already had a 6 month case open for 1600 SMBs and cannot  go through that again... ;-(

Thanks. 

[Tom_Hinoue]

You can disable SecureXL from WEBUI in Device-> Advanced settings by changing the parameter "Acceleration Settings - Acceleration state enabled" to "false"
Note: SecureXL is preferred enabled at all times

Also I was wondering if the same issue occurs on R81.10.05 and R81.10.07 which was released before R81.10.08 to understand if the issue lies only in the latest R81.10.08 or not, which is also worth testing before opening a ticket to TAC 🙂

[BorisL]

Will test off hours when possible. Thanks.

[RamGuy239]

I think this should be for testing purposes only. All my experience with SMB/Embedded/Quantum Spark, in general, has been that they feature very specialised hardware. Without SecureXL, you are going to cripple your performance tremendously. I've noticed this with Site-2-Site IPsec VPN on SMB in the past. If you walk outside the parameters accelerated through hardware, your IPsec VPN throughput becomes abysmal.

I think this is to be expected with such specialised hardware. Going outside of SecureXL on Gaia (not SMB), while still wasteful in terms of hardware utilisation, will work fine unless you are underspecified on the hardware side, as the X86-based hardware is far more capable of doing things in software.

And here we are talking about disabling SecureXL completely, no less. Not sure how SMB might differ from non-SMB, but on non-SMB, you aren't supposed to disable SecureXL since R80.20 and the re-designed SecureXL. Not sure if this will put the R81.10 SMB gateway in an unsupported state?

[BorisL]

Thanks TamGuy239. Good points. My only objective here would have been to determine if SecureXL had anything to do with the problems we experienced after firmware upgrade, so that Check Point can pinpoint the problem and solve it. I have no intention of using build 996001608  nor deactivating SecureXL in production environment until the problem is found and solved. We cannot dedicate the time and resources Check Point would need from us to test and debug through TAC.

[G_W_Albrecht]

You could

- change admin PW for WebGUI and Embedded clish

- export a backup

- change admin PW for WebGUI and Embedded clish back

- hand the backup over to CP TAC and let them replicate the issue

[G_W_Albrecht]

Specialized HW ? Small footprint it is usually called - no fan used, completely flash-based, ideal for industrial and on-desk deployments (excluding 1600/1800).

Disabling SecureXL is not only supported but sometimes even needed:

sk164793: How to disable SecureXL for specific ports on SMB appliances

sk65015: How to disable SecureXL permanently on SMB appliance

sk167532: Connections through port forwarding or from VPN to 1500 are slow when SecureXL is enabled

[Greg_Harbers]

Hi Boris,

We have deployed R81.10.08 build 1608 on over 60 appliances that previously were on R80.20.50. All our SMB devices are centrally managed via R81.20 mgmt.

The issues we have seen are higher memory usage and clish crashing when "show configuiration" is run.

We have S/Rs open for both issues. For the show configuration issue, yesterday we were provided build  R81.10.08 build 1638. This fixed that issue. When I asked the TAC engineer if any other issues were resolved by this build, he said he does not know any more detail. I am yet to try it on a production device.

As for the memory utilisation issue, I have been told that R&D are working on it.

Additonally we have some anecdotal evidence of people at sites powering off/on the appliances to clear lock up states, we are still trying to get to the bottom of that one.

[starmen2000]

How did you get R81.10.08 build 1638. On website there is only 1608 version for downloading.

Another question, did you upgrade all SMBs through Smart Console? How was your experience? 

[G_W_Albrecht]

As a fix for clish crashing when "show configuiration" is run

[Chris_Atkinson]

Asking via TAC for fixing a specific problem.

Such a process requires R81.20 Mgmt yes.

[Greg_Harbers]

As others have mentioned, we got 1638 in response to a TAC case regarding a clish crash when "show configuration" is run.

Hopefully it will be released to the web site.

Yes, I did use the smartconsole method, worked well if a bit cumbersome in that you need to select all of the gateways to run in batch. I found that if I selected one, hit go, then another and hit go, by the time I got to about 5 jobs running Smartconsole was unusable until they all completed.

Would be good if CPcdt could work with SMB devices

[TJ_Aus]

The new release R81.10.08 Build 996001683 that was published on the 20th of November fixes the "show configuration" issue which I too had witnessed.

[Besilikum]

Hi Boris,

we've also been experiencing issues with HTTPS traffic after updating our 1500 appliances to R81.10.x. It was found as many of our customers sporadically being unable to access HTTPS websites. Attempting to load such pages resulted in timeouts.

After disabling SecureXL via the WebUI using the "Acceleration Settings - Acceleration state enabled" parameter, the problems no longer occur.

I suspect it may be related to the "Smart Accel" feature introduced with R81.10.05 as we didn't encounter this problem with previous versions.

A case has already been created for this, but despite providing a backup, the issue couldn't be replicated. I believe this is because the problem only occurs in correlation with certain providers.

[BorisL]

Hi.

Sorry to hear you are having the same problem. Hopefully now they will acknowledge something is being done differently in R81.10.x

Is your Internet connection using PPoE? If you do a traceroute, do you see in it private addresses outside your network?

[Besilikum]

Hi Boris,

we don't use a direct PPPoE connection. The Check Point is behind a provider-supplied router (most likely from the German Telekom) with a static IP, which it uses as a gateway.

BR

[BorisL]

That would probably mean that between  your 1550 and the router there is a subnet with private addresses.  This might be a similarity to our environment in which the provider (Telefonica) uses private addresses within THEIR network to route our assigned public IP.

[Besilikum]

Exactly. There's a private subnet between the 1550 and the provider router.

It could be possible that the newly introduced "Smart Accel" feature is causing issues with HTTPS traffic in private networks.

Unfortunately, Check Point Support hasn't been very helpful so far, as they can't replicate the issue in their lab, and we can't demonstrate the problem since it occurs sporadically.

However, we have been able to create a video that clearly shows the problem and also shows that it is resolved immediately after disabling the acceleration.

[G_W_Albrecht]

My 1550 with R81.10.08_996001608 also sits behind an ISP router with a 192.168.1.x net between them, but i only have the show config issue.

[BorisL]

PPPoE as well?

When you access https://portal.checkpoint.com/signin do you see the region dropdown?

[G_W_Albrecht]

No PPPoE - that was troublesome even with the old Edges. I see the region dropdown.

[Naama_Specktor]

Hello @Besilikum 

My Name is Naama Specktor and I am Checkpoint employee .

I will appreciate it , if you will share the SR TAC number , here or on PM.

Thanks,

Naama Specktor

[Besilikum]

Hi Naama,

case Number is: SR#6-0003765072

BR

[FGA_Sys_And_Net]

Hello,

Same issue, our case number: SR#6-0003559283

Kind regards,