topic Re: 1500 problems with firmware R81.10.08_996001608 in Spark Firewall (SMB)

1500 problems with firmware R81.10.08_996001608

BorisL — Tue, 24 Oct 2023 07:15:30 GMT

Since we updated this past week to R81.10.08_996001608, we were unable to connect to Checkpoint services. Our Harmony endpoints lost connection and we could not log in to portal.checkpoint.com. All other internet access seemed completely normal. Debugging web page access we saw that some checkpoint sites were reachable and others timed out. We thought it was the internet provider and performed all kinds of tests. Also, we were not able to connect to FW using watchtower. Opened a service request with Checkpoint and they told us "it is your internet provider, not our problem".

We reverted the 1500 to 996000575. Problem solved.

Our connection to internet is fiber PPoE with Telefonica. Only particular about this connection (bad but it has been there for a long time) is that first ip address in traceroute, the fiber default router, is 192.168.x.x. I do not know if this is related to the problem or not. But definitely the firewall was blocking or not routing https responses.

FW has always worked and now works again with 996000575

Has anybody had problems with R81.10.08_996001608?

Re: 1500 problems with firmware R81.10.08_996001608

G_W_Albrecht — Tue, 24 Oct 2023 07:39:09 GMT

I have not had any issues with R81.10.08 996001608 and know of no customer that had - so better escalate this with TAC and if needed, also talk to your local CP SE. Can not be that it does work with 996000575 but not with 996001608 !

Re: 1500 problems with firmware R81.10.08_996001608

BorisL — Tue, 24 Oct 2023 07:46:19 GMT

Hi. Thanks for response, but we tried everything for days. We had to revert.
No other changes other than the FW version. In my opinion, there is some additional filtering in 996001608 which, in our scenario, blocks some https responses, including blocking watchtower connections. It might be, or not, related to the ISP using some private addresses in their routing.
I am sorry not to have the time or resources to deal with TAC and will stay with 996000575 for longer, until hopefully the problem pops up elsewhere.

Re: 1500 problems with firmware R81.10.08_996001608

Tom_Hinoue — Tue, 24 Oct 2023 09:30:24 GMT

Is your appliance locally managed?

If so, I assume there is a possibility that it can be caused by the Smart Accel feature added since R81.10.05.
I think we saw some issues when the CP1500 WAN internet connection is PPPoE and in combination with Smart Accel, where some services like iCloud is affected and cannot login. We currently have a case open to TAC for this issue, pending their updates.

So you might want to disable Smart Accel (or just disable Check Point services in Smart Accel) if you have a chance to see if it resolves your issue as well.

Re: 1500 problems with firmware R81.10.08_996001608

BorisL — Tue, 24 Oct 2023 09:42:22 GMT

Hi Tom.

Yes, locally managed and PPPoE internet connection.

In our R81.10 996000575 version Smart Accel is completely disabled. I doubt that the upgrade would have changed that setting.

Re: 1500 problems with firmware R81.10.08_996001608

Tom_Hinoue — Tue, 24 Oct 2023 10:48:32 GMT

Hi Boris,

Then I assume the culprit is not Smart Accel then... but did you by any chance disable SecureXL as well?
Though SecureXL for PPPoE is supported in Gaia Embedded (apart from maintrain Gaia), it caused many issues in the past for us.

If SecureXL nor PPPoE (MTU) settings does not resolve your case, then I think your issue needs to be investigated by TAC, as I haven't experienced any issues so far with CP services with PPPoE in R81.10.07/R81.10.08.

Re: 1500 problems with firmware R81.10.08_996001608

BorisL — Tue, 24 Oct 2023 11:14:25 GMT

I do not think I have enabled or disabled SecureXL. I do not know where that would be done. If you tell me how to test, I can try off hours. The MTU for our PPoE connection is 1500

The case is that we work normally in current version and have problems when updating firmware. So there is definitely a change in behaviour of the new version, which is most probably related to our specific environment.

As said before, I have no time or resources to spend with TAC. I have already had a 6 month case open for 1600 SMBs and cannot go through that again... ;-(

Thanks.

Re: 1500 problems with firmware R81.10.08_996001608

Tom_Hinoue — Wed, 25 Oct 2023 07:16:19 GMT

You can disable SecureXL from WEBUI in Device-> Advanced settings by changing the parameter "Acceleration Settings - Acceleration state enabled" to "false"
Note: SecureXL is preferred enabled at all times

Also I was wondering if the same issue occurs on R81.10.05 and R81.10.07 which was released before R81.10.08 to understand if the issue lies only in the latest R81.10.08 or not, which is also worth testing before opening a ticket to TAC 🙂

Re: 1500 problems with firmware R81.10.08_996001608

BorisL — Thu, 26 Oct 2023 08:07:07 GMT

Will test off hours when possible. Thanks.

Re: 1500 problems with firmware R81.10.08_996001608

RamGuy239 — Thu, 26 Oct 2023 14:24:31 GMT

I think this should be for testing purposes only. All my experience with SMB/Embedded/Quantum Spark, in general, has been that they feature very specialised hardware. Without SecureXL, you are going to cripple your performance tremendously. I've noticed this with Site-2-Site IPsec VPN on SMB in the past. If you walk outside the parameters accelerated through hardware, your IPsec VPN throughput becomes abysmal.

I think this is to be expected with such specialised hardware. Going outside of SecureXL on Gaia (not SMB), while still wasteful in terms of hardware utilisation, will work fine unless you are underspecified on the hardware side, as the X86-based hardware is far more capable of doing things in software.

And here we are talking about disabling SecureXL completely, no less. Not sure how SMB might differ from non-SMB, but on non-SMB, you aren't supposed to disable SecureXL since R80.20 and the re-designed SecureXL. Not sure if this will put the R81.10 SMB gateway in an unsupported state?

Re: 1500 problems with firmware R81.10.08_996001608

BorisL — Thu, 26 Oct 2023 15:10:24 GMT

Thanks TamGuy239. Good points. My only objective here would have been to determine if SecureXL had anything to do with the problems we experienced after firmware upgrade, so that Check Point can pinpoint the problem and solve it. I have no intention of using build 996001608 nor deactivating SecureXL in production environment until the problem is found and solved. We cannot dedicate the time and resources Check Point would need from us to test and debug through TAC.

Re: 1500 problems with firmware R81.10.08_996001608

Greg_Harbers — Thu, 26 Oct 2023 21:15:12 GMT

Hi Boris,

We have deployed R81.10.08 build 1608 on over 60 appliances that previously were on R80.20.50. All our SMB devices are centrally managed via R81.20 mgmt.

The issues we have seen are higher memory usage and clish crashing when "show configuiration" is run.

We have S/Rs open for both issues. For the show configuration issue, yesterday we were provided build R81.10.08 build 1638. This fixed that issue. When I asked the TAC engineer if any other issues were resolved by this build, he said he does not know any more detail. I am yet to try it on a production device.

As for the memory utilisation issue, I have been told that R&D are working on it.

Additonally we have some anecdotal evidence of people at sites powering off/on the appliances to clear lock up states, we are still trying to get to the bottom of that one.

Re: 1500 problems with firmware R81.10.08_996001608

starmen2000 — Mon, 30 Oct 2023 11:06:15 GMT

How did you get R81.10.08 build 1638. On website there is only 1608 version for downloading.

Another question, did you upgrade all SMBs through Smart Console? How was your experience?

Re: 1500 problems with firmware R81.10.08_996001608

G_W_Albrecht — Mon, 30 Oct 2023 12:59:43 GMT

As a fix for clish crashing when "show configuiration" is run

Re: 1500 problems with firmware R81.10.08_996001608

Chris_Atkinson — Mon, 30 Oct 2023 14:14:33 GMT

Asking via TAC for fixing a specific problem.

Such a process requires R81.20 Mgmt yes.

Re: 1500 problems with firmware R81.10.08_996001608

G_W_Albrecht — Tue, 31 Oct 2023 08:08:03 GMT

You could

- change admin PW for WebGUI and Embedded clish

- export a backup

- change admin PW for WebGUI and Embedded clish back

- hand the backup over to CP TAC and let them replicate the issue

Re: 1500 problems with firmware R81.10.08_996001608

Greg_Harbers — Tue, 31 Oct 2023 08:29:40 GMT

As others have mentioned, we got 1638 in response to a TAC case regarding a clish crash when "show configuration" is run.

Hopefully it will be released to the web site.

Yes, I did use the smartconsole method, worked well if a bit cumbersome in that you need to select all of the gateways to run in batch. I found that if I selected one, hit go, then another and hit go, by the time I got to about 5 jobs running Smartconsole was unusable until they all completed.

Would be good if CPcdt could work with SMB devices

Re: 1500 problems with firmware R81.10.08_996001608

G_W_Albrecht — Tue, 31 Oct 2023 09:04:43 GMT

Specialized HW ? Small footprint it is usually called - no fan used, completely flash-based, ideal for industrial and on-desk deployments (excluding 1600/1800).

Disabling SecureXL is not only supported but sometimes even needed:

sk164793: How to disable SecureXL for specific ports on SMB appliances

sk65015: How to disable SecureXL permanently on SMB appliance

sk167532: Connections through port forwarding or from VPN to 1500 are slow when SecureXL is enabled

Re: 1500 problems with firmware R81.10.08_996001608

Besilikum — Tue, 31 Oct 2023 13:23:02 GMT

Hi Boris,

we've also been experiencing issues with HTTPS traffic after updating our 1500 appliances to R81.10.x. It was found as many of our customers sporadically being unable to access HTTPS websites. Attempting to load such pages resulted in timeouts.

After disabling SecureXL via the WebUI using the "Acceleration Settings - Acceleration state enabled" parameter, the problems no longer occur.

I suspect it may be related to the "Smart Accel" feature introduced with R81.10.05 as we didn't encounter this problem with previous versions.

A case has already been created for this, but despite providing a backup, the issue couldn't be replicated. I believe this is because the problem only occurs in correlation with certain providers.

Re: 1500 problems with firmware R81.10.08_996001608

BorisL — Tue, 31 Oct 2023 14:33:27 GMT

Hi.

Sorry to hear you are having the same problem. Hopefully now they will acknowledge something is being done differently in R81.10.x

Is your Internet connection using PPoE? If you do a traceroute, do you see in it private addresses outside your network?