Local connectivity issue with R81.10.17

[nmelay2]

For this specific customer, after upgrading centrally managed 1530 firewalls from R81.10.10 to R81.10.17 (build 653), locally initiated network connections fail.
IPS/AV/AMW updates fail, and any curl_cli command from the gateway fails.
No drops in logs.
fw monitor and tcpdump show that connections are actually not being initiated at all.
I can't remember whether "fw ctl zdebug drop" showed anything.
Reinstalling the policy from the management did not fix the issue.

I did the upgrade remotely on off hours, without access to any device on the local networks, so I can't tell whether local connectivity was affected as well.
Same issue with four 1530 firewalls, on four different sites, same management server, same policy.
I rolled back to R81.10.10 for now, and everything is back to normal.

The funny thing is I did upgrade a fifth new 1530 from R80.20 to R81.10.17 just a week before, for the same customer, and it worked flawlessly. It's the only 1530 running R81.10.17 currently, without any reported issue.
As far as I can tell, the only difference is it did not go through the previous R81.10.x releases.

Management server is running R82 JHF 34.

And I've also been running R81.10.17 on a centrally managed 1570 for another customer without issue.

Did someone else encounter something like this?

[Lesley]

You picked the last GA build version so that is good. The step from R81.10.10 towards R81.10.17 is small so I don't see issues there. 

I think it well be best to try to reproduce the issue if possible. From there we can help more. Maybe simple mistake was made with packet capture. Because it sounds unlikely if you ping or curl the box is not doing anything. It could happen but I have never seen it 🙂 

Maybe the internet interface was down? But then you had to connect to the unit via VPN tunnel via a different interface? Maybe topology would help 

[Chris_Atkinson]

Agree definitely need to reproduce the issue so that TAC can aid troubleshooting / investigation.

Otherwise if you're in a position to test / change variables there are newer builds available per:

R81.10.17 Build 996004654 for 1500 Appliances - for DigiCert (sk183884)
R81.10.17 Build 996004708 for 1500 Appliances - for LDAPS with Windows Server 2025

[nmelay2]

Thanks for the suggestion Chris, but I'll play it safe here.
These 1530 gateways are in foreign countries around the globe, and I don't feel like jumping on a R81.10.17 world tour 2025. 😄

[the_rock]

Definitely worth TAC case, specially considering it works for a different client.

Andy

[nmelay2]

OK, I gave it another try.

Traffic is definitely being dropped. This is what I see after the upgrade:

@;54658;[cpu_2];[fw4_2];fw_log_drop_ex: Packet proto=6 <GATEWAY IP>:49814 -> <REMOTE IP>:80 dropped by fw_first_packet_xlation Reason: NAT rulematch failed;
@;56165;[cpu_1];[fw4_1];fw_log_drop_ex: Packet proto=6 <GATEWAY IP>:49814 -> <REMOTE IP>:80 dropped by fw_first_packet_xlation Reason: NAT rulematch failed;
@;59496;[cpu_0];[fw4_0];fw_log_drop_ex: Packet proto=6 <GATEWAY IP>:49814 -> <REMOTE IP>:80 dropped by fw_first_packet_xlation Reason: NAT rulematch failed;

So for some reason, NAT is misbehaving.

I also spotted another issue in Internet connection settings: I'm not able to make any change there.
When I hit the Save button, the Advanced tab is highlighted in red, and a red cross appears on the right, which when hovered says "Errors were found in other tabs".
And in the Advanced tab, as far as I can tell, each and every setting is set to its default value, and trying to fiddle with them goes nowhere.

Last but not least, when I look at the gateway settings in SmartConsole, the topology table is completely messed up.
Some default interfaces are listed instead of the real ones, with IP settings showing as "N/A". Only the WAN interface is displayed properly.
And if I try to refresh the interfaces, LAN interfaces do come up clean, but the WAN interface goes away!

When I look at network interfaces locally, everything looks fine.
Only the Internet settings seem to be broken, or at least cannot be altered in any way.

So to sum it up, the updated firewall seems to be unable to properly read and report its network settings, and this has a wide impact, notably breaking NAT.

Back to R81.10.10 again.
And sure, I'll bring this to TAC when I feel brave enough. 😉

Internet connection settingsFake topologyTopology update

[Lesley]

Have you seen? https://support.checkpoint.com/results/sk/sk163772

What is in the advanced tab btw? 

Note after an upgrade some features are enabled / disabled see: https://support.checkpoint.com/results/sk/sk183153

[nmelay2]

Yes I read about sk163772, but it does not seem to be related.

Advanced tab has settings for port speed/duplex, QoS, SDWAN, and a bunch of other stuff.
Again nothing that really seems related to the issue.

Features disabled by the upgrade probably do not apply to centrally managed gateways.

There are not many settings I can tinker with on a centrally managed gateway anyway.
What could probably fix the issue is switching to locally managed, and back to centrally managed.
Some part of the central policy must be computed/transformed locally, and not properly applied after the upgrade.
Just reinstalling the policy also did not fix anything by the way.

[the_rock]

Are you able to get interfaces without topology?

Andy

[nmelay2]

Yes, but as I said the WAN interface goes away when doing so, and the refreshed interface list does not seem to be saved anyway, I'm back to the "fake" interfaces as soon as I close and reopen the gateway.
(Hmm, the topology view for gaia embedded gateways might actually be totally dynamic, and only used for viewing, not sure about this).

[the_rock]

Im not SMB expert by any means, but I recall once working with customer that had few of those appliances (before they bought regular Gaia ones) and we were always able to do get interfaces without topology, so to me, logically, if that does not work for you, either something is wrong with that interface on OS level or possibly rougint issue?

Just my thinking...

Andy