Issue - R77.30 to R81.10 - No Connectivity + Security Policy blocks appliance WebUI and SSH

[Jesus_ESCOLAR]

Dear All,

I have migrated a CP4200 R77.30 to a CP6200 R81.10, all migration steps and checks were 100% successful.

• All objects and rules were migrated correctly.
• Two interfaces: eth1 (used for Internal) and mgmt (used for External/Internet).
• IP's, DNS, masks, etc --> All rebuilt the same.

However, when migrating the physical cables, there's no connectivity of any type: no internet, nothing passes through the firewall.

Also, when the security profile (policy) is applied in the appliance, the WebUI and SSH access to the appliance are no longer accessible.

• I remove the profile via fw unloadlocal and then the connectivity to the WebUI and SSH works again.
• I checked for potential blocking rules, but can't see to find one.

Question 1: What Am I doing wrong? What have I missed? May be the routing table is broken? But why the policy blocks the SSH and WebUI then?

Question 2: How can I see in the CLI what rule blocked the traffic? Best tool to open and read the blocking logs?

I have to work on the CLI at the moment, via the LOM interface, because otherwise I need to disconnect the old production firewall to plug the new one in order to access it via SmartConsole.

So any blocked log locations, log reading/parsing tools, and CLI commands to see what happened would be appreciated 😀.

Regards,

/JE

[Vladimir]

When you are saying that you have "migrated successfully", are you referring to replicating Gaia configuration only?

Can you confirm that SIC is working between management and the gateway?

Can you verify that the topology settings of the gateway object are correct?

Can you verify that you have changed the appliance model and version to 6200 and R81.10?

Is policy installation on the gateway succeeding and you are losing the ability to connect to it via WebUI and SSH afterwards?

Is the workstation from which you are attempting to connect located on the same internal network as the internal interface of the gateway or is that connection being routed to it?

Is your management server on the same internal network as the internal interface of the gateway or is it too, routed to it?

Simple example diagram (with fake public IPs)  could be helpful.

[Jesus_ESCOLAR]

It is a standlone appliance with both management and security gateway roles.

Answers to your questions:

When you are saying that you have "migrated successfully", are you referring to replicating Gaia configuration only?

JE: full successful export of the database and successful import in R80.40, then migration to R81.10, also successful, no errors presented.

Can you confirm that SIC is working between management and the gateway?

JE: Being in the same appliance, the SIC is working fine.

Can you verify that the topology settings of the gateway object are correct?

JE: They are as they were in the previous appliance, and discover topology matched the interfaces as they were.

Can you verify that you have changed the appliance model and version to 6200 and R81.10?

JE: Yes, all represented correctly.

Is policy installation on the gateway succeeding and you are losing the ability to connect to it via WebUI and SSH afterwards?

JE: Policy installation is successful and just after deployment that we loose ability to connect to WebUI and SSH, but we can use the SmartConsole to access the appliance, modify the policy and deploy it. 

Is the workstation from which you are attempting to connect located on the same internal network as the internal interface of the gateway or is that connection being routed to it?

JE: Yes, same flat network: 10.1.0.0/16

Is your management server on the same internal network as the internal interface of the gateway or is it too, routed to it?

JE: It is an standalone appliance, so both management and security gateway are in the same hardware,

Simple example diagram (with fake public IPs)  could be helpful.

JE: Simple diagram in ASCII here 😉

10.1.1.253/16 <-eth1->[CP6200]<-Mgmt->84.x.x.x/29 <-> Colt Router <-> The Internet

Note: eth1 assigned as the management interface, obviously.

Not sure what is wrong here, but all expert help would be highly appreciated.

[Vladimir]

I would suggest creating an explicit rule on top of your security policy allowing SSH2 and HTTPS access from your management workstation to the standalone appliance, if one does not already exist. Reload the policy and try connecting again.

There is a possibility that the setting allowing SSH and WebUI access was configured in Gaia on the old appliance and it was not replicated to the 6200 using migrate server tools.

[Jesus_ESCOLAR]

Dear @Vladimir ,

I'll do that tomorrow, but now I do have new issues:

- The routing table is not being replicated, therefore there's no Internet connectivity.

I have tried adding the static-route entries via the CLISH but they make no effect in the routing, nothing changes, no routing is happening.

This is really very frustrating because for such a simple migration, the number of issues is really massive.

What can be the source of no connectivity when both devices seem to have exactly the same confirguration?

[Vladimir]

@Jesus_ESCOLAR , did you get a chance of comparing the Gaia configuration between old and new devices?

When you are alluding to routing table, are you talking about static routes or the dynamic routes?

If there are connectivity issues related to routing, it makes sense to check the routing and arp entries on adjacent routers/switches- I have encountered a few instances where arp cache on upstream routers was not expiring and had to be cleared by ISPs.

One more place to look at is the networking/topology properties of the gateway object to verify that the interfaces are properly defined, their topology reflecting reality and the anti-spoofing configured as desired.

[Jesus_ESCOLAR]

Yes @Vladimir ,

And policies are the same. However, I'll ask Colt to clear the arp entries in their router, and make sure the arp is cleared in the local switch.

Topology was rediscovered after migration, interfaces reassigned, names matching interfaces and policies, relfecting the reality of the network, for the anti-spoofing, I'll have a look tomorrow and update every single configuration, one by one, and count how many where not converted by default, which could be very frustrating in cases where both appliances must have the same Mgmt IP address.

[Vladimir]

@Jesus_ESCOLAR , some other things to check and consider for connectivity issues:

1. can you ping the upstream router from Check Point appliance?

2. Did you get a chance to compare Gaia configs of old and new appliances?

3. Did you check the default gateway route presence on the new appliance?

[Jesus_ESCOLAR]

Hello @Vladimir ,

1. can you ping the upstream router from Check Point appliance?

Yes.

2. Did you get a chance to compare Gaia configs of old and new appliances?

Yes, all the same.

3. Did you check the default gateway route presence on the new appliance?

Yes, all included.

For information:

- From the appliance the DNS resolutions to Internet are OK.

- But any attempt to connect to the update URL's from Check Point fails, no update possible from the GAIA WebUI, it fails.

- DNS resolutions? 100% OK!

I have already a case opened but nobody seems to understand the issue complexity.

[PhoneBoy]

Please send me the relevant TAC SR in a PM.

[PhoneBoy]

Did you perform a policy install after performing the upgrade?
This is required when doing any version upgrades.

[Hugo_vd_Kooij]

Indeed something that gets forgotten a lot.

However the initial policy should allow at least HTTPS (port 443) and SSH (port22). If you move those to different ports you will have no access. I have tripped myself at least once into a lockout due to the move to different ports.

[Jesus_ESCOLAR]

Was done, WebUI is in 4443, SSH stays in 22 as nobody internally needs to use the 22 port for anything else.

Here the main problem is that when we plug the new appliance, no http or https from any endpoint works anymore, all goes to 404 or other errors, but DNS works 100% fine.

Weird!

[Jesus_ESCOLAR]

Hi @PhoneBoy & @Hugo_vd_Kooij ,

I suspect through naivity that policy install is the only policy install available in the SmartConsole, isn't it?

What I did first was the database install, then the policy deploy and the policies appeared in the SmartConsole GUI.

I checked via expert mode that there was a policy installed matching the existing unique policy in the database.

There's a rule to allow SSH and WebUI (it's the second rule in the list) and although it's ok, nothing happens, all is blocked, including basic Internet browsing.

I can't find the source of the issue and both issues (No Internet/Nothing passing through the firewall, and with the policy no WebUI or SSH) are not really in the same space.

Any ideas?

[PhoneBoy]

The ways to install policy:

• SmartConsole
• Relevant API command (can be issued via CLI): https://sc1.checkpoint.com/documents/latest/APIs/index.html#cli/install-policy~v1.8%20

The only time a gateway will install a different policy is:

• On initial install (InitialPolicy)
• When there is no valid license (DefaultFilter)

You can get a little more details on what policy is actually installed on the active gateway by using the tools described here: https://community.checkpoint.com/t5/General-Topics/Show-Ruleset-and-Objects-on-the-Gateway-Emergency... 

You might see what's actually dropping traffic to the gateway by using fw ctl zdebug +drop.

[Hugo_vd_Kooij]

I know it can be a little more tricky. We have an outstanding issue with 3000 appliances that may kick in the wrong policy if you instal a jumbo hotfix. The customer is all over the world so we need to schedule this with someone near the unit that can help us out if needed as we get locked out of the unit.

We had a bunch of tickets open for this over the last few years. Only on the last ticket we got conformation that this might be a more systematic issue and there are more customers having the same issue.

I have done Jumbo Hotfix updates on 2 location (each with a 3200 cluster) and our failsafe work. In /etc/rc.local we now have a line `fw fetch localhost`and that did save the day. But  updating a cluster of 3200's takes about 3 hours here. The machine needs 15 to 20 minutes after reboot to get back into a healthy cluster state.

Which is why I know things can be complicated then advertised. We are looking into putting serial console equipment in there as a failsafe. But with 20 clusters it will add up to the costs significantly.

[Hugo_vd_Kooij]

Coming to think of it I find a move from R77.30 to R81.10 a rather bold move. I would do this in stages myself.. R773.0 to R80.10 (welcome databases), then R80.40 (welcome SecureXL changes) then R81.10 (target reached).

The commmand `fw stat` should tell you which policy is active. The fist policy install always kills existing SSH connections in my experience. But restarting the ssh connection usually works. 

If you think the policy is right you can use `fw ctl zdebug drop` to see if the drops make sens to you.

But in the end the forum is there to exchange ideas and this looks like you need more support then a forum might provide.

[Jesus_ESCOLAR]

Absolutely, policy is 100% installed and running.

[Ilya_Yusupov]

Hi @Jesus_ESCOLAR,

In order to understand the case i suggest to check cpview history by running "cpview -t" press t and check at the time frame where you lost ssh and webui connectivity, i would like to understand if the policy at that time had a defaultfilter policy or the correct one.

can you please check that and verify that it's not a policy issue?

Thanks,

Ilya 

[Jesus_ESCOLAR]

Hello @Ilya_Yusupov ,

The policy seems to be exactly the same in the 4200 R77.30 and in the 6200 R81.10.

WebUI and SSH were corrected already.

We have lost since the beginning any http and https traffic. Nothings goes through for the endpoints.

But the device does DNS and any other protocol resolutions with no issue.

We ping all gateways to the internet and the corporate router.

Not sure why is this happening.

We even create an ACCEPT ALL test rule on top and it doesn't works.

[Ilya_Yusupov · Accepted solution]

Hi all,

Just updating the thread that the issue resolved post our today remote session.

We saw that the customer had 2 default routes which one of them were configured as "logical" route.

The OS choose this route to be as default which cause to the connectivity issue, once we disable the logical route the issue resolved.

@Jesus_ESCOLAR  - thank you very much for you time today, as i mention in our meeting, feel free to contact me directly in any case.

Thanks,

Ilya 

[_Val_]

Great job, @Ilya_Yusupov !