Allow Management over VPN

[PSushko]

We have an IPSEC tunnel setup between our Perimeter81 gateway and our on-prem R80.40 NGFWs. When I try to connect to the management server, or any gateway, my connection fails.

I have seen the following threads, but they appear to be for R77.30 and R80.10:

• https://community.checkpoint.com/t5/Management/Exclude-CPM-traffic-from-implied-rules/td-p/3934
• https://community.checkpoint.com/t5/Management/Exclude-CPM-Traffic-from-Implied-Rules/m-p/9187

Is the process the same for R80.40, and are there any security implications for including management traffic into the VPN tunnel?

[PhoneBoy]

Oh, you mean two Check Point gateways? 🙂 (In case you didn't hear, Check Point acquired Perimeter 81)

The process should be more or less the same.
The main issue with doing this is if the VPN goes down, you will not be able to manage your gateway.
Recovery from this situation could also be tricky as well.

[PSushko]

Yeah, I heard the great news 🎉.

So if I allow management of our on-prem NGFWs from the VPN gateway, if the tunnel is down, we can no longer manage the gateways? Is there any way to allow management of our on-prem resources both via VPN and on-prem even if the tunnel is down?

[G_W_Albrecht]

You have to be behind the R80.40 Cluster and have internal access to the SMS when VPN is down.

[PSushko]

So long as I am still on the network, I can still manage the cluster even if the tunnel is down correct?

[PhoneBoy]

Seeing as it's disabling the implied rules that allows management through the VPN in the first place, the only way to allow for that is to re-enable those rules and push policy to the relevant gateway.
To ensure the gateway can be managed without VPN in this situation, there will need to be explicit rules to permit the necessary traffic WITHOUT VPN (i.e. the VPN column should be "Any").
If you misconfigure any of this and get locked out of your remote gateway, it will most likely require access to the console to resolve.

[PSushko]

From what I understand:

• If I have a rule very high up that explicitly allows access from NET-X to the management server, and gateways on any VPN
• And I then also include the traffic going to the management server and gateways in the tunnel.

I would be able to manage the gateway and management both from the VPN tunnel, and from NET-X?

[PhoneBoy]

The problem is that when you use "Any" in the VPN field of a rule, it also matches any VPN community configured.
Which means the local gateway will try to send the management traffic over the VPN...which will fail if the remote end is down.
The only way to "fix" this is by re-enabling the implied rules, which won't use VPN.
That requires a policy recompile/push to the local gateway.
Bottom line: it's not really an "either or" proposition.

[PSushko]

So what would be a way that we could manage our on-prem system while away without losing our ability of interacting with it on premise?

[PhoneBoy]

The SIC over VPN discussion relates to the traffic between the gateway and management only.

It's not clear we're talking about the same thing, so let's take a step back to the beginning.
You said: "When I try to connect to the management server, or any gateway, my connection fails."
How are you connecting?
From where, exactly?
A simple diagram showing all the relevant components will go a long way in helping to understand exactly what the issue is and where it might need to be addressed.

[PSushko]

Accessing the management fails when I am off-premises connected to our P81 gateway.

If I am on premises, inside the NAT, I can connect to management just fine.

I'm not very good at creating network maps 😅, but hope this (very rudimentary illustration) helps to understand our issue.

The rectangle is our internal network... forgot to label that.

What we want to do is to remove that big red ✖️.

[G_W_Albrecht]

Use a RA VPN client to gain access to the SMS is the S2S VPN is down.

[PhoneBoy]

This is definitely outside the realm of SIC over VPN.
Do you see the relevant traffic reaching the P81 gateway?
And what methods are you using for access? (SSH, SmartConsole, ?)

[PSushko]

I can see the traffic getting to our on-prem gateway.

So far only SmartConsole, I'll try SSH.

[PhoneBoy]

SmartConsole uses TCP 18190, 19009, and 443.
All of that should work fine through Perimeter 81.
In any case, have you checked what happens beyond your gateway?
tcpdump or fw monitor would be exceptionally useful to see where things are going wrong.

[PSushko]

Not the best versed in the usage of tcpdump and fw monitor. What would the commands I need to run look like (roughly)?

[PhoneBoy]

If you're not familiar with these tools, the following website will help with syntax (courtesy of @Grave_Rose ) https://tcpdump101.com 

[Grave_Rose]

[edit: I think I misunderstood the original question but have cleared it up in my next post. Leaving this here 'cause everyone likes to learn PCaps.]

Wall of text incoming. 🙂

Since this is going across a VPN, you'll likely want to use 'fw monitor' and, if you're using https://tcpdump101.com you'll want to select the "New -F" version since you're on R80.40. Also, we're going to pretend that your Management/SMS IP address is 10.20.30.40, the Internal IP of your P81 is 192.168.192.168 and that we don't care about what Layer-3 protocol or Layer-4 ports we're capturing - We want them all. Take a look at the following screenshot which will show you what the command will look like. I've highlighted three items to pay attention to:

You will need to change the Source IP Address and Destination IP Address to match with your actual infrastructure. Also, take note that this version of 'fw monitor' captures both Pre- and Post-Encryption so you'll be able to see the packets as they pass through the VPN - There's no need to change anything here, I just wanted to highlight that for you.

Take the command (fw monitor -F "10.20.30.40,0,192.168.192.168,0,0") and run it on your R80.40 gateway - Do not do this from the management station or you'll end up with a recursive packet capture which may fill up your SSH buffer and that's not good. Once the command has loaded, send TCP traffic (SSH, SmartConsole, whatever) from the management station to the internal IP address of the P81 gateway. You should see something like the following:

[i] 10.20.30.40.12345 -> 192.169.192.168.22
[I] 10.20.30.40.12345 -> 192.169.192.168.22
[e] 10.20.30.40.12345 -> 192.169.192.168.22

This indicates that the packet has passed the wire and into the Check Point kernel (i), passed the first chain of inspection (I) and entered into a VPN tunnel (e). (There's a lot more happening but this is the basic version of what's happening). I'm 99.9% certain that this is already happening based on the screenshot of the logs you posted (In IT, never say 100% 😉) which is a good sign.

Next is to verify that the P81 is receiving the packets properly. Unfortunately, I'm not familiar with these devices but if they have either a packet capture tool (like tcpdump) or, even better, a kernel-level network flow diagnosis tool (like 'fw monitor') then we can use that to see what's happening to the packet. If it doesn't, here's something goofy you can try which will help narrow down whether this is an issue with traffic or the actual application (SSH, CPMI, whatever) connection:

On the remote (P81) side, instantiate a Linux box/VM. On that box, install the 'socat' utility. Run the command 'socat TCP-LISTEN:23,fork TCP:192.168.192.168:22' <-- This will create a LISTEN socket on TCP/23 of the Linux host and redirect any traffic it receives to 192.168.192.168 on TCP/22. Once configured, use an SSH client (like PuTTY) and connect to 'Linux_host_IP_address:23' and you should be prompted to log into the P81 box over SSH. If it works, then the service (SSH in this case) is working and it's a network issue. If not, then run a PCap on the Linux box (use tcpdump in this case) and see what's happening to the packets. Keep in mind that you can redirect any Layer-4 port with socat so, in theory, you could do this with CPMI but that will make things much (MUCH) harder to manage and I would only do this in a lab, not production.

Let me know if you need more info and I'll do my best to help out but free time isn't a commodity I have a lot of right now, sorry. Real-life, my job and all that jazz... Always gets in the way. 🙃

With all that being said, are you able to just peer CPMI over the WAN? It's all encrypted anyway.

Gr@ve_Rose

[PSushko]

Wow. What a detailed response. Thank you very much for all of this. I'm going to try it out.

Regarding access over CPM/CPMI, it does not appear to work. I can see the traffic reaching our on-prem management through TCP/19009, however, I get the following message from my SmartConsole:

Here is the traffic going through:

I'm going to run a fw monitor soon. I'll let you know my findings.

Thank you for everything so far!

[Grave_Rose]

[Apparently this was deleted or I missed posting it so here I go again.]

I think I misunderstood your initial request. I thought you wanted to manage the P81 device across the VPN and were having issues with that but it seems that you just want to connect a GUI client to your management station from the P81 encryption domain. Is that correct? If so, here's what you need to do from a packet flow situation.

First, for the 'fw monitor' set the source as your client IP address (192.168.192.100) and the destination as your management station (10.20.30.40) and run that on your R80.40 gateway. When that's running, attempt to connect your GUI client. You should see something like this:

(E) 192.168.192.100.12345 -> 10.20.30.40.18191
(o) 192.168.192.100.12345 -> 10.20.30.40.18191
(O) 192.168.192.100.12345 -> 10.20.30.40.18191

The big "O" indicates that the packet has left the Check Point kernel and is now destined for the wire. You can verify this with tcpdump but you likely don't need to.

Next, run a PCap on your management station. If it's Windows, use Wireshark. Fun fact: Wireshark uses BPF when capturing packets so you can use tcpdump syntax to filter. If it's GAiA or another *nix, use tcpdump. We're going to assume that you're running GAiA and that the physical interface on your management station is eth0. Run the following command without the quotes: "tcpdump -nn -vvv -e -s 0 -X -c 100 -i eth0 host 192.168.192.100 and host 10.20.30.40" which will capture packets whose source or destination is either of those IP addresses. Here's what you're looking for:

1. A SYN packet from the client. It will show up as [S] in the packet header. If you see this, that means the client is reaching the management station. If you don't see this, then there is something blocking the traffic between the firewall and the management station - It could be the firewall itself.
2. A SYN/ACK packet from the management station to the client. It will show up as [S.] in the packet header. If you see this, it means that the management station is accepting the connection. If you don't see this but you do see the SYN packet -or- you see the SYN packet but also see a RST [R] packet, then it means that the management station is not accepting the connection. Run 'cpconfig' and make sure you've added your client IP address as a GUI client.

Hope that helps and sorry for the confusion in my previous post.

Cheers,

Gr@ve_Rose

[PSushko]

Thank you for the reply, it was very useful.

I can see SYNs and SYN/ACKs, but no RSTs. I can clearly see that my host's P81 address isn't in the cpconfig GUI Clients list, but neither is my on-prem address from which I am currently logged into SmartConsole on our management. Matter of fact they all look like very bizarre IP addresses. However, the first entry is an Any. Am I missing something?

[Grave_Rose]

I try to be useful but it doesn't always work. 🙃 Okay, we're making good progress. Since you have "*Any" as the first line in your GUI clients, any IP address will be accepted which explains the SYN/ACK we saw leaving the management station. As an aside, take a screenshot of the "bizarre IP addresses" and post it here so we can take a look. The next step is to perform the same tasks in reverse. So...

1. Do you have any other intermediary devices between the R80.40 firewall and your management station? You likely have switches but do you have anything with an IP-based firewall on it? Even on switches, you can have a firewall - Are you using any firewalls on your switches?
2. Assuming not, we need to look at the firewall as our next hop. Log into the CLI and we're going to run another PCap. We know that we're expecting to see the SYN/ACK so let's look for that. We're starting on the wire with 'tcpdump' before moving into kernel space. Run (without the quotes): "tcpdump -nn -vvv -e -s 0 -X -c 100 -i eth-s1p1c0 host 192.168.192.100 and host 10.20.30.40" <-- Replace "eth-s1p1c0" with the physical interface name that faces your management station and, obviously, the IP addresses. 🙂
3. With that PCap running, attempt your connection again. You will see the SYN packet leaving [S] but we're interested in the SYN/ACK [S.] and whether it hits the firewall as it should. Here's where we get to divergent paths of our troubleshooting...

If we don't see the SYN/ACK [S.] packet on the R80.40 firewall, something is blocking the return packet and/or it's being routed through a different gateway. We'll save this troubleshooting for after the next update from your side. To prepare for that, gather the following information from your management station: The output of 'netstat -nr' and 'arp -na' - You may need to be in 'expert' mode to get this information.

If we see the SYN/ACK, our next step is to re-run the 'fw monitor' on the R80.40 firewall but with the source and destination reversed. We expect to see (i) (I) (e) as the packet enters the kernel, passes the first chain and enters the VPN tunnel. If you only see (i) or (i) (I) then the firewall isn't putting it into the VPN and/or is dropping the packet and it's time to call support.

But let's say we do see (i) (I) (e) - Then what? We know that the packet is being sent across the VPN. Run a PCap (if possible) on the internal interface of the P81 device. If you can't do that, then run a PCap (Wireshark) on your client and see if the SYN/ACK [S.] shows up there. If not, you'll have to create a SPAN/RSPAN/mirror port on the switch of the P81-side network and run a PCap to see what's happening to the packet.

I'm going to sound like a broken-record to anyone who knows me but PCaps will always point you to where the problem lies at the time. They may take a while but run them in a forward-flow direction and then reverse-flow direction: When you come across a device where the packet isn't processed properly, that's the first place you need to investigate. Rinse and repeat.

Looking forward to your next update.

Cheers,

Gr@ve_Rose

[virtualj]

Hello all,

I have very similar issue. In the specific case I have my Smart console client in one office, then it is connected through a VPN tunnel that ends on a gateway managed by the Management to witch I connect on CPM port 19009. What I see with tcpdump on the firewall is the syn packet received from VPN and sent to the management. Than I see the SYN-ACK coming from the management, but the packet is forwarded to the default gateway and not in the VPN tunnel. So the connection will not complete. In the log I see that traffic is accepted by an Implicit rule, but I can't understand witch of them. Anyway I think that this implicit rule does not have the VPN tunnel applied so the reply packet will not be encrypted. I can add that this was working fine with R80.30 and it does not work anymore after the upgrade to R81.20.

[PhoneBoy]

One of the things an upgrade does not bring across is manual changes to .def files used to compile the Access Policy for gateways.
These sorts of changes are required to allow SIC/CPM traffic to flow over a VPN and have to be re-applied.
Precisely where depends on both gateway and management versions involved.