R80.30 - Build 001 MDS Connectivity test before migration - connection refused??

[Firewallteam_DE]

Hello

We are implementing access rules, AS, and routing on existing R77.30 Firewalls (VS, VSX, Phys Clusters) for new R80 MDS/CMA to take over. 

Many firewalls (overall more than 1000) are involved daily and I am having problem to reliably test open ports From MDS and CMA context without time-consuming need to check that in tracker dashboard by dashboard.

My tests looks following (netcat):

nc -z -v -n -t -w 2 -s <R80.30 CMA IP> <R77.30 FW IP> 256
nc -z -v -n -t -w 2 -s <R80.30 CMA IP> <R77.30 FW IP> 18191
nc -z -v -n -t -w 2 -s <R80.30 CMA IP> <R77.30 FW IP> 18208
nc -z -v -n -t -w 2 -s <R80.30 MDS IP> <R77.30 FW IP> 443
nc -z -v -n -t -w 2 -s <R80.30 MDS IP> <R77.30 FW IP> 18208
nc  -v -n -u -w 2 -s <R80.30 MDS IP> <R77.30 FW IP> 161

These ports are needed for management and monitoring.

Even though FW allows it and smarttracker confirms it by log - I am getting "connection refused" in R80 CLI.

telnet and curl gives also "connection refused"

As you understand it is not enough to rely on this feedback and I have to check further in logs.

Can you help suggest simillar quick way how to test from R80 CLI and get confirmed response that ports are open?

Spending nights going from dashboard to dashboard is taking toll on me.

Is there any extensive cli tool that can do this and can be installed on gaia? Whatever can help me.

[PhoneBoy]

The log only shows you that the firewall passed the traffic.
It doesn't show you if the other end actually received the traffic.
A "connection refused" can happen for two reasons:
1. The traffic was rejected by something after the firewall and before the target is reached.
2. The traffic actully did reach the target but nothing is actually listening on the specified port.

From the far end of the connection, you can use tcptraceroute to determine which of these reasons applies.
tcptraceroute is like a regular traceroute except it uses TCP packets.
Unless something in the middle is dropping packets with a low TTL (which does happen), it should tell you which situation is at play.
If you make it to the destination host and you still see a connection refused, then you know the path is clear and the issue is the remote end isn't listening.
If the message is received from a different IP, then you have an idea where to begin troubleshooting.

It looks like tcptraceroute is included in R80.40.
Not sure if it's included in prior releases.

[Firewallteam_DE]

Thank you PhoneBoy

I found tcptraceroute on our version. I used:

tcptraceroute -T -s <source> <destination> -p <port>

This works fine until it reaches its destination

For test on UDP 161 its not very reliable:

snowing after first hop

Let me show whole Netcat test with VS example cause I believe things are ok on the path, but I am not sure about reception when packets arrive back:

Reaching Virtual FW Test before migration

Testing with netcat from R80:

nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 256
nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 18191
nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 18208
nc -z -v -n -t -w 2 -s <MDS IP> <Virtual System IP> 443
nc -z -v -n -t -w 2 -s <MDS IP> <Virtual System IP> 18208
nc -v -n -u -w 2 -s <MDS IP> <Virtual System IP> 161

Response received (connection refused):

[Expert@<R80 Test MDS>:0]# nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 256
nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 18191
nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 18208
nc -z -v -n -t -w 2 -s <MDS IP> <Virtual System IP> 443
nc -z -v -n -t -w 2 -s <MDS IP> <Virtual System IP> 18208
nc -v -n -u -w 2 -s <MDS IP> <Virtual System IP> 161<Virtual System IP> 256: Connection refused
[Expert@<R80 Test MDS>:0]# nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 18191
<Virtual System IP> 18191: Connection refused
[Expert@<R80 Test MDS>:0]# nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 18208
<Virtual System IP> 18208: Connection refused
[Expert@<R80 Test MDS>:0]# nc -z -v -n -t -w 2 -s <MDS IP> <Virtual System IP> 443
<Virtual System IP> 443 (https): Connection refused
[Expert@<R80 Test MDS>:0]# nc -z -v -n -t -w 2 -s <MDS IP> <Virtual System IP> 18208
<Virtual System IP> 18208: Connection refused
[Expert@<R80 Test MDS>:0]# nc -v -n -u -w 2 -s <MDS IP> <Virtual System IP> 161

read(net): Connection refused
[Expert@<R80 Test MDS>:0]# nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 256
<Virtual System IP> 256: Connection refused
[Expert@<R80 Test MDS>:0]# nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 18191
<Virtual System IP> 18191: Connection refused
[Expert@<R80 Test MDS>:0]# nc -z -v -n -t -w 2 -s <CMA IP> <Virtual System IP> 18208
<Virtual System IP> 18208: Connection refused
[Expert@<R80 Test MDS>:0]# nc -z -v -n -t -w 2 -s <MDS IP> <Virtual System IP> 443
<Virtual System IP> 443 (https): Connection refused
[Expert@<R80 Test MDS>:0]# nc -z -v -n -t -w 2 -s <MDS IP> <Virtual System IP> 18208
<Virtual System IP> 18208: Connection refused
[Expert@<R80 Test MDS>:0]# nc -v -n -u -w 2 -s <MDS IP> <Virtual System IP> 161

read(net): Connection refused
[Expert@<R80 Test MDS>:0]#

Virtual FW SmartTracker:

All tested green, allowed. Including 161 UDP

TCPDUMP confirms ack packets arriving back on R80 (with exception of UDP 161 test):
TCPDUMP confirms ack packets arriving back on R80 (with exception of UDP 161 test):

[Expert@<R80 Test MDS>:0]# tcpdump -v -n -i eth0 host <Virtual System IP>
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:36:53.894136 IP (tos 0x0, ttl 64, id 57091, offset 0, flags [DF], proto TCP (6), length 60)
<CMA IP>.44694 > <Virtual System IP>.256: Flags [S], cksum 0xa683 (correct), seq 984126082, win 29200, options [mss 1460,sackOK,TS val 704269697 ecr 0,nop,wscale 10], length 0
11:36:53.931642 IP (tos 0x0, ttl 49, id 0, offset 0, flags [DF], proto TCP (6), length 40)
<Virtual System IP>.256 > <CMA IP>.44694: Flags [R.], cksum 0xf7e1 (correct), seq 0, ack 984126083, win 0, length 0
11:36:53.945692 IP (tos 0x0, ttl 64, id 31980, offset 0, flags [DF], proto TCP (6), length 60)
<CMA IP>.44883 > <Virtual System IP>.18191: Flags [S], cksum 0x6813 (correct), seq 2470654296, win 29200, options [mss 1460,sackOK,TS val 704269748 ecr 0,nop,wscale 10], length 0
11:36:53.989738 IP (tos 0x0, ttl 49, id 0, offset 0, flags [DF], proto TCP (6), length 40)
<Virtual System IP>.18191 > <CMA IP>.44883: Flags [R.], cksum 0xb9a4 (correct), seq 0, ack 2470654297, win 0, length 0
11:36:54.005322 IP (tos 0x0, ttl 64, id 20320, offset 0, flags [DF], proto TCP (6), length 60)
<CMA IP>.46203 > <Virtual System IP>.18208: Flags [S], cksum 0x24d4 (correct), seq 4292149904, win 29200, options [mss 1460,sackOK,TS val 704269808 ecr 0,nop,wscale 10], length 0
11:36:54.042634 IP (tos 0x0, ttl 49, id 0, offset 0, flags [DF], proto TCP (6), length 40)
<Virtual System IP>.18208 > <CMA IP>.46203: Flags [R.], cksum 0x76a1 (correct), seq 0, ack 4292149905, win 0, length 0
11:36:54.049941 IP (tos 0x0, ttl 64, id 18486, offset 0, flags [DF], proto TCP (6), length 60)
<MDS IP>.36546 > <Virtual System IP>.https: Flags [S], cksum 0xb1ee (correct), seq 345179690, win 29200, options [mss 1460,sackOK,TS val 704269852 ecr 0,nop,wscale 10], length 0
11:36:54.081346 IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 40)
<Virtual System IP>.https > <MDS IP>.36546: Flags [R.], cksum 0x03e8 (correct), seq 0, ack 345179691, win 0, length 0
11:36:54.096731 IP (tos 0x0, ttl 64, id 42506, offset 0, flags [DF], proto TCP (6), length 60)
<MDS IP>.36084 > <Virtual System IP>.18208: Flags [S], cksum 0x3e3d (correct), seq 1650911297, win 29200, options [mss 1460,sackOK,TS val 704269899 ecr 0,nop,wscale 10], length 0
11:36:54.129502 IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 40)
<Virtual System IP>.18208 > <MDS IP>.36084: Flags [R.], cksum 0x9065 (correct), seq 0, ack 1650911298, win 0, length 0
11:36:57.298796 IP (tos 0x0, ttl 64, id 29227, offset 0, flags [DF], proto UDP (17), length 29)
<MDS IP>.44870 > <Virtual System IP>.snmp: [no asnlen]
11:36:57.363000 IP (tos 0xc0, ttl 50, id 9191, offset 0, flags [none], proto ICMP (1), length 57)
<Virtual System IP> > <MDS IP>: ICMP <Virtual System IP> udp port snmp unreachable, length 37
IP (tos 0x0, ttl 50, id 29227, offset 0, flags [DF], proto UDP (17), length 29)
<MDS IP>.44870 > <Virtual System IP>.snmp: [no asnlen]

I don't understand "connection refused" here. Any ideas? Will it accept traffic of new management when we will migrate reliably?

[Maarten_Sjouw]

First of all have a good look at the document @HeikoAnkerbrand has made:
https://community.checkpoint.com/docs/DOC-2740-r80x-ports-used-for-communication-by-various-check-po...
For instance the port 256 is used From FW to CMA for logs.

[PhoneBoy]

The only way to confirm these ports are open with complete metaphysical certitude is to log into the destination host and see, with tcpdump, that the traffic is, in fact, reaching the destination.
Without that, we can only make educated guesses using the available tools.

If you got "Connection Refused" messages and the tcptraceroute suggests it actually reached the destination (based on knowledge of the network), then it's likely the firewall ports are open.

Note that tcptraceroute is only for TCP traffic.
It will not determine if UDP ports are open.

UDP ports are a lot harder to confirm are open due to the stateless nature of UDP.
The equivalent to a SYN RST for TCP (which is what "connection refused" is at the packet level) is an ICMP Destination Unreachable.
This is precisely how regular UDP-based traceroute works and may be blocked by an intermediary firewall.
The fact you don't get an ICMP Destination Unreachable message is, therefore, no indication of whether or not that specific port is open.