Post-Migration Issue: Lost HTTPS and External SSH Access to Cluster Member

[Moudar]

hi,

Following an SMS migrate_server import, I am experiencing an issue with one of our cluster members.

Currently, HTTPS access is completely unavailable, and SSH access to this specific member fails from all external sources. Interestingly, SSH is only successful when connecting directly from the other cluster member.

tcpdump -ni any "host 10.110.16.32 and tcp port 22"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
08:19:06.985053 ethertype IPv4, IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:06.985053 ethertype IPv4, IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:06.985053 IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:07.985604 ethertype IPv4, IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:07.985604 ethertype IPv4, IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:07.985604 IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:09.990909 ethertype IPv4, IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:09.990909 ethertype IPv4, IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:09.990909 IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:14.000770 ethertype IPv4, IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:14.000770 ethertype IPv4, IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:14.000770 IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:22.003240 ethertype IPv4, IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:22.003240 ethertype IPv4, IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
08:19:22.003240 IP 10.110.16.32.10400 > 192.168.99.250.ssh: Flags [S], seq 2819899382, win 65535, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0

you can see no syn-ack!

Smartconsole logs also show that the request is accepted, and there are no antispoofing issues.

there is nothing logged when running zdebug drop, so for all intent and purposes, the packet is received, then "vanishes" after that.

here is what fw monitor -F shows:

[vs_0][ppak_0] bond1.99:i[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50403
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][fw_5] bond1.99:i[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50403
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][fw_5] bond1.99:I[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50403
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][ppak_0] bond1.99:i[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50404
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][fw_5] bond1.99:i[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50404
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][fw_5] bond1.99:I[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50404
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][ppak_0] bond1.99:i[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50405
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][fw_5] bond1.99:i[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50405
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][fw_5] bond1.99:I[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50405
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][ppak_0] bond1.99:i[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50406
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][fw_5] bond1.99:i[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50406
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][fw_5] bond1.99:I[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50406
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][ppak_0] bond1.99:i[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50407
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][fw_5] bond1.99:i[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50407
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000
[vs_0][fw_5] bond1.99:I[52]: 10.110.16.32 -> 192.168.99.250 (TCP) len=52 id=50407
TCP: 10400 -> 22 .S.... seq=275afd3d ack=00000000

so only i and I

there is a correct route:

S               10.110.16.32/32     via 192.168.99.254, bond1.99, cost 0, age 52435

where 192.168.99.254 is vip 

Anyone seen anything similar before?

[simonemantovani]

Did you check also what is reported in the file /var/log/secure?

[simonemantovani]

Do you have the same behaviour when you connect in HTTPS? (no reply from gateway)

No allowed-client configured right?

[Moudar]

10.110.16.35 is then natted to 10.110.16.32

[simonemantovani]

Is not clear for me ... the IP of the firewall is 192.168.99.250 or 10.110.16.35? Should it be 192.168.99.250 right?

[Moudar]

192.168.99.250 
10.110.16.35 is natted ip

[Moudar]

 /var/log/secure 
shows only active node not standby what i could see

[Moudar]

Now i got this

tcpdump -eni bond1.99 arp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bond1.99, link-type EN10MB (Ethernet), capture size 262144 bytes 09:07:45.691591 00:1c:7f:a4:60:9d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.99.254 tell 192.168.99.250, length 28 09:07:46.691856 00:1c:7f:a4:60:9d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.99.254 tell 192.168.99.250, length 28 9:07:47.693856 00:1c:7f:a4:60:9d > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 192.168.99.254 tell 192.168.99.250, length 28

firewall member .250 is ARPing for but no body answers, 

But 192.168.99.254 is the ClusterXL VIP.

So the member is literally unable to resolve the cluster VIP MAC. I am confused

[Martijn]

Hi,

Did you perform an upgrade / hotfix installation on the gateway.
It is my experience the web ssl-port reverts to 443 when installing a hotfix. Can you check the configured ssl-port?

#show web ssl-port

Is the NAT performed on the cluster you are trying to access?
Can you add a network diagram so we have a picture of what is configured?

Is the an allowed client list on the gateways configured?

Martijn

[Moudar · Accepted solution]

Hi

Maybe my story was not complete but here is the

Solution: https://support.checkpoint.com/results/sk/sk104468

ClusterXL: Accessing a Standby Cluster Member through IPSec VPN

is good to read:

sk93204 - ClusterXL: Accessing a Standby Cluster Member through IPSec VPN