Re: Asymmetric routing

Moudar · ‎2023-01-04

Hi,

Is there any way to find if there is asymmetric routing (a tool or command?)?

I got so many packets dropped on 52.112.0.0/14 which is MS-Teams network.

This is happening all the time and people are complaining!

Any ideas?

all these packets are dropped for reason "First packet isn't SYN"

_Val_ · ‎2023-01-08

Check the routing on the FW, and run some tracing to see how traffic is actually going.

Moudar · ‎2023-01-09

Any example of how to do it?

I got my CCSA but still newbie and do not know how to do these!

Is it in Expert mode? what command?

G_W_Albrecht · ‎2023-01-09

sk169154: Asymmetric Connections in ClusterXL R80.20 and Higher

An example will not really help much - better go thru the process together with a senior engineer or take a WireShark class.

CCSE / CCTE / CTPS / CCME / CCSM Elite / SMB Specialist

Chris_Atkinson · ‎2023-01-09

The following will help you review the routing table and you can compare the interface identifiers in the log cards...

show route

netstat -rn

Another consideration could be the following as resolved in R81.10 JHF T82 (refer also: sk180364).

PRJ-42445,
PRHF-26215

SecureXL

The Security Gateway may prematurely expire half-closed TCP connections and drop VoIP and HTTPS packets with "First packet isn't SYN".

CCSM R77/R80/ELITE

Moudar · ‎2023-01-10

We run 81.10

Will this version have bugg in SecureXL?

Chris_Atkinson · ‎2023-01-10

Depends on the Jumbo Hotfix level (take), you can confirm via "cpinfo -y all" or work with TAC to verify that this is indeed the issue that you are encountering.

CCSM R77/R80/ELITE

Moudar · ‎2023-01-10

fw01> cpinfo -y all

This is Check Point CPinfo Build 914000231 for GAIA
[CPFC]
        No hotfixes..
[IDA]
        No hotfixes..
[MGMT]
        HOTFIX_R81_10_JUMBO_HF_MAIN     Take:  79
[FW1]
        HOTFIX_R80_40_MAAS_TUNNEL_AUTOUPDATE
        HOTFIX_R81_10_JUMBO_HF_MAIN     Take:  79
        HOTFIX_GOT_TPCONF_AUTOUPDATE
        HOTFIX_PUBLIC_CLOUD_CA_BUNDLE_AUTOUPDATE

FW1 build number:
This is Check Point's software version R81.10 - Build 029
kernel: R81.10 - Build 030
[SecurePlatform]
        HOTFIX_ENDER_V17_AUTOUPDATE
        HOTFIX_R81_10_JUMBO_HF_MAIN     Take:  79
[AutoUpdater]
        No hotfixes..
[CPinfo]
        No hotfixes..
[PPACK]
        HOTFIX_R81_10_JUMBO_HF_MAIN     Take:  79
[DIAG]
        No hotfixes..
[CVPN]
        HOTFIX_R81_10_JUMBO_HF_MAIN     Take:  79
        HOTFIX_ESOD_SWS_AUTOUPDATE
        HOTFIX_ESOD_CSHELL_AUTOUPDATE
        HOTFIX_ESOD_SCANNER_AUTOUPDATE
[CPDepInst]
        No hotfixes..
[CPUpdates]
        BUNDLE_ENDER_V17_AUTOUPDATE     Take:  21
        BUNDLE_CPOTELCOL_AUTOUPDATE     Take:  22
        BUNDLE_CPVIEWEXPORTER_AUTOUPDATE        Take:  25
        BUNDLE_R80_40_MAAS_TUNNEL_AUTOUPDATE    Take:  49
        BUNDLE_R81_10_JUMBO_HF_MAIN     Take:  79
        BUNDLE_GOT_TPCONF_AUTOUPDATE    Take:  111
        BUNDLE_PUBLIC_CLOUD_CA_BUNDLE_AUTOUPDATE        Take:  19
        BUNDLE_HCP_AUTOUPDATE   Take:  58
        BUNDLE_GENERAL_AUTOUPDATE       Take:  13
        BUNDLE_ESOD_SWS_AUTOUPDATE      Take:  14
        BUNDLE_ESOD_CSHELL_AUTOUPDATE   Take:  18
        BUNDLE_ESOD_SCANNER_AUTOUPDATE  Take:  10
        BUNDLE_CPSDC_AUTOUPDATE Take:  21
        BUNDLE_CORE_FILE_UPLOADER_AUTOUPDATE    Take:  21
        BUNDLE_INFRA_AUTOUPDATE Take:  58
        BUNDLE_DEP_INSTALLER_AUTOUPDATE Take:  25
[core_uploader]
        HOTFIX_CHARON_HF
[cpsdc_wrapper]
        HOTFIX_CPSDC_AUTOUPDATE
[hcp_wrapper]
        HOTFIX_HCP_AUTOUPDATE
[CPviewExporter]
        HOTFIX_OTLP_GA
[CPotelcol]
        HOTFIX_OTLP_GA

I could not see the "R81.10 R81_10_jumbo_hf_main 38" among hotfixes we have!

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

could it be TCP Half Closed timer issue?

G_W_Albrecht · ‎2023-01-10

Installed R81.10 JumboTake 79 does contain all hotfixes from JT 9 up to JT 78, so also JT 38 ! I would suggest working with TAC on this to get it resolved quickly !

CCSE / CCTE / CTPS / CCME / CCSM Elite / SMB Specialist

Chris_Atkinson · ‎2023-01-10

As you can see it could potentially be multiple somewhat related things and requires further diagnosis.

At this time I cannot blindly recommend installing JHF T82 as it is yet to reach "Recommended" status so you should first verify with TAC if this is relevant for your specific issue with the relevant debugs etc.

I.e. If your confident the routing checks out then move onto looking further into the packet captures / debugs with TAC.

CCSM R77/R80/ELITE

G_W_Albrecht · ‎2023-01-10

We do know virtually nothing about your deployment, could you provide a topology ? Is the GW a GA cluster ?

CCSE / CCTE / CTPS / CCME / CCSM Elite / SMB Specialist

Moudar · ‎2023-01-10

This is a simple topology, we have cluster but only one is active

We have the same setup sitting in another site as backup if this one fail.

Blason_R · ‎2023-01-09

Try shutting down the secureXL and see?

Run #fwaccel off from expert mode.

Thanks and Regards,
Blason R
CCSA,CCSE,CCCS

Moudar · ‎2023-01-10

What can happen when disabling SecureXL?

fw01> show securexl status
+---------------------------------------------------------------------------------+
|Id|Name     |Status     |Interfaces               |Features                      |
+---------------------------------------------------------------------------------+
|0 |KPPAK    |enabled    |Sync,Mgmt,eth1-01,       |Acceleration,Cryptography     |
|  |         |           |eth1-03,eth1-04          |                              |
|  |         |           |                         |Crypto: Tunnel,UDPEncap,MD5,  |
|  |         |           |                         |SHA1,3DES,DES,AES-128,AES-256,|
|  |         |           |                         |ESP,LinkSelection,DynamicVPN, |
|  |         |           |                         |NatTraversal,AES-XCBC,SHA256, |
|  |         |           |                         |SHA384,SHA512                 |
+---------------------------------------------------------------------------------+

Accept Templates : disabled by Firewall
                   Layer Network disables template offloads from rule #19
                   Throughput acceleration still enabled.
Drop Templates   : enabled
NAT Templates    : disabled by Firewall
                   Layer Network disables template offloads from rule #19
                   Throughput acceleration still enabled.
LightSpeed Accel : disabled

AaronCP · ‎2023-01-09

Can you provide a full log card (sensitive info redacted), please?

Zolocofxp · ‎2023-01-11

Hello,

Run a traceroute from a client pc. It may happen that the first tcp packet is reaching your WAN without going through the firewall but returning traffic is. A packet capture on your security gateway will confirm this.

Moudar · ‎2023-01-11

Should i Traceroute my WAN address from a client?

With a packet capture on my security gateway, you mean capture packets of the same client PC?

If that right what is the command to run TCPdump on one client?

Zolocofxp · ‎2023-01-11

Run a traceroute from client to a MS-Teams destination IP. For example:

tracert -d 52.112.120.205

Have look at the hops. If you don't see your Security Gateway IP, there is a routing issue.

Packet capture should be ran in your fw01(or active cluster member for that matter) in expert mode and traffic must be generated on the client side(using tcping is good for this). For example:

tcpdump -nni any host [client IP] and host 52.112.120.205

Good routing will show complete TCP handshake. Like this:

I hope it helps.

Moudar · ‎2023-01-12

The trace was successful, i mean the trace showed my fw01 ip address. I tested from one site and i will test other sites also.

I collected some dumps, do you see anything wrong?

16:49:41.293688 ethertype IPv4, IP 10.162.0.17.56495 > 52.113.194.132.443: Flags [.], seq 2571763857:2571763858, ack 2579271690, win 516, length 1
16:49:41.293688 ethertype IPv4, IP 10.162.0.17.56495 > 52.113.194.132.443: Flags [.], seq 0:1, ack 1, win 516, length 1
16:49:41.293688 IP 10.162.0.17.56495 > 52.113.194.132.443: Flags [.], seq 0:1, ack 1, win 516, length 1
16:49:41.310470 IP 52.113.194.132.443 > 10.162.0.17.56495: Flags [.], ack 1, win 16384, options [nop,nop,sack 1 {0:1}], length 0
16:49:41.310474 ethertype IPv4, IP 52.113.194.132.443 > 10.162.0.17.56495: Flags [.], ack 1, win 16384, options [nop,nop,sack 1 {0:1}], length 0
16:49:41.310475 ethertype IPv4, IP 52.113.194.132.443 > 10.162.0.17.56495: Flags [.], ack 1, win 16384, options [nop,nop,sack 1 {0:1}], length 0

16:49:49.435781 ethertype IPv4, IP 10.162.0.17.56464 > 52.113.205.32.443: Flags [.], seq 714859464:714859465, ack 640452860, win 512, length 1
16:49:49.435781 ethertype IPv4, IP 10.162.0.17.56464 > 52.113.205.32.443: Flags [.], seq 0:1, ack 1, win 512, length 1
16:49:49.435781 IP 10.162.0.17.56464 > 52.113.205.32.443: Flags [.], seq 0:1, ack 1, win 512, length 1
16:49:49.479585 IP 52.113.205.32.443 > 10.162.0.17.56464: Flags [.], ack 1, win 16384, options [nop,nop,sack 1 {0:1}], length 0
16:49:49.479588 ethertype IPv4, IP 52.113.205.32.443 > 10.162.0.17.56464: Flags [.], ack 1, win 16384, options [nop,nop,sack 1 {0:1}], length 0
16:49:49.479589 ethertype IPv4, IP 52.113.205.32.443 > 10.162.0.17.56464: Flags [.], ack 1, win 16384, options [nop,nop,sack 1 {0:1}], length 0
16:49:50.237243 ethertype IPv4, IP 10.162.0.17.55863 > 52.114.77.102.443: Flags [P.], seq 3059722093:3059722151, ack 2468649631, win 516, length 58
16:49:50.237243 ethertype IPv4, IP 10.162.0.17.55863 > 52.114.77.102.443: Flags [P.], seq 0:58, ack 1, win 516, length 58
16:49:50.237243 IP 10.162.0.17.55863 > 52.114.77.102.443: Flags [P.], seq 0:58, ack 1, win 516, length 58
16:49:50.279690 IP 52.114.77.102.443 > 10.162.0.17.55863: Flags [P.], seq 1:48, ack 58, win 2047, length 47
16:49:50.279692 ethertype IPv4, IP 52.114.77.102.443 > 10.162.0.17.55863: Flags [P.], seq 1:48, ack 58, win 2047, length 47
16:49:50.279693 ethertype IPv4, IP 52.114.77.102.443 > 10.162.0.17.55863: Flags [P.], seq 1:48, ack 58, win 2047, length 47
16:49:50.337035 ethertype IPv4, IP 10.162.0.17.55863 > 52.114.77.102.443: Flags [.], ack 48, win 516, length 0
16:49:50.337035 ethertype IPv4, IP 10.162.0.17.55863 > 52.114.77.102.443: Flags [.], ack 48, win 516, length 0
16:49:50.337035 IP 10.162.0.17.55863 > 52.114.77.102.443: Flags [.], ack 48, win 516, length 0

16:49:52.441361 ethertype IPv4, IP 10.162.0.17.55209 > 52.113.205.21.443: Flags [P.], seq 2052906264:2052906323, ack 3315725357, win 517, length 59
16:49:52.441361 ethertype IPv4, IP 10.162.0.17.55209 > 52.113.205.21.443: Flags [P.], seq 0:59, ack 1, win 517, length 59
16:49:52.441361 IP 10.162.0.17.55209 > 52.113.205.21.443: Flags [P.], seq 0:59, ack 1, win 517, length 59
16:49:52.485603 IP 52.113.205.21.443 > 10.162.0.17.55209: Flags [P.], seq 1:49, ack 59, win 2052, length 48
16:49:52.485606 ethertype IPv4, IP 52.113.205.21.443 > 10.162.0.17.55209: Flags [P.], seq 1:49, ack 59, win 2052, length 48
16:49:52.485609 ethertype IPv4, IP 52.113.205.21.443 > 10.162.0.17.55209: Flags [P.], seq 1:49, ack 59, win 2052, length 48
16:49:52.541371 ethertype IPv4, IP 10.162.0.17.55209 > 52.113.205.21.443: Flags [.], ack 49, win 517, length 0
16:49:52.541371 ethertype IPv4, IP 10.162.0.17.55209 > 52.113.205.21.443: Flags [.], ack 49, win 517, length 0
16:49:52.541371 IP 10.162.0.17.55209 > 52.113.205.21.443: Flags [.], ack 49, win 517, length 0

17:03:23.213539 ethertype IPv4, IP 10.2.9.191.63511 > 52.112.120.213.443: Flags [P.], seq 1434179809:1434179866, ack 967007099, win 514, length 57
17:03:23.213539 ethertype IPv4, IP 10.2.9.191.63511 > 52.112.120.213.443: Flags [P.], seq 0:57, ack 1, win 514, length 57
17:03:23.213539 IP 10.2.9.191.63511 > 52.112.120.213.443: Flags [P.], seq 0:57, ack 1, win 514, length 57
17:03:23.231052 IP 52.112.120.213.443 > 10.2.9.191.63511: Flags [P.], seq 1:47, ack 57, win 2048, length 46
17:03:23.231054 ethertype IPv4, IP 52.112.120.213.443 > 10.2.9.191.63511: Flags [P.], seq 1:47, ack 57, win 2048, length 46
17:03:23.231055 ethertype IPv4, IP 52.112.120.213.443 > 10.2.9.191.63511: Flags [P.], seq 1:47, ack 57, win 2048, length 46
17:03:23.281896 ethertype IPv4, IP 10.2.9.191.63511 > 52.112.120.213.443: Flags [.], ack 47, win 513, length 0
17:03:23.281896 ethertype IPv4, IP 10.2.9.191.63511 > 52.112.120.213.443: Flags [.], ack 47, win 513, length 0
17:03:23.281896 IP 10.2.9.191.63511 > 52.112.120.213.443: Flags [.], ack 47, win 513, length 0
17:03:28.529736 ethertype IPv4, IP 10.2.9.191.64282 > 52.113.205.34.443: Flags [.], seq 517848:517849, ack 3720984262, win 511, length 1
17:03:28.529736 ethertype IPv4, IP 10.2.9.191.64282 > 52.113.205.34.443: Flags [.], seq 0:1, ack 1, win 511, length 1
17:03:28.529736 IP 10.2.9.191.64282 > 52.113.205.34.443: Flags [.], seq 0:1, ack 1, win 511, length 1
17:03:28.573151 IP 52.113.205.34.443 > 10.2.9.191.64282: Flags [.], ack 1, win 16382, options [nop,nop,sack 1 {0:1}], length 0
17:03:28.573153 ethertype IPv4, IP 52.113.205.34.443 > 10.2.9.191.64282: Flags [.], ack 1, win 16382, options [nop,nop,sack 1 {0:1}], length 0
17:03:28.573155 ethertype IPv4, IP 52.113.205.34.443 > 10.2.9.191.64282: Flags [.], ack 1, win 16382, options [nop,nop,sack 1 {0:1}], length 0
17:03:34.817684 ethertype IPv4, IP 10.2.9.191.63516 > 52.113.205.20.443: Flags [P.], seq 1275410869:1275410927, ack 853534051, win 517, length 58
17:03:34.817684 ethertype IPv4, IP 10.2.9.191.63516 > 52.113.205.20.443: Flags [P.], seq 0:58, ack 1, win 517, length 58
17:03:34.817684 IP 10.2.9.191.63516 > 52.113.205.20.443: Flags [P.], seq 0:58, ack 1, win 517, length 58
17:03:34.862009 IP 52.113.205.20.443 > 10.2.9.191.63516: Flags [P.], seq 1:48, ack 58, win 2047, length 47
17:03:34.862012 ethertype IPv4, IP 52.113.205.20.443 > 10.2.9.191.63516: Flags [P.], seq 1:48, ack 58, win 2047, length 47
17:03:34.862013 ethertype IPv4, IP 52.113.205.20.443 > 10.2.9.191.63516: Flags [P.], seq 1:48, ack 58, win 2047, length 47
17:03:34.903886 ethertype IPv4, IP 10.2.9.191.63516 > 52.113.205.20.443: Flags [.], ack 48, win 516, length 0
17:03:34.903886 ethertype IPv4, IP 10.2.9.191.63516 > 52.113.205.20.443: Flags [.], ack 48, win 516, length 0
17:03:34.903886 IP 10.2.9.191.63516 > 52.113.205.20.443: Flags [.], ack 48, win 516, length 0

Zolocofxp · ‎2023-01-18

I don't see any Sync flags [S]. Run the capture again and export it to a file with -w <file.pcap>. Attach that file her to have good look at it. I believe you can always create an exception on your Inspection Settings.

Paste the complete log detail from one of those drops too, to see the correct exception to apply.

Moudar · ‎2023-01-18

The .pcap file i am getting is not working in Wireshark, i get this message from Wireshark:

"The capture file appears to have been cut short in the middle of a packet"

the command i used is:

tcpdump -i any -nn -e dst net 52.112.0.0/14 and port 443 -c 20 -w /var/log/capture.pcap

Zolocofxp · ‎2023-01-19

Hello,

Remove "-c 20" and add "-s 0" instead.

Moudar · ‎2023-01-19

I am getting a new message from Wireshark:

The capture file appears to be damaged or corrupt.

(pcap: File has 1282454272-byte packet, bigger than maximum of 262144)

tcpdump -i any -nn -e -s 0 dst net 52.112.0.0/14 and port 443 -w /var/log/capture1.pcap

Are you a member of CheckMates?

Asymmetric routing