Packets visible in tcpdump but not in fw monitor

MartinOles · ‎2022-07-21

Hi all,

during investigation of traffic issue I have found, that packets are visible in tcpdump but surprisingly not in fw monitor. I am trying to figure out reason for such observation.

Source 10.200.251.119 - routed behind interface eth4.2264
Destination 10.230.174.77 – directly connected behind bond1.3031

All acceleration has been switched off:

[Expert@FW002A:0]# fwaccel stat
+-----------------------------------------------------------------------------+
|Id|Name |Status     |Interfaces               |Features                      |
+-----------------------------------------------------------------------------+
|0 |SND  |disabled   |eth1,eth2,eth7,eth3,Mgmt,|
|  |     |           |eth4                     |Acceleration,Cryptography     |
|  |     |           |                         |Crypto: Tunnel,UDPEncap,MD5,  |
|  |     |           |                         |SHA1,NULL,3DES,DES,CAST,      |
|  |     |           |                         |CAST-40,AES-128,AES-256,ESP,  |
|  |     |           |                         |LinkSelection,DynamicVPN,     |
|  |     |           |                         |NatTraversal,AES-XCBC,SHA256  |
+-----------------------------------------------------------------------------+

[Expert@FW002A:0]# vpn accel stat
VPN acceleration is disabled.

On interface facing source IP traffic is visible in one direction only:

[Expert@FW002A:0]# tcpdump -i eth4.2264 host 10.200.251.119 and host 10.230.174.77 -c 200
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth4.2264, link-type EN10MB (Ethernet), capture size 96 bytes
21:42:32.513186 IP 10.200.251.119 > 10.230.174.77: ICMP echo request, id 27447, seq 10556, length 40
21:42:37.506527 IP 10.200.251.119 > 10.230.174.77: ICMP echo request, id 27447, seq 10557, length 40
21:42:42.516885 IP 10.200.251.119 > 10.230.174.77: ICMP echo request, id 27447, seq 10558, length 40

On interface directly connected destination, traffic visible in both directions:

[Expert@FW002A:0]# tcpdump -i bond1.3031 host 10.200.251.119 and host 10.230.174.77 -c 200
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond1.3031, link-type EN10MB (Ethernet), capture size 96 bytes
21:55:01.526995 IP 10.200.251.119 > 10.230.174.77: ICMP echo request, id 27447, seq 10755, length 40
21:55:01.527745 IP 10.230.174.77 > 10.200.251.119: ICMP echo reply, id 27447, seq 10755, length 40
21:55:06.518245 IP 10.200.251.119 > 10.230.174.77: ICMP echo request, id 27447, seq 10756, length 40
21:55:06.519021 IP 10.230.174.77 > 10.200.251.119: ICMP echo reply, id 27447, seq 10756, length 40

To address where traffic is dropped, fw monitor is running, and traffic is visible in one direction only:

[Expert@FW002A:0]# fw monitor -e "accept host(10.230.174.77) and host(10.200.251.119);"
PPAK 0: Get before set operation succeeded of fwmonitor_kiss_enable
PPAK 0: Get before set operation succeeded of simple_debug_filter_off
PPAK 0: Get before set operation succeeded of kiss_debug_force_kdprintf_enable
PPAK 0: Get before set operation succeeded of fwmonitorfreebufs
**************************************************************  NOTE  **************************************************************
*** Using "-e" filter will not monitor accelerated traffic. To monitor and filter accelerated traffic please use the "-F" filter ***
************************************************************************************************************************************
FW monitor will record only ip & transport layers in a packet
For capturing the whole packet please do -w
PPAK 0: Get before set operation succeeded of fwmonitor_ppak_all_position
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
PPAK 0: Get before set operation succeeded of fwmonitormaxpacket
PPAK 0: Get before set operation succeeded of fwmonitormask
PPAK 0: Get before set operation succeeded of printuuid
PPAK 0: Get before set operation succeeded of fwmonitorallocbufs
[vs_0][fw_0] eth4.2264:i[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31129
ICMP: type=8 code=0 echo request id=27447 seq=10892
[vs_0][fw_0] eth4.2264:I[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31129
ICMP: type=8 code=0 echo request id=27447 seq=10892
[vs_0][fw_0] bond1.3031:o[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31129
ICMP: type=8 code=0 echo request id=27447 seq=10892
[vs_0][fw_0] bond1.3031:O[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31129
ICMP: type=8 code=0 echo request id=27447 seq=10892

[vs_0][fw_0] eth4.2264:i[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31130
ICMP: type=8 code=0 echo request id=27447 seq=10893
[vs_0][fw_0] eth4.2264:I[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31130
ICMP: type=8 code=0 echo request id=27447 seq=10893
[vs_0][fw_0] bond1.3031:o[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31130
ICMP: type=8 code=0 echo request id=27447 seq=10893
[vs_0][fw_0] bond1.3031:O[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31130
ICMP: type=8 code=0 echo request id=27447 seq=10893

[vs_0][fw_0] eth4.2264:i[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31131
ICMP: type=8 code=0 echo request id=27447 seq=10894
[vs_0][fw_0] eth4.2264:I[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31131
ICMP: type=8 code=0 echo request id=27447 seq=10894
[vs_0][fw_0] bond1.3031:o[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31131
ICMP: type=8 code=0 echo request id=27447 seq=10894
[vs_0][fw_0] bond1.3031:O[44]: 10.200.251.119 -> 10.230.174.77 (ICMP) len=60 id=31131
ICMP: type=8 code=0 echo request id=27447 seq=10894

monitor: caught sig 2
monitor: unloading
PPAK 0: Get before set operation succeeded of fwmonitor_kiss_enable
PPAK 0: Get before set operation succeeded of simple_debug_filter_off
PPAK 0: Get before set operation succeeded of kiss_debug_force_kdprintf_enable
PPAK 0: Get before set operation succeeded of fwmonitorfreebufs

Also "fw ctl zdebug + drop" has been run during test, nothing related visible to be dropped.

Have you encountered similar behavior? I have failed to find, in which position of packet flow tcpdump is placed, to help explain, why traffic is reaching the gateway, but not be processed by firewall.

Thank you.

_Val_ · ‎2022-07-21

Which version are you running on the GW?

MartinOles · ‎2022-07-21

[Expert@FW002A:0]# cpinfo -y all

This is Check Point CPinfo Build 914000227 for GAIA
[IDA]
        No hotfixes..
[MGMT]
        HOTFIX_R80_30_JUMBO_HF_MAIN     Take:  251
[CPFC]
        HOTFIX_R80_30_JUMBO_HF_MAIN     Take:  251
[FW1]
        HOTFIX_PUBLIC_CLOUD_CA_BUNDLE_AUTOUPDATE
        HOTFIX_MAAS_TUNNEL_AUTOUPDATE
        HOTFIX_R80_30_JUMBO_HF_MAIN     Take:  251

FW1 build number:
This is Check Point's software version R80.30 - Build 224
kernel: R80.30 - Build 231
[SecurePlatform]
        HOTFIX_R80_30_JUMBO_HF_MAIN     Take:  251
[CPinfo]
        No hotfixes..
[PPACK]
        HOTFIX_R80_30_JUMBO_HF_MAIN     Take:  251
[CVPN]
        HOTFIX_R80_30_JUMBO_HF_MAIN     Take:  251
[CPUpdates]
        BUNDLE_PUBLIC_CLOUD_CA_BUNDLE_AUTOUPDATE        Take:  14
        BUNDLE_CPSDC_AUTOUPDATE Take:  19
        BUNDLE_MAAS_TUNNEL_AUTOUPDATE   Take:  61
        BUNDLE_HCP_AUTOUPDATE   Take:  49
        BUNDLE_R80_30_JUMBO_HF_MAIN     Take:  251
[CPDepInst]
        No hotfixes..
[AutoUpdater]
        No hotfixes..
[hcp_wrapper]
        HOTFIX_HCP_AUTOUPDATE
[DIAG]
        No hotfixes..
[cpsdc_wrapper]
        HOTFIX_CPSDC_AUTOUPDATE

_Val_ · ‎2022-07-21

According to both fw monitor and tcpdump from your output, the return packets/frames do not reach FW appliance.

Check on the destination that ARP entry for FW IP address actually corresponds to FW mac address on that interface. Also, just in case, flash ARP tables on the destination and adjacent switches.

I have seen a similar situation where a static arp entry was at fault when return frames could not reach the FW interface.

MartinOles · ‎2022-07-21

Destination IP is a server, no know static ARP entry according to our knowledge. But should be there "wrong ARP record" would not be visible then packet in tcpdump? ... ok, maybe it arrives as unknown unicast, such might explain why is visible in tcpdump (switching line card to promiscuous mode) and dropped just afterwards as destination MAC does not belongs to getaway. We will try to confirm it with next test.

Thank you.

_Val_ · ‎2022-07-21

The first step: run tcpdump on the destination server to see what the return ICMP looks like. It seems to be all good, add a sniffer on the same network, and run additional traces with capturing not just the packet, but the frame data with mac addresses for all. If any switch/router is in the middle, include those

Are you a member of CheckMates?

Packets visible in tcpdump but not in fw monitor