"UDP checksum is incorrect" - 100s of IPv6 DROPs in fw.log - TP/IPS responsible?

[Jerry]

hi folks

quick one

one of my customers just upgraded to R82 last night and found in fw.log 100s of drops due to "UDP checksum is incorrect".

knowing how UDP works I presume that TP/IPS is to blame but which protection is responsible for that? 

any clues?

[Timothy_Hall]

Pretty sure this offload check is handled by the NIC driver itself, which was significantly updated by Check Point for R82 due to the new Linux kernel.  Please post output of these commands run from expert mode:

ethtool -k  (interface)

ethtool -i  (interface)

ethtool -S  (interface)

[Jerry]

Thanks Tim, I do appreciate your response, please see below:

[Expert@cp:0]# ethtool -k bond1.102
Features for bond1.102:
rx-checksumming: off [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [requested on]
tx-checksum-sctp: off [requested on]
scatter-gather: off
tx-scatter-gather: off [requested on]
tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
tx-tcp-ecn-segmentation: off [requested on]
tx-tcp-mangleid-segmentation: off [requested on]
tx-tcp6-segmentation: off [requested on]
udp-fragmentation-offload: off
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: off [requested on]
tx-gre-csum-segmentation: off [requested on]
tx-ipxip4-segmentation: off [requested on]
tx-ipxip6-segmentation: off [requested on]
tx-udp_tnl-segmentation: off [requested on]
tx-udp_tnl-csum-segmentation: off [requested on]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: on
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: on
rx-udp-gro-forwarding: off
rx-gro-list: off
tls-hw-rx-offload: off [fixed]
fcoe-mtu: off [requested on]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
[Expert@cp:0]# ethtool -i bond1.102
driver: 802.1Q VLAN Support
version: 1.8
firmware-version: N/A
expansion-rom-version:
bus-info:
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
[Expert@cp0]# ethtool -S bond1.102
no stats available
[Expert@cp:0]#

[Timothy_Hall]

Please post the log card with sensitive details redacted.  Also kind of sounds like this:

https://www.reddit.com/r/pcmasterrace/comments/wrpph1/what_does_tcp_and_udp_checksum_offload_do/

There have been numerous issues with NIC offloads breaking things in the past on Check Point gateways:

sk114804: How to troubleshoot "Interface Active Check" pnote on ClusterXL

sk101547: Policy installation fails due timeout on Security Gateway with Broadcom NetXtreme interfac...

RX out of buffer drops on 25/40/100G (Mellanox) interfaces

Also please post the three command outputs for one of the underlying physical interfaces that compose the bond, not the bond interface itself.  Probably Mellanox/NVIDIA.

[Jerry]

Thanks Tim, wonder what UDP checksum offload have to do with UDP checksum incorrect error ... anyway, here is the log card.

I did workaround to drop that port for ALL sources (ipv4/ipv6) but still unable to understand how NIC drivers may impact just THAT port on layer 4 and mainly on IPv6. Any clues?

[Jerry]

re. output from hysical interfaces forming a bond:

[Expert@cp:0]# ethtool -k eth1-01
Features for eth1-01:
rx-checksumming: on
tx-checksumming: off
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: off
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
rx-udp-gro-forwarding: off
rx-gro-list: off
tls-hw-rx-offload: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
[Expert@cp:0]# ethtool -i eth1-01
driver: ixgbe
version: 5.15.2 (V1.0.1_ckp)
firmware-version: 0x800000cb
expansion-rom-version:
bus-info: 0000:87:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
[Expert@cp:0]# ethtool -S eth1-01
NIC statistics:
rx_packets: 10183095
tx_packets: 10102675
rx_bytes: 9857574189
tx_bytes: 9280350076
rx_errors: 1226
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 116807
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 10183169
tx_pkts_nic: 10102732
rx_bytes_nic: 9938585795
tx_bytes_nic: 9361479747
lsc_int: 6
tx_busy: 0
non_eop_descs: 0
broadcast: 3
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_length_errors: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_errors: 1226
alloc_rx_page: 0
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 0
fdir_overflow: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_skipped: 0
rx_hwtstamp_cleared: 0
tx_queue_0_packets: 1352366
tx_queue_0_bytes: 1028641399
tx_queue_1_packets: 5671045
tx_queue_1_bytes: 6022464309
tx_queue_2_packets: 0
tx_queue_2_bytes: 0
tx_queue_3_packets: 3
tx_queue_3_bytes: 588
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_5_packets: 1
tx_queue_5_bytes: 124
tx_queue_6_packets: 2
tx_queue_6_bytes: 248
tx_queue_7_packets: 0
tx_queue_7_bytes: 0
tx_queue_8_packets: 1668124
tx_queue_8_bytes: 1356552814
tx_queue_9_packets: 1411131
tx_queue_9_bytes: 872690244
tx_queue_10_packets: 2
tx_queue_10_bytes: 226
tx_queue_11_packets: 0
tx_queue_11_bytes: 0
tx_queue_12_packets: 0
tx_queue_12_bytes: 0
tx_queue_13_packets: 1
tx_queue_13_bytes: 124
tx_queue_14_packets: 0
tx_queue_14_bytes: 0
tx_queue_15_packets: 0
tx_queue_15_bytes: 0
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_17_packets: 0
tx_queue_17_bytes: 0
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_19_packets: 0
tx_queue_19_bytes: 0
tx_queue_20_packets: 0
tx_queue_20_bytes: 0
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_22_packets: 0
tx_queue_22_bytes: 0
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_25_packets: 0
tx_queue_25_bytes: 0
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_27_packets: 0
tx_queue_27_bytes: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_29_packets: 0
tx_queue_29_bytes: 0
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_32_packets: 0
tx_queue_32_bytes: 0
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_34_packets: 0
tx_queue_34_bytes: 0
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_36_packets: 0
tx_queue_36_bytes: 0
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_40_packets: 0
tx_queue_40_bytes: 0
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_42_packets: 0
tx_queue_42_bytes: 0
tx_queue_43_packets: 0
tx_queue_43_bytes: 0
tx_queue_44_packets: 0
tx_queue_44_bytes: 0
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_48_packets: 0
tx_queue_48_bytes: 0
tx_queue_49_packets: 0
tx_queue_49_bytes: 0
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_53_packets: 0
tx_queue_53_bytes: 0
tx_queue_54_packets: 0
tx_queue_54_bytes: 0
tx_queue_55_packets: 0
tx_queue_55_bytes: 0
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_58_packets: 0
tx_queue_58_bytes: 0
tx_queue_59_packets: 0
tx_queue_59_bytes: 0
tx_queue_60_packets: 0
tx_queue_60_bytes: 0
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_62_packets: 0
tx_queue_62_bytes: 0
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
rx_queue_0_packets: 1227664
rx_queue_0_bytes: 714561911
rx_queue_1_packets: 2443594
rx_queue_1_bytes: 1938929258
rx_queue_2_packets: 0
rx_queue_2_bytes: 0
rx_queue_3_packets: 0
rx_queue_3_bytes: 0
rx_queue_4_packets: 41
rx_queue_4_bytes: 3878
rx_queue_5_packets: 0
rx_queue_5_bytes: 0
rx_queue_6_packets: 0
rx_queue_6_bytes: 0
rx_queue_7_packets: 0
rx_queue_7_bytes: 0
rx_queue_8_packets: 1845156
rx_queue_8_bytes: 1769636197
rx_queue_9_packets: 4666590
rx_queue_9_bytes: 5434438041
rx_queue_10_packets: 0
rx_queue_10_bytes: 0
rx_queue_11_packets: 0
rx_queue_11_bytes: 0
rx_queue_12_packets: 50
rx_queue_12_bytes: 4904
rx_queue_13_packets: 0
rx_queue_13_bytes: 0
rx_queue_14_packets: 0
rx_queue_14_bytes: 0
rx_queue_15_packets: 0
rx_queue_15_bytes: 0
rx_queue_16_packets: 0
rx_queue_16_bytes: 0
rx_queue_17_packets: 0
rx_queue_17_bytes: 0
rx_queue_18_packets: 0
rx_queue_18_bytes: 0
rx_queue_19_packets: 0
rx_queue_19_bytes: 0
rx_queue_20_packets: 0
rx_queue_20_bytes: 0
rx_queue_21_packets: 0
rx_queue_21_bytes: 0
rx_queue_22_packets: 0
rx_queue_22_bytes: 0
rx_queue_23_packets: 0
rx_queue_23_bytes: 0
rx_queue_24_packets: 0
rx_queue_24_bytes: 0
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_26_packets: 0
rx_queue_26_bytes: 0
rx_queue_27_packets: 0
rx_queue_27_bytes: 0
rx_queue_28_packets: 0
rx_queue_28_bytes: 0
rx_queue_29_packets: 0
rx_queue_29_bytes: 0
rx_queue_30_packets: 0
rx_queue_30_bytes: 0
rx_queue_31_packets: 0
rx_queue_31_bytes: 0
rx_queue_32_packets: 0
rx_queue_32_bytes: 0
rx_queue_33_packets: 0
rx_queue_33_bytes: 0
rx_queue_34_packets: 0
rx_queue_34_bytes: 0
rx_queue_35_packets: 0
rx_queue_35_bytes: 0
rx_queue_36_packets: 0
rx_queue_36_bytes: 0
rx_queue_37_packets: 0
rx_queue_37_bytes: 0
rx_queue_38_packets: 0
rx_queue_38_bytes: 0
rx_queue_39_packets: 0
rx_queue_39_bytes: 0
rx_queue_40_packets: 0
rx_queue_40_bytes: 0
rx_queue_41_packets: 0
rx_queue_41_bytes: 0
rx_queue_42_packets: 0
rx_queue_42_bytes: 0
rx_queue_43_packets: 0
rx_queue_43_bytes: 0
rx_queue_44_packets: 0
rx_queue_44_bytes: 0
rx_queue_45_packets: 0
rx_queue_45_bytes: 0
rx_queue_46_packets: 0
rx_queue_46_bytes: 0
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_48_packets: 0
rx_queue_48_bytes: 0
rx_queue_49_packets: 0
rx_queue_49_bytes: 0
rx_queue_50_packets: 0
rx_queue_50_bytes: 0
rx_queue_51_packets: 0
rx_queue_51_bytes: 0
rx_queue_52_packets: 0
rx_queue_52_bytes: 0
rx_queue_53_packets: 0
rx_queue_53_bytes: 0
rx_queue_54_packets: 0
rx_queue_54_bytes: 0
rx_queue_55_packets: 0
rx_queue_55_bytes: 0
rx_queue_56_packets: 0
rx_queue_56_bytes: 0
rx_queue_57_packets: 0
rx_queue_57_bytes: 0
rx_queue_58_packets: 0
rx_queue_58_bytes: 0
rx_queue_59_packets: 0
rx_queue_59_bytes: 0
rx_queue_60_packets: 0
rx_queue_60_bytes: 0
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_62_packets: 0
rx_queue_62_bytes: 0
rx_queue_63_packets: 0
rx_queue_63_bytes: 0
tx_pb_0_pxon: 0
tx_pb_0_pxoff: 0
tx_pb_1_pxon: 0
tx_pb_1_pxoff: 0
tx_pb_2_pxon: 0
tx_pb_2_pxoff: 0
tx_pb_3_pxon: 0
tx_pb_3_pxoff: 0
tx_pb_4_pxon: 0
tx_pb_4_pxoff: 0
tx_pb_5_pxon: 0
tx_pb_5_pxoff: 0
tx_pb_6_pxon: 0
tx_pb_6_pxoff: 0
tx_pb_7_pxon: 0
tx_pb_7_pxoff: 0
rx_pb_0_pxon: 0
rx_pb_0_pxoff: 0
rx_pb_1_pxon: 0
rx_pb_1_pxoff: 0
rx_pb_2_pxon: 0
rx_pb_2_pxoff: 0
rx_pb_3_pxon: 0
rx_pb_3_pxoff: 0
rx_pb_4_pxon: 0
rx_pb_4_pxoff: 0
rx_pb_5_pxon: 0
rx_pb_5_pxoff: 0
rx_pb_6_pxon: 0
rx_pb_6_pxoff: 0
rx_pb_7_pxon: 0
rx_pb_7_pxoff: 0

[Jerry]

Tim, please look at the ERRORS on the physical interfaces as well, see below:

indeed looks like the driver issue 😞 hope we can get it sorted soon ...

---

eth1-01 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:10190946 errors:1226 dropped:0 overruns:0 frame:0
TX packets:10256551 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9859384679 (9.1 GiB) TX bytes:9328404302 (8.6 GiB)

eth1-02 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:11738071 errors:1093 dropped:0 overruns:0 frame:0
TX packets:19270526 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9686310294 (9.0 GiB) TX bytes:20261030202 (18.8 GiB)

eth1-03 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:17722379 errors:1122 dropped:0 overruns:0 frame:0
TX packets:17975267 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:17073204816 (15.9 GiB) TX bytes:14036782625 (13.0 GiB)

eth1-04 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:20361399 errors:1123 dropped:0 overruns:0 frame:0
TX packets:8553512 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:16408311053 (15.2 GiB) TX bytes:6937213841 (6.4 GiB)

eth2-01 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:10020214 errors:996 dropped:0 overruns:0 frame:0
TX packets:11381828 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:10450927382 (9.7 GiB) TX bytes:9138580946 (8.5 GiB)

eth2-02 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:12779546 errors:1089 dropped:0 overruns:0 frame:0
TX packets:14954103 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:11709983777 (10.9 GiB) TX bytes:14467458109 (13.4 GiB)

[Timothy_Hall]

Yep, those errors are checksum offload errors.  Sounds like Shai had you turn that off from the Check Point code end as opposed to on the direct NIC settings.

[Jerry]

do you remember Tim how do use ethtool and CLEAR of the stats I mean all of the stats from the interfaces?

I could see "rx_csum_offload_errors: 1226" within the output and now it's clear where we are. many thanks !

Cheers!

[Timothy_Hall]

The counters can be cleared without a reboot but technically it will cause a very brief outage on the interface as its driver is reloaded by Gaia.  Normally it is so quick that ClusterXL won't even see it; run these commands from expert mode:

ifdown eth1-01;ifup eth1-01

[Jerry]

did that and reboot anyway as that is my LAB device for R82 tests.

all works now let's see if errors in counters increases as well as fw.log drops continues ...

Cheers Tim!

[Jerry]

bad news after a night and flows going on, the physical interfaces are again having the error's we've been talking about yesterday, please see below: any thoughts folks?

eth1-01 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:5668515 errors:691 dropped:0 overruns:0 frame:0
TX packets:6207346 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4239526335 (3.9 GiB) TX bytes:4881635141 (4.5 GiB)

eth1-02 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:7510273 errors:675 dropped:0 overruns:0 frame:0
TX packets:8202791 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3520307127 (3.2 GiB) TX bytes:3845844972 (3.5 GiB)

eth1-03 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:4773128 errors:769 dropped:0 overruns:0 frame:0
TX packets:6777600 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3790682170 (3.5 GiB) TX bytes:5406914065 (5.0 GiB)

eth1-04 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:5101784 errors:719 dropped:0 overruns:0 frame:0
TX packets:3769340 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3916296128 (3.6 GiB) TX bytes:2608019482 (2.4 GiB)

eth2-01 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:5745856 errors:675 dropped:0 overruns:0 frame:0
TX packets:4967433 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4156484916 (3.8 GiB) TX bytes:3228825843 (3.0 GiB)

eth2-02 Link encap:Ethernet HWaddr 00:1C:7F:69:35:BC
UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
RX packets:6050805 errors:711 dropped:0 overruns:0 frame:0
TX packets:4878890 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4904375317 (4.5 GiB) TX bytes:3779688508 (3.5 GiB)

[Jerry]

https://www.ducea.com/2006/09/08/resetting-ifconfig-counters/

that doesn't help I'm afraid ... 😞 

[Jerry]

interesting is that that error in logs is shown only heading towards the G-Suite (Google Workspace) destinations IPv6 whilst IPv4 is just fine ...

[the_rock]

Wow M8, very brave of that customer to upgrade so quick...

Andy

[Jerry]

yes I'm very brave indeed buddy to be that customer myself 😄 

[the_rock]

Okay...that surely means I can NOT be stroppy with you 😉

Andy

[Jerry]

seems you've planned that "activity" already mate but to the point Andy,

problem is that through R82 I've made serious traffic tests with Google Workspace apps and firewall fw.log shows ridiculous message as "explanation" of what he did hence my post. If Tim is right then we've got an issue. I've temp. dropped udp/19305 by the ruleset and now I've got no drops at all as it is silenced rule hence no show of that error again however, I believe something has changed on the kernel 4 so that's one of my reasons I had to share it with you guys 🙂

[the_rock]

You know its light hearted joke, I have high respect for you, always!

Anyway, back to the issue. I agree that if what Tim advised is true, there is a problem. I think doing what you did, it sounds more really "masking" the issue. 

Can you send what it shows with below command?

Andy

[Expert@R82:0]# fw ver -k
This is Check Point's software version R82 - Build 151
kernel: R82 - Build 137

[Expert@R82:0]# uname -a
Linux R82 4.18.0-372.9.1cpx86_64 #1 SMP Sat Oct 12 01:24:43 IDT 2024 x86_64 x86_64 x86_64 GNU/Linux

[Jerry]

agree! you're spot on Andy as always 🙂 here it comes:

[Expert@cp:0]# fw ver -k
This is Check Point's software version R82 - Build 151
kernel: R82 - Build 137
[Expert@cp:0]# uname -a
Linux cp15k 4.18.0-372.9.1cpx86_64 #1 SMP Sat Oct 12 01:24:43 IDT 2024 x86_64 x86_64 x86_64 GNU/Linux
[Expert@cp:0]#

[the_rock]

Did you end up opening TAC case?

[Jerry]

nop 🙂 should I really ?

[the_rock]

I would, because R82 is GA now and when you open the case, they even have that option you can select as the version, so they have to support it.

Andy

[Jerry]

now after the reboot all the stats of the bonded NIC looks as following (no errors as you'll see below): 

[Expert@cp:0]# ethtool -S eth1-01
NIC statistics:
rx_packets: 120602
tx_packets: 102632
rx_bytes: 87580233
tx_bytes: 60074563
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 1244
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 124135
tx_pkts_nic: 105380
rx_bytes_nic: 89627787
tx_bytes_nic: 62048558
lsc_int: 8
tx_busy: 0
non_eop_descs: 0
broadcast: 0
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_length_errors: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_csum_offload_errors: 0
alloc_rx_page: 0
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 0
fdir_miss: 0
fdir_overflow: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_skipped: 0
rx_hwtstamp_cleared: 0
tx_queue_0_packets: 23050
tx_queue_0_bytes: 4669165
tx_queue_1_packets: 53924
tx_queue_1_bytes: 44644505
tx_queue_2_packets: 554
tx_queue_2_bytes: 154924
tx_queue_3_packets: 532
tx_queue_3_bytes: 533224
tx_queue_4_packets: 189
tx_queue_4_bytes: 35733
tx_queue_5_packets: 140
tx_queue_5_bytes: 32870
tx_queue_6_packets: 191
tx_queue_6_bytes: 133814
tx_queue_7_packets: 209
tx_queue_7_bytes: 62128
tx_queue_8_packets: 15549
tx_queue_8_bytes: 6628038
tx_queue_9_packets: 6803
tx_queue_9_bytes: 2916961
tx_queue_10_packets: 722
tx_queue_10_bytes: 106419
tx_queue_11_packets: 156
tx_queue_11_bytes: 32257
tx_queue_12_packets: 169
tx_queue_12_bytes: 35748
tx_queue_13_packets: 199
tx_queue_13_bytes: 33805
tx_queue_14_packets: 136
tx_queue_14_bytes: 31000
tx_queue_15_packets: 109
tx_queue_15_bytes: 23972
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_17_packets: 0
tx_queue_17_bytes: 0
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_19_packets: 0
tx_queue_19_bytes: 0
tx_queue_20_packets: 0
tx_queue_20_bytes: 0
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_22_packets: 0
tx_queue_22_bytes: 0
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_25_packets: 0
tx_queue_25_bytes: 0
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_27_packets: 0
tx_queue_27_bytes: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_29_packets: 0
tx_queue_29_bytes: 0
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_32_packets: 0
tx_queue_32_bytes: 0
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_34_packets: 0
tx_queue_34_bytes: 0
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_36_packets: 0
tx_queue_36_bytes: 0
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_40_packets: 0
tx_queue_40_bytes: 0
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_42_packets: 0
tx_queue_42_bytes: 0
tx_queue_43_packets: 0
tx_queue_43_bytes: 0
tx_queue_44_packets: 0
tx_queue_44_bytes: 0
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_48_packets: 0
tx_queue_48_bytes: 0
tx_queue_49_packets: 0
tx_queue_49_bytes: 0
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_53_packets: 0
tx_queue_53_bytes: 0
tx_queue_54_packets: 0
tx_queue_54_bytes: 0
tx_queue_55_packets: 0
tx_queue_55_bytes: 0
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_58_packets: 0
tx_queue_58_bytes: 0
tx_queue_59_packets: 0
tx_queue_59_bytes: 0
tx_queue_60_packets: 0
tx_queue_60_bytes: 0
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_62_packets: 0
tx_queue_62_bytes: 0
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
rx_queue_0_packets: 9538
rx_queue_0_bytes: 2334163
rx_queue_1_packets: 28469
rx_queue_1_bytes: 26204517
rx_queue_2_packets: 277
rx_queue_2_bytes: 281260
rx_queue_3_packets: 73
rx_queue_3_bytes: 19830
rx_queue_4_packets: 226
rx_queue_4_bytes: 86747
rx_queue_5_packets: 207
rx_queue_5_bytes: 30344
rx_queue_6_packets: 172
rx_queue_6_bytes: 59011
rx_queue_7_packets: 121
rx_queue_7_bytes: 30756
rx_queue_8_packets: 73462
rx_queue_8_bytes: 54473549
rx_queue_9_packets: 6603
rx_queue_9_bytes: 3495483
rx_queue_10_packets: 141
rx_queue_10_bytes: 39893
rx_queue_11_packets: 193
rx_queue_11_bytes: 69144
rx_queue_12_packets: 265
rx_queue_12_bytes: 137921
rx_queue_13_packets: 653
rx_queue_13_bytes: 279280
rx_queue_14_packets: 92
rx_queue_14_bytes: 15866
rx_queue_15_packets: 110
rx_queue_15_bytes: 22469
rx_queue_16_packets: 0
rx_queue_16_bytes: 0
rx_queue_17_packets: 0
rx_queue_17_bytes: 0
rx_queue_18_packets: 0
rx_queue_18_bytes: 0
rx_queue_19_packets: 0
rx_queue_19_bytes: 0
rx_queue_20_packets: 0
rx_queue_20_bytes: 0
rx_queue_21_packets: 0
rx_queue_21_bytes: 0
rx_queue_22_packets: 0
rx_queue_22_bytes: 0
rx_queue_23_packets: 0
rx_queue_23_bytes: 0
rx_queue_24_packets: 0
rx_queue_24_bytes: 0
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_26_packets: 0
rx_queue_26_bytes: 0
rx_queue_27_packets: 0
rx_queue_27_bytes: 0
rx_queue_28_packets: 0
rx_queue_28_bytes: 0
rx_queue_29_packets: 0
rx_queue_29_bytes: 0
rx_queue_30_packets: 0
rx_queue_30_bytes: 0
rx_queue_31_packets: 0
rx_queue_31_bytes: 0
rx_queue_32_packets: 0
rx_queue_32_bytes: 0
rx_queue_33_packets: 0
rx_queue_33_bytes: 0
rx_queue_34_packets: 0
rx_queue_34_bytes: 0
rx_queue_35_packets: 0
rx_queue_35_bytes: 0
rx_queue_36_packets: 0
rx_queue_36_bytes: 0
rx_queue_37_packets: 0
rx_queue_37_bytes: 0
rx_queue_38_packets: 0
rx_queue_38_bytes: 0
rx_queue_39_packets: 0
rx_queue_39_bytes: 0
rx_queue_40_packets: 0
rx_queue_40_bytes: 0
rx_queue_41_packets: 0
rx_queue_41_bytes: 0
rx_queue_42_packets: 0
rx_queue_42_bytes: 0
rx_queue_43_packets: 0
rx_queue_43_bytes: 0
rx_queue_44_packets: 0
rx_queue_44_bytes: 0
rx_queue_45_packets: 0
rx_queue_45_bytes: 0
rx_queue_46_packets: 0
rx_queue_46_bytes: 0
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_48_packets: 0
rx_queue_48_bytes: 0
rx_queue_49_packets: 0
rx_queue_49_bytes: 0
rx_queue_50_packets: 0
rx_queue_50_bytes: 0
rx_queue_51_packets: 0
rx_queue_51_bytes: 0
rx_queue_52_packets: 0
rx_queue_52_bytes: 0
rx_queue_53_packets: 0
rx_queue_53_bytes: 0
rx_queue_54_packets: 0
rx_queue_54_bytes: 0
rx_queue_55_packets: 0
rx_queue_55_bytes: 0
rx_queue_56_packets: 0
rx_queue_56_bytes: 0
rx_queue_57_packets: 0
rx_queue_57_bytes: 0
rx_queue_58_packets: 0
rx_queue_58_bytes: 0
rx_queue_59_packets: 0
rx_queue_59_bytes: 0
rx_queue_60_packets: 0
rx_queue_60_bytes: 0
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_62_packets: 0
rx_queue_62_bytes: 0
rx_queue_63_packets: 0
rx_queue_63_bytes: 0
tx_pb_0_pxon: 0
tx_pb_0_pxoff: 0
tx_pb_1_pxon: 0
tx_pb_1_pxoff: 0
tx_pb_2_pxon: 0
tx_pb_2_pxoff: 0
tx_pb_3_pxon: 0
tx_pb_3_pxoff: 0
tx_pb_4_pxon: 0
tx_pb_4_pxoff: 0
tx_pb_5_pxon: 0
tx_pb_5_pxoff: 0
tx_pb_6_pxon: 0
tx_pb_6_pxoff: 0
tx_pb_7_pxon: 0
tx_pb_7_pxoff: 0
rx_pb_0_pxon: 0
rx_pb_0_pxoff: 0
rx_pb_1_pxon: 0
rx_pb_1_pxoff: 0
rx_pb_2_pxon: 0
rx_pb_2_pxoff: 0
rx_pb_3_pxon: 0
rx_pb_3_pxoff: 0
rx_pb_4_pxon: 0
rx_pb_4_pxoff: 0
rx_pb_5_pxon: 0
rx_pb_5_pxoff: 0
rx_pb_6_pxon: 0
rx_pb_6_pxoff: 0
rx_pb_7_pxon: 0
rx_pb_7_pxoff: 0

[shais]

Hi
Can you please open a task for this? we would like to investigate this issue.

In the meantime, you can set udp_is_verify_cksum to 0 to mitigate this 

[Jerry]

Thanks, I will but re. your line - in which file I have to add this line set udp_is_verify_cksum to 0 ?

[the_rock]

Hey fam,

Just do this.

Andy

[Expert@R82:0]# fw ctl set int udp_is_verify_cksum 0
[Expert@R82:0]# fw ctl get int udp_is_verify_cksum
udp_is_verify_cksum = 0
[Expert@R82:0]#

[Jerry]

done:

[Expert@cp:0]# fw ctl set int udp_is_verify_cksum 0
[Expert@cp:0]# fw ctl get int udp_is_verify_cksum
udp_is_verify_cksum = 0
[Expert@cp:0]#

[the_rock]

To make it permanent, do below buddy.

Andy

[Expert@R82:0]# fw ctl set -f int udp_is_verify_cksum 0
"fwkern.conf" was updated successfully
[Expert@R82:0]# more /opt/CPsuite-R82/fw1/boot/modules/fwkern.conf
udp_is_verify_cksum=0
[Expert@R82:0]#

[Jerry]

does it survive the reboot and should I reboot or just push the policy/DB ?