topic Re: Reduction in throughput (MB) when traffic passes through the firewall in Firewall and Security Management

Reduction in throughput (MB) when traffic passes through the firewall

jennyado — Wed, 22 Jan 2025 17:51:41 GMT

Hi,

I have the following network where a certain amount of 150MB was contracted on the L2L link.
Servers> Switch > Cluster > L2L link > Cluster> Switch > Servers

We performed throughput tests and noticed that the total MB is reduced to 60MB when the traffic passes through the FWs:Server > Switch > Cluster > L2L link > Cluster > Switch > Servers

We performed another test removing the FWs and noticed that the total 150MB is what we should have:
Server > Switch > L2L link > Switch > Servers

Therefore we conclude that the problem is in the FW since when the traffic passes through there we see a decrease in MB.
We have checked that the interfaces do not have problems at layer 1 level.

Any idea what's going on?

Re: Reduction in throughput (MB) when traffic passes through the firewall

the_rock — Thu, 16 Jan 2025 23:11:53 GMT

I would start with cpview and see if there is anything unusual there. If not, run ethtool -S and also see from ifconfig output what errors show. You can also see this from interface config in web UI, just click on advanced or monitoring in upper right (cant recall the exact wording now).

Andy

Re: Reduction in throughput (MB) when traffic passes through the firewall

Chris_Atkinson — Fri, 17 Jan 2025 02:47:46 GMT

How is the test being performed - are multiple concurrent connections used?

Which appliance do you have and which version/JHF?

What security blades are enabled, do you observe high CPU?

Re: Reduction in throughput (MB) when traffic passes through the firewall

AkosBakos — Fri, 17 Jan 2025 07:17:56 GMT

Hi @jennyado

And the third tip is the MTU. Tipically the MTU is 1500 on the interfaces. Do you have the same setting?

Re: Reduction in throughput (MB) when traffic passes through the firewall

jennyado — Fri, 17 Jan 2025 17:10:11 GMT

We have reviewed everything you mentioned but everything is fine in this part

Re: Reduction in throughput (MB) when traffic passes through the firewall

jennyado — Wed, 22 Jan 2025 17:52:15 GMT

It is a dedicated link. At the request of a third-party provider, they need the bandwidth to be 150 MB and to go through an encrypted channel.

This channel is only used for a synchronization of database servers. On the channel it goes like this:
Server -> SW -> Cluster -> L2L link -> Cluster -> SW -> Server

The problem is that when traffic goes through the VPN, throughput is significantly reduced. Better said, when it passes through the FW it decreases since tests were also carried out where they sent the traffic through another interface that does not use VPN and the traffic continues to decrease.

If they "remove the fw" by directly passing the traffic through the switch, something like this:
Server -> SW -> L2L link -> SW -> Server
It is seen that the throughput increases.

They have R81.20 JHF76 and it is a 7000 appliance

It is not observed that the CPU is high.

Re: Reduction in throughput (MB) when traffic passes through the firewall

the_rock — Fri, 17 Jan 2025 17:20:55 GMT

I see the point Chris made...can you send us output of enabled_blades from the fw, like below?

Andy

[Expert@CP-GW:0]# enabled_blades
fw vpn cvpn urlf appi ips identityServer anti_bot content_awareness qos mon
[Expert@CP-GW:0]#

Re: Reduction in throughput (MB) when traffic passes through the firewall

jennyado — Fri, 17 Jan 2025 17:38:08 GMT

I don't have access to the FWs. I only have the cpinfo for these, but it seems to me that with the following we can see which blades are activated:

Re: Reduction in throughput (MB) when traffic passes through the firewall

jennyado — Fri, 17 Jan 2025 17:57:09 GMT

Yes, I have the same setting

Re: Reduction in throughput (MB) when traffic passes through the firewall

the_rock — Fri, 17 Jan 2025 17:58:54 GMT

Also, see if below may help.

Andy

https://community.checkpoint.com/t5/Security-Gateways/81-20-Performance-CPU-issue/td-p/214709

Re: Reduction in throughput (MB) when traffic passes through the firewall

the_rock — Fri, 17 Jan 2025 18:37:55 GMT

Maybe worth mentioning, if VPN tunnels are involved, below also would be relevant.

Andy

https://support.checkpoint.com/results/sk/sk73980

Re: Reduction in throughput (MB) when traffic passes through the firewall

PhoneBoy — Fri, 17 Jan 2025 22:50:01 GMT

How precisely is the traffic generated to test throughput?
What are the characteristics of this traffic?
What does the physical connectivity look like?
What rule is the traffic matching on?
The more details you can provide, the more likely we're going to be able to help.

Re: Reduction in throughput (MB) when traffic passes through the firewall

genisis__ — Sat, 18 Jan 2025 12:03:06 GMT

Another thing to check, and it may likely have been done:

fwaccel stats -s

fw ctl affinity -l -a -v -r

netstat -i

Re: Reduction in throughput (MB) when traffic passes through the firewall

sjni01 — Wed, 22 Jan 2025 16:37:54 GMT

Hello @jennyado, I saw your comment regarding the discarding of L1 troubles. Please provide the output from each firewall and ensure you specifically highlight the interfaces where there is an L2 link using the command #sar -n EDEV.

Re: Reduction in throughput (MB) when traffic passes through the firewall

jennyado — Wed, 22 Jan 2025 17:50:35 GMT

I’ve been reviewing the eth1 interface statistics in more detail, which is the one used to manage this traffic, and I’ve noticed it has overruns and drops.

I’ve also checked that the ring size for these interfaces on each of the firewalls is set to 256. Could this be causing a decrease in the bandwidth?

ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 1024

ethtool -S eth1
NIC statistics:
rx_packets: 33281913246
tx_packets: 33328419871
rx_bytes: 19499354566426
tx_bytes: 31128961135376
rx_broadcast: 10525135
tx_broadcast: 2865716
rx_multicast: 366448181
tx_multicast: 92
multicast: 366448181
collisions: 0
rx_crc_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 19499354566426
tx_dma_out_of_sync: 0
lro_aggregated: 0
lro_flushed: 0
tx_smbus: 0
rx_smbus: 0
dropped_smbus: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_hwtstamp_timeouts: 0
rx_hwtstamp_cleared: 0
rx_errors: 0
tx_errors: 0
tx_dropped: 0
rx_length_errors: 0
rx_over_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 10414
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_queue_0_packets: 22163685367
tx_queue_0_bytes: 27454953867297
tx_queue_0_restart: 205
tx_queue_1_packets: 3710893626
tx_queue_1_bytes: 1177179673622
tx_queue_1_restart: 4
tx_queue_2_packets: 170956434
tx_queue_2_bytes: 26039849200
tx_queue_2_restart: 1
tx_queue_3_packets: 22977570
tx_queue_3_bytes: 5991192278
tx_queue_3_restart: 0
tx_queue_4_packets: 3584766673
tx_queue_4_bytes: 1075935552368
tx_queue_4_restart: 3
tx_queue_5_packets: 3501026893
tx_queue_5_bytes: 1098124542577
tx_queue_5_restart: 4
tx_queue_6_packets: 155095793
tx_queue_6_bytes: 20664541636
tx_queue_6_restart: 0
tx_queue_7_packets: 19012812
tx_queue_7_bytes: 3258882810
tx_queue_7_restart: 0
rx_queue_0_packets: 10834354627
rx_queue_0_bytes: 5846924618988
rx_queue_0_drops: 3702
rx_queue_0_csum_err: 551
rx_queue_0_alloc_failed: 0
rx_queue_1_packets: 4427755449
rx_queue_1_bytes: 3613105372007
rx_queue_1_drops: 2853
rx_queue_1_csum_err: 527
rx_queue_1_alloc_failed: 0
rx_queue_2_packets: 384852344
rx_queue_2_bytes: 503988561696
rx_queue_2_drops: 0
rx_queue_2_csum_err: 10
rx_queue_2_alloc_failed: 0
rx_queue_3_packets: 24838989
rx_queue_3_bytes: 25435851414
rx_queue_3_drops: 0
rx_queue_3_csum_err: 12
rx_queue_3_alloc_failed: 0
rx_queue_4_packets: 12371906883
rx_queue_4_bytes: 5082961009940
rx_queue_4_drops: 1675
rx_queue_4_csum_err: 564
rx_queue_4_alloc_failed: 0
rx_queue_5_packets: 4952681509
rx_queue_5_bytes: 3827003759534
rx_queue_5_drops: 2184
rx_queue_5_csum_err: 510
rx_queue_5_alloc_failed: 0
rx_queue_6_packets: 242992997
rx_queue_6_bytes: 295919177519
rx_queue_6_drops: 0
rx_queue_6_csum_err: 10
rx_queue_6_alloc_failed: 0
rx_queue_7_packets: 42513806
rx_queue_7_bytes: 37813168209
rx_queue_7_drops: 0
rx_queue_7_csum_err: 14
rx_queue_7_alloc_failed: 0

ifconfig -a
eth1 Link encap:Ethernet HWaddr 00:1C:7F:C0:AD:53
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:33282211602 errors:0 dropped:0 overruns:10414 frame:0
TX packets:33328754328 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19233245027003 (17.4 TiB) TX bytes:30862483517204 (28.0 TiB)

On both firewalls, I see that the statistics show values in the same overruns and drops counters.

Re: Reduction in throughput (MB) when traffic passes through the firewall

jennyado — Wed, 22 Jan 2025 18:03:42 GMT

They have a dedicated L2L link providing 150 MB. The tests performed were as follows:

They connected their laptop directly to the switch, bypassing the firewall, and observed that the bandwidth was 150 MB. I don't have information about the tool they used to measure the bandwidth.
They connected their laptop directly to the firewall, and here they reported that the bandwidth dropped to 60 MB. They have a VPN connecting site A to site B through the L2L, as they want to synchronize a database over an encrypted channel. To rule out the encryption as the cause, they excluded the database services from their VPN, but they still get 60 MB.

I’ve reviewed the eth1 interface statistics in more detail and noticed overruns and drops. This interface is where they have the VLAN to manage the mentioned traffic. Additionally, we observed that the ring size is set to 256, which we believe could be causing the decrease in traffic. The information regarding the statistics has been shared earlier in the post.

Re: Reduction in throughput (MB) when traffic passes through the firewall

Lesley — Wed, 22 Jan 2025 19:02:37 GMT

It is OK to increase RX buffers from 256 to 1024

Here a how-to:

https://support.checkpoint.com/results/sk/sk42181

When you change the buffer interface goes down / up! Takes a few seconds.

Re: Reduction in throughput (MB) when traffic passes through the firewall

jennyado — Wed, 22 Jan 2025 19:07:05 GMT

Is there any possibility that any other issues or side effects could arise when making this change to the RX buffers, apart from the brief interface up/down interruption?

Re: Reduction in throughput (MB) when traffic passes through the firewall

Lesley — Wed, 22 Jan 2025 21:05:38 GMT

You should be all good! 1024 is still little amount.

Increase the size of the ring buffer to 1024 descriptors.
If needed, keep increasing the size to 2048 descriptors, and to the maximum of 4096 descriptors.

What are the side effects of increasing the buffer? What are the downsides of implementing the buffer increase?

Changing the ring buffer size with 'ethtool' takes the interface momentarily off-line. As a result, it might cause a short traffic outage or a failover in cluster configurations.

Increasing the ring buffer size causes the interface to store more data before sending an interrupt to the CPU to process that data. The longer (bigger) the queue, the longer the time it will take for the packets to be de-queued (processed). This will cause latency in traffic, especially if the majority of the traffic are small packets (e.g., ICMP packets, some UDP datagrams). The main reason to increase the ring buffer size on an interface is to lower the amount of interrupts sent to the CPU. Due to the inevitable latency factor, increasing the ring buffer size must be done gradually, followed by a test period.

Re: Reduction in throughput (MB) when traffic passes through the firewall

the_rock — Thu, 23 Jan 2025 00:47:08 GMT

I hope that would help in this case.

Andy