Reduction in throughput (MB) when traffic passes through the firewall

[JenniferYado]

Hi,

I have the following network where a certain amount of 150MB was contracted on the L2L link.
Servers> Switch > Cluster > L2L link > Cluster> Switch > Servers

We performed throughput tests and noticed that the total MB is reduced to 60MB when the traffic passes through the FWs:Server > Switch > Cluster > L2L link > Cluster > Switch > Servers

We performed another test removing the FWs and noticed that the total 150MB is what we should have:
Server > Switch > L2L link > Switch > Servers

Therefore we conclude that the problem is in the FW since when the traffic passes through there we see a decrease in MB.
We have checked that the interfaces do not have problems at layer 1 level.

Any idea what's going on?

[the_rock]

I would start with cpview and see if there is anything unusual there. If not, run ethtool -S and also see from ifconfig output what errors show. You can also see this from interface config in web UI, just click on advanced or monitoring in upper right (cant recall the exact wording now).

Andy

[JenniferYado]

We have reviewed everything you mentioned but everything is fine in this part

[JenniferYado]

I’ve been reviewing the eth1 interface statistics in more detail, which is the one used to manage this traffic, and I’ve noticed it has overruns and drops.

I’ve also checked that the ring size for these interfaces on each of the firewalls is set to 256. Could this be causing a decrease in the bandwidth?

ethtool -g eth1 
Ring parameters for eth1:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 1024

ethtool -S eth1
NIC statistics:
rx_packets: 33281913246
tx_packets: 33328419871
rx_bytes: 19499354566426
tx_bytes: 31128961135376
rx_broadcast: 10525135
tx_broadcast: 2865716
rx_multicast: 366448181
tx_multicast: 92
multicast: 366448181
collisions: 0
rx_crc_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 19499354566426
tx_dma_out_of_sync: 0
lro_aggregated: 0
lro_flushed: 0
tx_smbus: 0
rx_smbus: 0
dropped_smbus: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_hwtstamp_timeouts: 0
rx_hwtstamp_cleared: 0
rx_errors: 0
tx_errors: 0
tx_dropped: 0
rx_length_errors: 0
rx_over_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 10414
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_queue_0_packets: 22163685367
tx_queue_0_bytes: 27454953867297
tx_queue_0_restart: 205
tx_queue_1_packets: 3710893626
tx_queue_1_bytes: 1177179673622
tx_queue_1_restart: 4
tx_queue_2_packets: 170956434
tx_queue_2_bytes: 26039849200
tx_queue_2_restart: 1
tx_queue_3_packets: 22977570
tx_queue_3_bytes: 5991192278
tx_queue_3_restart: 0
tx_queue_4_packets: 3584766673
tx_queue_4_bytes: 1075935552368
tx_queue_4_restart: 3
tx_queue_5_packets: 3501026893
tx_queue_5_bytes: 1098124542577
tx_queue_5_restart: 4
tx_queue_6_packets: 155095793
tx_queue_6_bytes: 20664541636
tx_queue_6_restart: 0
tx_queue_7_packets: 19012812
tx_queue_7_bytes: 3258882810
tx_queue_7_restart: 0
rx_queue_0_packets: 10834354627
rx_queue_0_bytes: 5846924618988
rx_queue_0_drops: 3702
rx_queue_0_csum_err: 551
rx_queue_0_alloc_failed: 0
rx_queue_1_packets: 4427755449
rx_queue_1_bytes: 3613105372007
rx_queue_1_drops: 2853
rx_queue_1_csum_err: 527
rx_queue_1_alloc_failed: 0
rx_queue_2_packets: 384852344
rx_queue_2_bytes: 503988561696
rx_queue_2_drops: 0
rx_queue_2_csum_err: 10
rx_queue_2_alloc_failed: 0
rx_queue_3_packets: 24838989
rx_queue_3_bytes: 25435851414
rx_queue_3_drops: 0
rx_queue_3_csum_err: 12
rx_queue_3_alloc_failed: 0
rx_queue_4_packets: 12371906883
rx_queue_4_bytes: 5082961009940
rx_queue_4_drops: 1675
rx_queue_4_csum_err: 564
rx_queue_4_alloc_failed: 0
rx_queue_5_packets: 4952681509
rx_queue_5_bytes: 3827003759534
rx_queue_5_drops: 2184
rx_queue_5_csum_err: 510
rx_queue_5_alloc_failed: 0
rx_queue_6_packets: 242992997
rx_queue_6_bytes: 295919177519
rx_queue_6_drops: 0
rx_queue_6_csum_err: 10
rx_queue_6_alloc_failed: 0
rx_queue_7_packets: 42513806
rx_queue_7_bytes: 37813168209
rx_queue_7_drops: 0
rx_queue_7_csum_err: 14
rx_queue_7_alloc_failed: 0

ifconfig -a
eth1 Link encap:Ethernet HWaddr 00:1C:7F:C0:AD:53
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:33282211602 errors:0 dropped:0 overruns:10414 frame:0
TX packets:33328754328 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19233245027003 (17.4 TiB) TX bytes:30862483517204 (28.0 TiB)

On both firewalls, I see that the statistics show values in the same overruns and drops counters.

[sjni01]

Hello @JenniferYado, I saw your comment regarding the discarding of L1 troubles. Please provide the output from each firewall and ensure you specifically highlight the interfaces where there is an L2 link using the command #sar -n EDEV. 

[Chris_Atkinson]

How is the test being performed - are multiple concurrent connections used?

Which appliance do you have and which version/JHF?

What security blades are enabled, do you observe high CPU?

[JenniferYado]

It is a dedicated link. At the request of a third-party provider, they need the bandwidth to be 150 MB and to go through an encrypted channel.

This channel is only used for a synchronization of database servers. On the channel it goes like this:
Server -> SW -> Cluster -> L2L link -> Cluster -> SW -> Server

The problem is that when traffic goes through the VPN, throughput is significantly reduced. Better said, when it passes through the FW it decreases since tests were also carried out where they sent the traffic through another interface that does not use VPN and the traffic continues to decrease.

If they "remove the fw" by directly passing the traffic through the switch, something like this:
Server -> SW -> L2L link -> SW -> Server
It is seen that the throughput increases.

They have R81.20 JHF76 and it is a 7000 appliance 

It is not observed that the CPU is high.

[the_rock]

I see the point Chris made...can you send us output of enabled_blades from the fw, like below?

Andy

[Expert@CP-GW:0]# enabled_blades
fw vpn cvpn urlf appi ips identityServer anti_bot content_awareness qos mon
[Expert@CP-GW:0]#

[JenniferYado]

I don't have access to the FWs. I only have the cpinfo for these, but it seems to me that with the following we can see which blades are activated:

 

[PhoneBoy]

How precisely is the traffic generated to test throughput?
What are the characteristics of this traffic? 
What does the physical connectivity look like?
What rule is the traffic matching on?
The more details you can provide, the more likely we're going to be able to help.

[JenniferYado]

They have a dedicated L2L link providing 150 MB. The tests performed were as follows:

1. They connected their laptop directly to the switch, bypassing the firewall, and observed that the bandwidth was 150 MB. I don't have information about the tool they used to measure the bandwidth.
2. They connected their laptop directly to the firewall, and here they reported that the bandwidth dropped to 60 MB. They have a VPN connecting site A to site B through the L2L, as they want to synchronize a database over an encrypted channel. To rule out the encryption as the cause, they excluded the database services from their VPN, but they still get 60 MB.

I’ve reviewed the eth1 interface statistics in more detail and noticed overruns and drops. This interface is where they have the VLAN to manage the mentioned traffic. Additionally, we observed that the ring size is set to 256, which we believe could be causing the decrease in traffic. The information regarding the statistics has been shared earlier in the post.

[Lesley]

It is OK to increase RX buffers from 256 to 1024

Here a how-to:

https://support.checkpoint.com/results/sk/sk42181

When you change the buffer interface goes down / up! Takes a few seconds. 

[JenniferYado]

Is there any possibility that any other issues or side effects could arise when making this change to the RX buffers, apart from the brief interface up/down interruption?

[Lesley]

You should be all good! 1024 is still little amount. 

Increase the size of the ring buffer to 1024 descriptors.
If needed, keep increasing the size to 2048 descriptors, and to the maximum of 4096 descriptors.

What are the side effects of increasing the buffer? What are the downsides of implementing the buffe...

• Changing the ring buffer size with 'ethtool' takes the interface momentarily off-line. As a result, it might cause a short traffic outage or a failover in cluster configurations.
  
  
• Increasing the ring buffer size causes the interface to store more data before sending an interrupt to the CPU to process that data. The longer (bigger) the queue, the longer the time it will take for the packets to be de-queued (processed). This will cause latency in traffic, especially if the majority of the traffic are small packets (e.g., ICMP packets, some UDP datagrams). The main reason to increase the ring buffer size on an interface is to lower the amount of interrupts sent to the CPU. Due to the inevitable latency factor, increasing the ring buffer size must be done gradually, followed by a test period.

[the_rock]

I hope that would help in this case.

Andy

[JenniferYado]

If not, is there anything else to check at layer 1?

[the_rock]

I cant think of much apart from just making sure speed/duplex is good. You can try change the cables as a test.

Andy

[JozkoMrkvicka]

Is eth1 interface used on both clusters ? If so, I suspect it is build-in RJ45 connection with 1GB speed. Did you check speed/duplex ?

Are RJ45 connections used on all devices from client to server ? Is there 1G RJ45 SFP used on switch ports leading to FWs ?

What is distance between FWs and switches ?

[AkosBakos]

Hi @JenniferYado 

And the third tip is the MTU. Tipically the MTU is 1500 on the interfaces. Do you have the same setting?

A

[JenniferYado]

Yes, I have the same setting 

[the_rock]

Also, see if below may help.

Andy

https://community.checkpoint.com/t5/Security-Gateways/81-20-Performance-CPU-issue/td-p/214709

[the_rock]

Maybe worth mentioning, if VPN tunnels are involved, below also would be relevant.

Andy

https://support.checkpoint.com/results/sk/sk73980

[genisis__]

Another thing to check, and it may likely have been done:

fwaccel stats -s

fw ctl affinity -l -a -v -r

netstat -i

[sjni01]

Hello @JenniferYado, I saw your comment regarding the discarding of L1 troubles. Please provide the output from each firewall and ensure you specifically highlight the interfaces where there is an L2 link using the command #sar -n EDEV.