topic Re: iperf test speeds are different on internal and external for QoS testing in Firewall and Security Management

iperf test speeds are different on internal and external for QoS testing

kamilazat — Thu, 29 Aug 2024 15:10:37 GMT

Hello all!

I've been playing with QoS to understand what goes on and how goes on. Then I noticed a weird behavior.

Here's my simple setup (all R81.20):

PC1 (192.168.1.0/24) -- GW1 ---(10.0.0.0/24)--- GW2 -- PC2 (192.168.4.0/24)

Everything is in an isolated VMware environment, so I don't care about security or systems breaking.

When I do iperf3 from PC1 to GW1 I'm getting speeds up to 1gbps. But when I do the same thing between GW1 and GW2 I get a maximum of 315mbps.

At first I thought I wasn't setting QoS properly. But when set a rule with 100mbps limit, everything works with a maximum of 100mbps. I tried parallel connections with iperf3 -c x.x.x.x -P 4, and with different -P values, but the result is always the same.

I looked at fw ctl chain to maybe see something and noticed that inbound and outbound names for QoS are different.

[Expert@GW1:0]# fw ctl chain
in chain (22):
0: -7fffffff (0000000000000000) (00000000) SecureXL stateless check (sxl_state_check)
1: -7ffffffe (0000000000000000) (00000000) SecureXL VPN before decryption (vpn_in_before_decrypt)
2: -7ffffffd (0000000000000000) (00000000) SecureXL VPN after decryption (vpn_in_after_decrypt)
3: 6 (0000000000000000) (00000000) SecureXL lookup (sxl_lookup)
4: 7 (0000000000000000) (00000000) SecureXL QOS inbound (sxl_qos_inbound)
5: 8 (0000000000000000) (00000000) SecureXL inbound (sxl_inbound)
6: 9 (0000000000000000) (00000000) SecureXL medium path streaming (sxl_medium_path_streaming)
7: 10 (0000000000000000) (00000000) SecureXL inline path streaming (sxl_inline_path_streaming)
8: 11 (0000000000000000) (00000000) SecureXL Routing (sxl_routing)
9: -7f800000 (00007fd748ef22e2) (ffffffff) IP Options Strip (in) (ipopt_strip)
10: - 1fffff8 (00007fd748ef2f40) (00000001) Stateless verifications (in) (asm)
11: - 1fffff7 (00007fd748ef23c0) (00000001) fw multik misc proto forwarding
12: 0 (00007fd748f0fb40) (00000001) fw VM inbound (fw)
13: 2 (00007fd748ef2a80) (00000001) fw SCV inbound (scv)
14: 4 (00007fd746bbff30) (00000003) QoS inbound offload chain module
15: 5 (00007fd748f26dc0) (00000003) fw offload inbound (offload_in)
16: 20 (00007fd748f26080) (00000001) fw post VM inbound (post_vm)
17: 100000 (00007fd748f05196) (00000001) fw accounting inbound (acct)
18: 22000000 (00007fd746bc1ab0) (00000003) QoS slowpath inbound chain mod (fg_sched)
19: 7f730000 (00007fd748ef324e) (00000001) passive streaming (in) (pass_str)
20: 7f750000 (00007fd748f43940) (00000001) TCP streaming (in) (cpas)
21: 7f800000 (00007fd748ef25c8) (ffffffff) IP Options Restore (in) (ipopt_res)
out chain (16):
0: -7f800000 (00007fd748ef22e2) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: - 1fffff0 (00007fd748ef2dc0) (00000001) TCP streaming (out) (cpas)
2: - 1ffff50 (00007fd748ef324e) (00000001) passive streaming (out) (pass_str)
3: - 1f00000 (00007fd748ef2f40) (00000001) Stateless verifications (out) (asm)
4: 0 (00007fd748f0fb40) (00000001) fw VM outbound (fw)
5: 10 (00007fd748f26080) (00000001) fw post VM outbound (post_vm)
6: 15000000 (00007fd746bc0590) (00000003) QoS outbound offload chain modul (fg_pol)
7: 21000000 (00007fd746bc1ab0) (00000003) QoS slowpath outbound chain mod (fg_sched)
8: 7f000000 (00007fd748f05196) (00000001) fw accounting outbound (acct)
9: 7f700000 (00007fd748ef95d8) (00000001) TCP streaming post VM (cpas)
10: 7f800000 (00007fd748ef25c8) (ffffffff) IP Options Restore (out) (ipopt_res)
11: 7f900000 (0000000000000000) (00000000) SecureXL outbound (sxl_outbound)
12: 7fa00000 (0000000000000000) (00000000) SecureXL QOS outbound (sxl_qos_outbound)
13: 7fb00000 (0000000000000000) (00000000) SecureXL VPN before encryption (vpn_in_before_encrypt)
14: 7fc00000 (0000000000000000) (00000000) SecureXL VPN after encryption (vpn_in_after_encrypt)
15: 7fd00000 (0000000000000000) (00000000) SecureXL Deliver (sxl_deliver)

Maybe it is because the internal networks are set as "Internal" and the network between the gateways are set as "External" and the gateways are doing something that slows down the connection. I'm pretty sure that I don't know something about this and everything is working as expected.

So I would deeply appreciate if anyone can illuminate me on this.

Cheers!

Re: iperf test speeds are different on internal and external for QoS testing

AkosBakos — Thu, 29 Aug 2024 15:43:19 GMT

Hi @kamilazat

What blades are enabled on GW-s? Enabled blades are degradating the throughput. This is a known behaviour.
The CPU-s are highly utilized?
- What are the CPU utilization when you reach only the GW1, and what if you reach the GW2 through the GW1?
and the througput values.
- but you said that, there is no bottleneck in your demo LAB

I think, this is where the dog is buried.

cpview command is you friend, monitor the values during the iperf test.

Akos

Re: iperf test speeds are different on internal and external for QoS testing

the_rock — Thu, 29 Aug 2024 16:10:30 GMT

Akos made all super valid points, I would certainly check those as well.

Andy

Re: iperf test speeds are different on internal and external for QoS testing

kamilazat — Thu, 29 Aug 2024 18:31:11 GMT

Thank you!

1. Only firewall, QoS and Monitoring is enabled on both gateways. I enabled monitoring later to potentially find something out.

2. CPUs are highly utilized in both GW-GW and PC-GW scenarios. Here they are:

GW-GW (GW1 is server, cpview from it)

PC-GW (GW1 is again server)

The values are pretty close. All machines are vanilla installations, except GW1 has Take 76 and GW2 is without any JHF. But I'm not sure if that's the issue. I'm now updating that as well anyway.

It's a VMware environment. I still have RAM and CPU left on the host. Both GWs have 4GB RAM with 8 cores, and PCs 2GB RAM and 4 cores.

Please ask for more information 🙂

Re: iperf test speeds are different on internal and external for QoS testing

the_rock — Thu, 29 Aug 2024 18:34:41 GMT

I know in the "old days" of CP, lots of people would fix this by simply disabling corexl from cpconfig, rebooting, re-enable corexl, reboot again.

Not sure that necessarily might be needed in newer versions, but just wondering, can you maybe run top and ps -auxw commands, so we can see what exactly could be causing cpu problem?

Andy

Re: iperf test speeds are different on internal and external for QoS testing

AkosBakos — Thu, 29 Aug 2024 18:40:30 GMT

Hi @kamilazat

The 4 GB memory is not few a little bit? What does free -m say?

Akos

Re: iperf test speeds are different on internal and external for QoS testing

the_rock — Thu, 29 Aug 2024 18:42:55 GMT

I agree with @AkosBakos , 4 GB ram is NOT nearly enough, even if you had 8, I would say that is barely enough...

Yes, run below and send the output.

Andy

[Expert@CP-GW:0]# free -h
total used free shared buff/cache available
Mem: 22G 5.9G 11G 31M 5.6G 15G
Swap: 8.0G 0B 8.0G
[Expert@CP-GW:0]#

Re: iperf test speeds are different on internal and external for QoS testing

kamilazat — Thu, 29 Aug 2024 18:47:43 GMT

Sure.

GW1
[Expert@GW1:0]# free -h
total used free shared buff/cache available
Mem: 3.6G 2.0G 283M 30M 1.3G 721M
Swap: 7.7G 4.2M 7.7G

GW2

[Expert@GW3:0]# free -h
total used free shared buff/cache available
Mem: 3.6G 2.0G 230M 20M 1.4G 782M
Swap: 8.0G 1.2M 8.0G

And still I don't understand why I can get 1gbps when I initiate the test from PC, while between GWs it's 300mbps. Just updated the other GW as well btw, still the same.

Re: iperf test speeds are different on internal and external for QoS testing

the_rock — Thu, 29 Aug 2024 18:53:44 GMT

Less than 1 GB free memory? I hate to say this, but thats not nearly enough my friend : - (

Re: iperf test speeds are different on internal and external for QoS testing

AkosBakos — Thu, 29 Aug 2024 19:09:54 GMT

Hi @kamilazat

Change the core and the memory

4 Cores, 8 GB.

Akos

Re: iperf test speeds are different on internal and external for QoS testing

the_rock — Thu, 29 Aug 2024 19:11:50 GMT

Personally, I would bump it to 6 cores/ at least 12 GB ram (16 if possible)

🙂

Andy

Re: iperf test speeds are different on internal and external for QoS testing

kamilazat — Thu, 29 Aug 2024 19:26:42 GMT

I completely agree with you. Increased both machines to 10GB and redid the test.

Here's the free -h:

[Expert@GW1:0]# free -h
total used free shared buff/cache available
Mem: 9.5G 2.2G 4.4G 20M 2.9G 6.3G
Swap: 7.7G 0B 7.7G

Both machines show the same numbers.

I did watch -n 1 "ps -auxww" during the test and it didn't change.

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
admin 1 0.0 0.0 2632 704 ? Ss 21:58 0:01 init [3]
admin 2 0.0 0.0 0 0 ? S 21:58 0:00 [kthreadd]
admin 3 0.0 0.0 0 0 ? S 21:58 0:00 [kworker/0:0]
admin 4 0.0 0.0 0 0 ? S< 21:58 0:00 [kworker/0:0H]
admin 6 1.9 0.0 0 0 ? S 21:58 0:28 [ksoftirqd/0]
admin 7 0.0 0.0 0 0 ? S 21:58 0:00 [migration/0]
admin 8 0.0 0.0 0 0 ? S 21:58 0:00 [rcu_bh]
admin 9 0.2 0.0 0 0 ? S 21:58 0:03 [rcu_sched]
admin 10 0.0 0.0 0 0 ? S 21:58 0:00 [rcuob/0]
admin 11 0.0 0.0 0 0 ? S 21:58 0:00 [rcuos/0]
admin 12 0.0 0.0 0 0 ? S< 21:58 0:00 [lru-add-drain
]
admin 13 0.0 0.0 0 0 ? S 21:58 0:00 [watchdog/0]
admin 14 0.0 0.0 0 0 ? S 21:58 0:00 [watchdog/1]
admin 15 0.0 0.0 0 0 ? S 21:58 0:00 [migration/1]
admin 16 0.0 0.0 0 0 ? S 21:58 0:00 [ksoftirqd/1]
admin 18 0.0 0.0 0 0 ? S< 21:58 0:00 [kworker/1:0H]
admin 19 0.0 0.0 0 0 ? S 21:58 0:00 [rcuob/1]
admin 20 0.0 0.0 0 0 ? S 21:58 0:00 [rcuos/1]
admin 21 0.0 0.0 0 0 ? S 21:58 0:00 [watchdog/2]
admin 22 0.0 0.0 0 0 ? S 21:58 0:00 [migration/2]

And top showed:

It looks like all the traffic goes to f2f, which is understandable. iperf uses TCP traffic unless UDP is specified with -u. But with -u the bandwidth goes even lower (played with it to increase the rate but to no avail).

On the other hand, when PC is the iperf client the bandwidth is still close to 1gbps. I don't understand what changes.

Re: iperf test speeds are different on internal and external for QoS testing

AkosBakos — Thu, 29 Aug 2024 19:34:27 GMT

What are the outputs of this commads?

fwaccel stats -s

fwaccel stat

fw ctl affinity -l -r

Akos

Re: iperf test speeds are different on internal and external for QoS testing

the_rock — Thu, 29 Aug 2024 19:35:46 GMT

You are saying even with more ram, though now it shows much more free memory, result is the same?

Andy

Re: iperf test speeds are different on internal and external for QoS testing

kamilazat — Thu, 29 Aug 2024 19:40:51 GMT

[Expert@GW1:0]# fwaccel stats -s
Accelerated conns/Total conns : 9/9 (100%)
LightSpeed conns/Total conns : 0/9 (0%)
Accelerated pkts/Total pkts : 3670/10364 (35%)
LightSpeed pkts/Total pkts : 0/10364 (0%)
F2Fed pkts/Total pkts : 6694/10364 (64%)
F2V pkts/Total pkts : 142/10364 (1%)
CPASXL pkts/Total pkts : 0/10364 (0%)
PSLXL pkts/Total pkts : 0/10364 (0%)
CPAS pipeline pkts/Total pkts : 0/10364 (0%)
PSL pipeline pkts/Total pkts : 0/10364 (0%)
QOS inbound pkts/Total pkts : 5779/10364 (55%)
QOS outbound pkts/Total pkts : 5046/10364 (48%)
Corrected pkts/Total pkts : 0/10364 (0%)

Accept Templates : enabled
Drop Templates : disabled
NAT Templates : enabled
LightSpeed Accel : disabled

[Expert@GW1:0]# fw ctl affinity -l -r
CPU 0:
CPU 1:
CPU 2:
CPU 3: fw_4 (active)
cprid lpd mpdaemon fwd core_uploader fgd50 in.asessiond rtmd cprid cpd msgd
CPU 4: fw_3 (active)
cprid lpd mpdaemon fwd core_uploader fgd50 in.asessiond rtmd cprid cpd msgd
CPU 5: fw_2 (active)
cprid lpd mpdaemon fwd core_uploader fgd50 in.asessiond rtmd cprid cpd msgd
CPU 6: fw_1 (active)
cprid lpd mpdaemon fwd core_uploader fgd50 in.asessiond rtmd cprid cpd msgd
CPU 7: fw_0 (active)
cprid lpd mpdaemon fwd core_uploader fgd50 in.asessiond rtmd cprid cpd msgd
All:
Interface eth0: has multi queue enabled
Interface eth1: has multi queue enabled

Just increased the SND cores to test, but still the same.

Re: iperf test speeds are different on internal and external for QoS testing

kamilazat — Thu, 29 Aug 2024 19:41:13 GMT

True..

Re: iperf test speeds are different on internal and external for QoS testing

the_rock — Thu, 29 Aug 2024 19:55:16 GMT

What does cpview show now? Anything different than before you added the ram?

Andy

Re: iperf test speeds are different on internal and external for QoS testing

kamilazat — Thu, 29 Aug 2024 20:00:05 GMT

RAM doesn't change while doing the test. And CPU goes crazy as before. At this point I started thinking that it's a VMWare issue. Maybe it calculates the traffic in a different way when it comes to different machines, despite PC having only 2GB RAM and 4 cores. I wouldn't be surprised. I'll try rebooting the host.

Re: iperf test speeds are different on internal and external for QoS testing

the_rock — Thu, 29 Aug 2024 20:02:05 GMT

I agree, thats probably a good idea.

Re: iperf test speeds are different on internal and external for QoS testing

kamilazat — Thu, 29 Aug 2024 20:21:05 GMT

I couldn't find anything related to VMware either. Rebooted the host, restarted VMWare NAT service. Tomorrow I will use the lab in the company esxi and try cranking up the specs on my machines there. I will update when I get anything new.

Thank you for your prompt reaction. I appreciate it!