topic Re: Accelerated SYNC Connections in Hyperscale Firewall (Maestro)

Accelerated SYNC Connections

kamilazat — Mon, 03 Jun 2024 13:13:39 GMT

Hi all!

On Maestro R81.10 JHF Take 139, we are seeing a high rate of F2F traffic on the SGMs.

When we looked at fw_tab_t_connections_u (slow-path connections table) we saw that the SYNC connections comprise the majority of the total connections there. After that comes the interface that RA clients connect to.

SYNC interface:

0 x.x.x.15 50070 x.x.x.1 2010 6 TCP Estab. 3600/3600 N/A Connection non-accel(EXFLAG set) 4.88M 1.61GB 16h50m30s 0s

The VLAN that RA clients connect to (shown as a.b.c.d, the e.f.g.h is a placeholder for arbitrary RA IPs):

Line 46: 0 e.f.g.h 5921 a.b.c.d 443 6 TCP Estab. 3598/3600 N/A Local incoming conn 1.86M 488.61MB 7h26m51s 2s

Line 62: 0 e.f.g.h 1653 a.b.c.d 443 6 TCP Estab. 3597/3600 N/A Local incoming conn 2.28M 464.45MB 6h30m36s 2s

Line 79: 1 a.b.c.d 443 e.f.g.h 5921 6 Link

Line 81: 0 e.f.g.h 53275 a.b.c.d 443 6 TCP Estab. 3595/3600 N/A Local incoming conn 671.68K 193.26MB 7h1m5s 5s

Line 216: 1 a.b.c.d 443 e.f.g.h 53275 6 Link

Line 255: 1 a.b.c.d 443 e.f.g.h 1653 6 Link

Line 287: 1 a.b.c.d 443 e.f.g.h 48395 6 Link

Line 291: 1 a.b.c.d 443 e.f.g.h 1858 6 Link

Line 315: 1 a.b.c.d 443 e.f.g.h 8253 6 Link

Line 357: 0 e.f.g.h 3570 a.b.c.d 443 6 TCP Estab. 3593/3600 N/A Local incoming conn 1.79K 310.62KB 14m27s 7s

Line 384: 1 a.b.c.d 443 e.f.g.h 3570 6 Link

Line 394: 0 e.f.g.h 48395 a.b.c.d 443 6 TCP Estab. 3598/3600 N/A Local incoming conn 1.71M 390.99MB 8h46m4s 0s

Line 407: 0 e.f.g.h 1858 a.b.c.d 443 6 TCP Estab. 3598/3600 N/A Local incoming conn 1.27M 436.31MB 9h24m25s 0s

Line 486: 0 e.f.g.h 8253 a.b.c.d 443 6 TCP Estab. 3600/3600 N/A Local incoming conn 591.67K 146.13MB 8h10m59s 0s

Line 518: 1 a.b.c.d 443 e.f.g.h 9597 6 Link

Line 558: 1 a.b.c.d 443 e.f.g.h 50137 6 Link

Line 596: 0 e.f.g.h 50137 a.b.c.d 443 6 TCP Estab. 3600/3600 N/A Local incoming conn 617.45K 250.11MB 10h50m53s 0s

Line 646: 0 e.f.g.h 9597 a.b.c.d 443 6 TCP Estab. 3597/3600 N/A Local incoming conn 1.43M 348.83MB 8h13m47s 1s

Line 672: 0 e.f.g.h 18107 a.b.c.d 18234 17 UDP 33/40 N/A Local incoming conn 1 40B 7s 7s

Line 722: 0 e.f.g.h 18106 a.b.c.d 18234 17 UDP 12/40 N/A Local incoming conn 1 40B 28s 28s

- The only enabled blades are fw, vpn, cvpn and identityServer.

- Accept and NAT templates are enabled

- We don’t see the IP a.b.c.d in the accelerated connections table

- F2F stats:

----------------------

F2F packets:

--------------

Violation Packets Violation Packets

-------------------- --------------- -------------------- ---------------

Pkt has IP options 0 ICMP miss conn 262948

TCP-SYN miss conn 65008 TCP-other miss conn 7605080

UDP miss conn 1739604 Other miss conn 883569

VPN returned F2F 0 Uni-directional viol 0

Possible spoof viol 159585 TCP state viol 0

SCTP state affecting 0 Out if not def/accl 0

Bridge src=dst 0 Routing decision err 0

Sanity checks failed 0 Fwd to non-pivot 0

Broadcast/multicast 0 Cluster message 4667249

Cluster forward 0 Chain forwarding 0

F2V conn match pkts 0 General reason 0

Route changes 0 VPN multicast traffic 0

GTP non-accelerated 0 Unresolved nexthop 0

----------------------

fwaccel stats -s

----------------------

Accelerated conns/Total conns : 45/45 (100%)

LightSpeed conns/Total conns : 0/45 (0%)

Accelerated pkts/Total pkts : 8058132/29058892 (27%)

LightSpeed pkts/Total pkts : 0/29058892 (0%)

F2Fed pkts/Total pkts : 21000760/29058892 (72%)

F2V pkts/Total pkts : 55079/29058892 (0%)

CPASXL pkts/Total pkts : 0/29058892 (0%)

PSLXL pkts/Total pkts : 0/29058892 (0%)

CPAS pipeline pkts/Total pkts : 0/29058892 (0%)

PSL pipeline pkts/Total pkts : 0/29058892 (0%)

CPAS inline pkts/Total pkts : 0/29058892 (0%)

PSL inline pkts/Total pkts : 0/29058892 (0%)

QOS inbound pkts/Total pkts : 0/29058892 (0%)

QOS outbound pkts/Total pkts : 0/29058892 (0%)

Corrected pkts/Total pkts : 0/29058892 (0%)

After doing a little research I noticed that RA clients connect to a.b.c.d on port 443, instead of UDP 4500, although the vpnd process has this port open. Visitor Mode is enabled and UDP 4500 is NOT blocked.

I looked at vpnd.elg and noticed there are thousands of the following errors:

CPRTI: got error 105 buffer is full

: No buffer space available

-- and --

Unable to open '/vs0/dev/fw6v0': Connection refused

Not being sure, I opened a TAC case, here’s what they said:

- Upgrade the hardware specs (4 CPU - 8GB RAM)

- SYNC connections do not get accelerated (really? why?)

- “Unable to open '/vs0/dev/fw6v0': Connection refused” is a pdp problem and we should open a new ticket for VPN and pdp teams.

Thanks in advance for all the opinions and advice!

Re: Accelerated SYNC Connections

the_rock — Mon, 03 Jun 2024 13:26:03 GMT

I know they gave me the same answer before for sync connections not being accelerated and I think that is actually true, but maybe someone else can confirm 100%.

Andy

Re: Accelerated SYNC Connections

kamilazat — Mon, 03 Jun 2024 13:28:46 GMT

Do you think there is a documentation for why that might be the case? Or did they say why?

Re: Accelerated SYNC Connections

_Val_ — Mon, 03 Jun 2024 13:32:11 GMT

Sync traffic is not accelerated by definition, as it goes to or from FW itself. SecureXL can only accelerate some of the traffic that crosses the GW, and never traffic where GW is the source or destination.

More info in sk32578 and SecureXL ATRG

Re: Accelerated SYNC Connections

the_rock — Mon, 03 Jun 2024 13:37:22 GMT

@_Val_ "beat" me for that answer, but thats exactly what they mentioned, the sk he gave and thats its not accelerated by default.

Best,

Andy

Re: Accelerated SYNC Connections

kamilazat — Mon, 03 Jun 2024 13:48:23 GMT

Thank you for the answer and the sk.

What about the vpn connections that are not accelerated? How can we find out why port 443 is used, instead of 4500? Or turning off Visitor Mode cause issues with the clients that are connected on 443?

Re: Accelerated SYNC Connections

_Val_ — Mon, 03 Jun 2024 13:55:22 GMT

From the very same SK I just gave you:

(1) Acceleration of packets

When SecureXL is enabled, all packets should be accelerated, except packets that match the following conditions:
........

IPsec VPN Visitor Mode packets.

Visitor means TLS is used instead of IPsec, and it needs to be terminated on the FW itself, which basically falls into the same "no acceleration of packets going to and from FW" line.

Disable Visitor Mode and see if it makes any difference.

Re: Accelerated SYNC Connections

kamilazat — Tue, 04 Jun 2024 15:03:20 GMT

Thank you for clarification @_Val_ !

We disabled Visitor Mode and now it looks like this:

# vpn tu tlist
(0) Site-to-Site tunnels are up:
IPSEC 0
NAT-T 0

(9) Number of Active Clients:
NAT-T 9
Visitor Mode 0
SSL 0
L2TP 0
StrongSwan 0

However, we still have the picture below and don't quite understand why.

We still see massive SYNC traffic in slow-path connections table:

0 s.y.n.c 33016 s.y.n.c 2010 6 TCP Estab. 3600/3600 N/A Connection non-accel(EXFLAG set) 36.16M 11.60GB 121h11m39s 0s

At the same time we see the following lines, which is understandable since this is a big environment. But I want to understand if this traffic somehow adds up to SYNC traffic, because of topology sharing between the nodes.

Line 92: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 599.79K 493.88MB 121h17m16s 1s
Line 163: 0 x.y.z.f 0 224.0.0.6 0 89 55/60 N/A Connection non-accel(EXFLAG set) 413.84K 327.01MB 121h17m16s 5s
Line 166: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 604.22K 493.63MB 121h20m39s 1s
Line 194: 0 x.y.z.f 0 224.0.0.6 0 89 54/60 N/A Connection non-accel(EXFLAG set) 426.94K 333.82MB 121h17m16s 5s
Line 220: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 604.09K 491.69MB 121h20m39s 1s
Line 276: 0 x.y.z.f 0 224.0.0.6 0 89 58/60 N/A Connection non-accel(EXFLAG set) 440.15K 332.51MB 121h17m16s 2s
Line 278: 0 x.y.z.f 0 224.0.0.6 0 89 60/60 N/A Connection non-accel(EXFLAG set) 427.81K 332.82MB 121h17m16s 0s
Line 279: 0 x.y.z.f 0 224.0.0.6 0 89 60/60 N/A Connection non-accel(EXFLAG set) 440.37K 335.76MB 121h17m16s 0s
Line 298: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 601.53K 492.11MB 121h17m16s 1s
Line 481: 0 x.y.z.f 0 224.0.0.6 0 89 57/60 N/A Connection non-accel(EXFLAG set) 441.06K 337.06MB 121h17m16s 3s
Line 504: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 600.22K 492.66MB 121h17m16s 1s
Line 523: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 603.21K 496.28MB 121h17m16s 1s

Is there a way to solve this riddle?

Re: Accelerated SYNC Connections

the_rock — Tue, 04 Jun 2024 15:07:20 GMT

@kamilazat I believe thats normal, as again, those wont be accelerated, so you wont see them as fast path.

Andy