Accelerated SYNC Connections

[kamilazat]

Hi all!

On Maestro R81.10 JHF Take 139, we are seeing a high rate of F2F traffic on the SGMs. 

When we looked at fw_tab_t_connections_u (slow-path connections table) we saw that the SYNC connections comprise the majority of the total connections there. After that comes the interface that RA clients connect to.

SYNC interface:

0    x.x.x.15       50070  x.x.x.1        2010   6   TCP Estab.       3600/3600      N/A     Connection non-accel(EXFLAG set)  4.88M       1.61GB       16h50m30s   0s 

The VLAN that RA clients connect to (shown as a.b.c.d, the e.f.g.h is a placeholder for arbitrary RA IPs):

Line  46: 0    e.f.g.h   5921   a.b.c.d   443    6   TCP Estab.       3598/3600      N/A     Local incoming conn               1.86M       488.61MB     7h26m51s    2s        

Line  62: 0    e.f.g.h   1653   a.b.c.d   443    6   TCP Estab.       3597/3600      N/A     Local incoming conn               2.28M       464.45MB     6h30m36s    2s        

Line  79: 1    a.b.c.d   443    e.f.g.h   5921   6   Link             

Line  81: 0    e.f.g.h  53275   a.b.c.d   443    6   TCP Estab.       3595/3600      N/A     Local incoming conn               671.68K     193.26MB     7h1m5s      5s        

Line 216: 1    a.b.c.d   443    e.f.g.h   53275  6   Link             

Line 255: 1    a.b.c.d   443    e.f.g.h   1653   6   Link             

Line 287: 1    a.b.c.d   443    e.f.g.h   48395  6   Link             

Line 291: 1    a.b.c.d   443    e.f.g.h   1858   6   Link             

Line 315: 1    a.b.c.d   443    e.f.g.h   8253   6   Link             

Line 357: 0    e.f.g.h   3570   a.b.c.d   443    6   TCP Estab.       3593/3600      N/A     Local incoming conn               1.79K       310.62KB     14m27s      7s        

Line 384: 1    a.b.c.d   443    e.f.g.h   3570   6   Link             

Line 394: 0    e.f.g.h  48395   a.b.c.d   443    6   TCP Estab.       3598/3600      N/A     Local incoming conn               1.71M       390.99MB     8h46m4s     0s        

Line 407: 0    e.f.g.h   1858   a.b.c.d   443    6   TCP Estab.       3598/3600      N/A     Local incoming conn               1.27M       436.31MB     9h24m25s    0s        

Line 486: 0    e.f.g.h   8253   a.b.c.d   443    6   TCP Estab.       3600/3600      N/A     Local incoming conn               591.67K     146.13MB     8h10m59s    0s        

Line 518: 1    a.b.c.d   443    e.f.g.h   9597   6   Link             

Line 558: 1    a.b.c.d   443    e.f.g.h   50137  6   Link             

Line 596: 0    e.f.g.h  50137   a.b.c.d   443    6   TCP Estab.       3600/3600      N/A     Local incoming conn               617.45K     250.11MB     10h50m53s   0s        

Line 646: 0    e.f.g.h   9597   a.b.c.d   443    6   TCP Estab.       3597/3600      N/A     Local incoming conn               1.43M       348.83MB     8h13m47s    1s        

Line 672: 0    e.f.g.h  18107   a.b.c.d   18234  17  UDP              33/40          N/A     Local incoming conn               1           40B          7s          7s        

Line 722: 0    e.f.g.h  18106   a.b.c.d   18234  17  UDP              12/40          N/A     Local incoming conn               1           40B          28s         28s       

- The only enabled blades are fw, vpn, cvpn and identityServer. 

- Accept and NAT templates are enabled

- We don’t see the IP a.b.c.d in the accelerated connections table

- F2F stats:

----------------------

F2F packets:

--------------

Violation             Packets            Violation             Packets        

--------------------  ---------------    --------------------  ---------------

Pkt has IP options                  0    ICMP miss conn                 262948

TCP-SYN miss conn               65008    TCP-other miss conn           7605080

UDP miss conn                 1739604    Other miss conn                883569

VPN returned F2F                    0    Uni-directional viol                0

Possible spoof viol            159585    TCP state viol                      0

SCTP state affecting                0    Out if not def/accl                 0

Bridge src=dst                      0    Routing decision err                0

Sanity checks failed                0    Fwd to non-pivot                    0

Broadcast/multicast                 0    Cluster message               4667249

Cluster forward                     0    Chain forwarding                    0

F2V conn match pkts                 0    General reason                      0

Route changes                       0    VPN multicast traffic               0

GTP non-accelerated                 0    Unresolved nexthop                  0

----------------------

fwaccel stats -s

----------------------

Accelerated conns/Total conns    : 45/45 (100%)

LightSpeed conns/Total conns     : 0/45 (0%)

Accelerated pkts/Total pkts      : 8058132/29058892 (27%)

LightSpeed pkts/Total pkts       : 0/29058892 (0%)

F2Fed pkts/Total pkts            : 21000760/29058892 (72%)

F2V pkts/Total pkts              : 55079/29058892 (0%)

CPASXL pkts/Total pkts           : 0/29058892 (0%)

PSLXL pkts/Total pkts            : 0/29058892 (0%)

CPAS pipeline pkts/Total pkts    : 0/29058892 (0%)

PSL pipeline pkts/Total pkts     : 0/29058892 (0%)

CPAS inline pkts/Total pkts      : 0/29058892 (0%)

PSL inline pkts/Total pkts       : 0/29058892 (0%)

QOS inbound pkts/Total pkts      : 0/29058892 (0%)

QOS outbound pkts/Total pkts     : 0/29058892 (0%)

Corrected pkts/Total pkts        : 0/29058892 (0%)

After doing a little research I noticed that RA clients connect to a.b.c.d on port 443, instead of UDP 4500, although the vpnd process has this port open. Visitor Mode is enabled and UDP 4500 is NOT blocked. 

I looked at vpnd.elg and noticed there are thousands of the following errors:

CPRTI: got error 105 buffer is full 

: No buffer space available

-- and -- 

Unable to open '/vs0/dev/fw6v0': Connection refused

Not being sure, I opened a TAC case, here’s what they said:

- Upgrade the hardware specs (4 CPU - 8GB RAM) 

- SYNC connections do not get accelerated (really? why?)

- “Unable to open '/vs0/dev/fw6v0': Connection refused” is a pdp problem and we should open a new ticket for VPN and pdp teams.

Thanks in advance for all the opinions and advice!

[the_rock]

I know they gave me the same answer before for sync connections not being accelerated and I think that is actually true, but maybe someone else can confirm 100%.

Andy

[kamilazat]

Do you think there is a documentation for why that might be the case? Or did they say why?

[the_rock]

@_Val_ "beat" me for that answer, but thats exactly what they mentioned, the sk he gave and thats its not accelerated by default.

Best,

Andy

[_Val_ · Accepted solution]

Sync traffic is not accelerated by definition, as it goes to or from FW itself. SecureXL can only accelerate some of the traffic that crosses the GW, and never traffic where GW is the source or destination.

More info in sk32578 and SecureXL ATRG

[kamilazat]

Thank you for the answer and the sk.

What about the vpn connections that are not accelerated? How can we find out why port 443 is used, instead of 4500? Or turning off Visitor Mode cause issues with the clients that are connected on 443?

[_Val_]

From the very same SK I just gave you: 

(1) Acceleration of packets

When SecureXL is enabled, all packets should be accelerated, except packets that match the following conditions:
........

• IPsec VPN Visitor Mode packets.

Visitor means TLS is used instead of IPsec, and it needs to be terminated on the FW itself, which basically falls into the same "no acceleration of packets going to and from FW" line.

Disable Visitor Mode and see if it makes any difference.

[kamilazat]

Thank you for clarification @_Val_ !

We disabled Visitor Mode and now it looks like this:

# vpn tu tlist
(0) Site-to-Site tunnels are up:
IPSEC 0
NAT-T 0

(9) Number of Active Clients:
NAT-T 9
Visitor Mode 0
SSL 0
L2TP 0
StrongSwan 0

However, we still have the picture below and don't quite understand why. 

We still see massive SYNC traffic in slow-path connections table:

0 s.y.n.c 33016 s.y.n.c 2010 6 TCP Estab. 3600/3600 N/A Connection non-accel(EXFLAG set) 36.16M 11.60GB 121h11m39s 0s

At the same time we see the following lines, which is understandable since this is a big environment. But I want to understand if this traffic somehow adds up to SYNC traffic, because of topology sharing between the nodes.

Line 92: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 599.79K 493.88MB 121h17m16s 1s
Line 163: 0 x.y.z.f 0 224.0.0.6 0 89 55/60 N/A Connection non-accel(EXFLAG set) 413.84K 327.01MB 121h17m16s 5s
Line 166: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 604.22K 493.63MB 121h20m39s 1s
Line 194: 0 x.y.z.f 0 224.0.0.6 0 89 54/60 N/A Connection non-accel(EXFLAG set) 426.94K 333.82MB 121h17m16s 5s
Line 220: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 604.09K 491.69MB 121h20m39s 1s
Line 276: 0 x.y.z.f 0 224.0.0.6 0 89 58/60 N/A Connection non-accel(EXFLAG set) 440.15K 332.51MB 121h17m16s 2s
Line 278: 0 x.y.z.f 0 224.0.0.6 0 89 60/60 N/A Connection non-accel(EXFLAG set) 427.81K 332.82MB 121h17m16s 0s
Line 279: 0 x.y.z.f 0 224.0.0.6 0 89 60/60 N/A Connection non-accel(EXFLAG set) 440.37K 335.76MB 121h17m16s 0s
Line 298: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 601.53K 492.11MB 121h17m16s 1s
Line 481: 0 x.y.z.f 0 224.0.0.6 0 89 57/60 N/A Connection non-accel(EXFLAG set) 441.06K 337.06MB 121h17m16s 3s
Line 504: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 600.22K 492.66MB 121h17m16s 1s
Line 523: 0 x.y.z.f 0 224.0.0.5 0 89 59/60 N/A Connection non-accel(EXFLAG set) 603.21K 496.28MB 121h17m16s 1s

Is there a way to solve this riddle?

[the_rock]

@kamilazat I believe thats normal, as again, those wont be accelerated, so you wont see them as fast path.

Andy