topic Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes? in Hyperscale Firewall (Maestro)

Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

kamilazat — Fri, 15 Nov 2024 08:08:58 GMT

Hello everyone,

One of our customers constantly having issues with failovers due to routed PNOTE. We noticed that they have around 12k OSPF routes. After the failover we see that the routed daemon on the newly active node already uses 3GB of memory. Here's the output from the currently active node after failover:

[Global] ch02-01> show routed memory

1_01:

Total Memory Usage: 32 MB

Core: 3 MB

BGP: 360 B

MFC: 236 B

OSPF: 28 MB

Policy: 2 KB

2_01:

Total Memory Usage: 3 GB

Core: 3 MB

BGP: 360 B

MFC: 236 B

OSPF: 3 GB

Policy: 2 KB

[Global] MAB-CL1-ch02-01> show routed resources

1_01:

Total Uptime : 3 hrs 9 mins 44 secs

Total User Time : 3 mins 2 secs

Total System Time : 13 secs

Page Faults : 85

Page Reclaims : 16070

Total Swaps: : 0

Voluntary Context Switches : 624724

Involuntary Context Switches : 70850

2_01:

Total Uptime : 21 days 1 hr 49 mins 52 secs

Total User Time : 12 hrs 3 mins 26 secs

Total System Time : 27 mins 46 secs

Page Faults : 807

Page Reclaims : 1023538

Total Swaps: : 0

Voluntary Context Switches : 63305978

Involuntary Context Switches : 12788009

At first, we thought that there was some memory leak going on, but there are no coredumps or anything that points to a memory leak. TAC agreed with us on this as well. What I'm wondering is if it's normal when there are 12k OSPF routes resulting in 3GB memory use. Or should we look at some underlying issues that ends up with this symptom?

If this is normal, then how much would route aggregation help us with the situation (before we go for a memory increasing path)?

As always, thank you very much.

Cheers!

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

Lesley — Fri, 15 Nov 2024 09:34:51 GMT

What version you are running and jumbo take? (cpinfo -y all)

Maybe worth checking this:

Best Practice - Limit OSPF areas to about 50 routers based on the limitations of OSPF (traffic overhead, table size, convergence, and so on).

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

kamilazat — Fri, 15 Nov 2024 12:18:55 GMT

@Lesley Thank you for the reply. The version is R81.10 JHF Take 156. The behavior is seen since Take 95. I'm not sure there will be any control on OSPF routers though.

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

Chris_Atkinson — Fri, 15 Nov 2024 14:01:11 GMT

How stable is the overall OSPF topology?

How many neighbors / what does it look like?

Is graceful restart used?

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

kamilazat — Fri, 15 Nov 2024 14:47:17 GMT

- Graceful restart is on.

- The output of 'show ospf neighbor' looks like this:

show ospf neighbor

Instance: default
Neighbor state flag: G - graceful restart

Neighbor ID Pri State Dead Address Interface Errors
x.a.a.a 0 FULL 38 x.x.x.162 x.x.x.168 35
x.b.b.b 0 FULL 40 x.x.x.163 x.x.x.168 33
x.c.c.c 0 FULL 35 x.x.x.164 x.x.x.168 21
x.d.d.d 0 FULL 35 x.x.x.165 x.x.x.168 28
x.e.e.e 0 FULL 36 x.x.x.166 x.x.x.168 27
x.f.f.f 0 FULL 37 x.x.x.167 x.x.x.168 37

I don't know how stable the overall OSPF topology is, but for what it's worth, here's the OSPF summary:

FW1> show route summary

RouteSource Networks

connected 5
kernel 2
static 9
aggregate 0
bgp 0
igrp 0
ospf(default) 12901
rip 0
nat-pool 0
Total 12917

I think it could probably be considered 'relatively' stable since we're also working on aggregating the OSPF routes. But I don't have any information about potential reasons that may cause it to become unstable, like LSA flooding, timer differences etc.

By the way, routed daemon somehow released 3GB memory and now it's at 39MB. For the last two months, there have been multiple failovers due to routed crashing 'without' a coredump. We're currently trying to find the underlying reason with TAC, but they haven't been extremely helpful so far.

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

the_rock — Fri, 15 Nov 2024 15:10:55 GMT

I alsways found that ospf/bgp in general is better in R81.20 than before. Not saying upgrading would fix the issue, but Im sure it would get better.

Andy

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

PhoneBoy — Fri, 15 Nov 2024 15:32:01 GMT

Send me the TAC SR in a PM.

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

JozkoMrkvicka — Fri, 15 Nov 2024 20:06:41 GMT

do you see routed restarting (ps aux | grep routed) ?

What remote OSPF peers say ? Do they see some strange things happening, like restarting or something not correct ?

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

kamilazat — Mon, 18 Nov 2024 07:15:49 GMT

Well, a new version usually means better code, of course. But we couldn't find a resolved issue that addresses our situation and blindly updating doesn't seem like a good idea currently.

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

kamilazat — Mon, 18 Nov 2024 07:20:15 GMT

routed keeps using more and more memory until it crashes and restarts, which results in failover. I will ask for information from remote peers, but not sure about what we'll be looking for. What should be looking for on the remote peers?

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

JozkoMrkvicka — Mon, 18 Nov 2024 21:20:58 GMT

If they see some strange logs related to OSPF or maybe compare configs if everything is correctly.

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

kamilazat — Tue, 19 Nov 2024 13:01:04 GMT

I will definitely have them look into it.

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

PhoneBoy — Tue, 19 Nov 2024 16:05:00 GMT

According to R&D, 13k routes is a lot of routes for OSPF.
From the output, it looks like 6 peers and one interface is used, correct?

OSPF exchanges router and link information, which is stored in a database.
The router then has to do a route calculation that takes into account all entries in the database to build routes as a result. Considering a deployment like this has to store a massive database and also perform very large route calculations, it is not surprising a lot of memory is being used.

Route aggregation on our gateway won't help much.
This does not decrease the number of OSPF neighbors, interfaces, and the size of the database exchanged within the OSPF network.
If the aggregation is done on other routers in the environment, it might help, but the impact may not be significant.

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

Lesley — Tue, 19 Nov 2024 19:16:24 GMT

Are we sure there are no core dumps? If you run HCP health check any interesting in there?

PRJ-56903,
PMTR-108071

Routing

The ROUTED daemon may exit with a core dump during a BGP or OSPF restart.

You state now that it has been reduced from 3gb to 30mb does it slowly increase again and then it will crash around 3gb?

See anything in monitoring tool (if you use) if you see memory increase slowly maybe there is memory leak.

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

kamilazat — Wed, 20 Nov 2024 09:18:59 GMT

Yes, triple checked, hoping that a simple update would resolve the issue...

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

kamilazat — Wed, 20 Nov 2024 09:22:19 GMT

@PhoneBoy Thank you for the effort!

We also heard from TAC, saying that also consulted with R&D, and apparently they think that a memory leak is likely and a Valgrind debug is on the way.

Judging by cpview data, routed consumed up to 16GB memory (on a 64GB machine) and then exits. I don't know what stops it at that point, or how to remedy this.

I will update this post as soon as we have new information.

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

the_rock — Wed, 20 Nov 2024 11:43:42 GMT

Maybe 16 GB is some sort of "limit" for it? Let us know what they say about it.

Andy

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

JozkoMrkvicka — Wed, 20 Nov 2024 17:08:10 GMT

there are so many limitations that it is impossible to discovered them and document them properly. Some are even not known and were never tested.

Re: Is it normal for routed daemon to use 3GB of RAM for 12k OSPF routes?

PhoneBoy — Wed, 20 Nov 2024 18:08:54 GMT

I'm inclined to agree there's a memory leak somewhere based on what you've reported.