Cluster failure - routed cannot allocate memory

[dam_jim]

Hello. We have a 2 members cluster and each 3 or 4 days they swap. We have 10 VRF's set with OSPF and there is this particular one which has the most routes who is giving troubles.

This is the message i've got:

Click to Expand

Mar 27 22:47:16 2020 CHCKPFW-1 routed[17209]: igmp_recv: packet from non-local neighbor 0.0.0.0
Mar 27 22:48:24 2020 CHCKPFW-1 routed[1193]: task_block_sbrk: sbrk(4096): Cannot allocate memory
Mar 27 22:48:24 2020 CHCKPFW-1 routed[1193]: Abort routed[1193] version routed-06.10.2014-23:11:13: Cannot allocate memory
Mar 27 22:48:24 2020 CHCKPFW-1 kernel: do_coredump: corename = |/etc/coredump/compress.sh /var/log/dump/usermode/routed.1193.core
Mar 27 22:48:24 2020 CHCKPFW-1 kernel: do_coredump: argv_arr[0] = /etc/coredump/compress.sh
Mar 27 22:48:24 2020 CHCKPFW-1 kernel: do_coredump: argv_arr[1] = /var/log/dump/usermode/routed.1193.core

Mar 27 22:49:21 2020 CHCKPFW-1 routed[17209]: igmp_recv: packet from non-local neighbor 0.0.0.0

And before this, all the times messages like this until it crashes:

Click to Expand

Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.3-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.5-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.6-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.7-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.8-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.9-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.10-10.106.18.122> seq 0x80000003 age 1 violates m
inimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.11-10.106.18.122> seq 0x80000003 age 1 violates m
inimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600

show ospf command list lots of errors:

Click to Expand

CHCKPFW-1:TACP-15> show ospf errors
show instance 4 ospf errors

Hello Protocol Errors

    Bad Size                  0            Network Mask Mismatch     0        
    Dead Interval Mismatch    0            Hello Duplicate Router ID 0        
    External Option Mismatch  0            NSSA Option Mismatch      0        
    Runt                      0            Hello Timer Mismatch      0        

Link State Update Errors

    Runt                      0            LSU Duplicate Router ID   0        
    LSU TooLow                5            BadCSum                   0        
    BadLSType                 0            ASEinStub                 0        
    Type7inNonNSSA            0            LSU TooNew                2200505  
    BadLSReq                  0            SeqNumWrap                0        
    Invalid SeqNum            0            SummaryinTotalStub        0        
    BadRouterLSASize          0            BadNetworkLSASize         0        
    BadSummaryLSASize         0            BadASELSASize             0        
    BadType7LSASize           0        

Link State ACK Errors

    LSAck Duplicate Router ID 0            LSAck TooLow              0        
    BadSize                   0            QuestionAck               102338311  
    BadLSType                 0        

Link State Request Errors

    LSR Duplicate Router ID   0            BadSize                   0        
    BadState                  0            Empty Request             0        

Database Description Errors

    ASEinStub                 0            Type7inNonNSSA            0        
    MTU                       0            BadLSType                 0        
    NotDuplicate              0            BadSize                   0        
    OptionsMismatch           0            DuplicateLSA              0        
    DD Duplicate Router ID    0            InitSet                   132      
    Runt                      0            MasterMismatch            0        
    SlaveSeq                  0            MasterSeq                 0        
    DD TooLow                 0        

Protocol Errors

    Bad Area ID               0            Area ID Mismatch          0        
    AuthCryptoSeq             0            AuthKey                   0        
    AuthKeyId                 0            AuthKeyTime               0        
    AuthKeyType               0            BadDestination            0        
    Checksum                  0            NoNeighbor                0        
    NoOspf                    0            Size                      0        
    Version                   0            NonLocal                  0        
    VirtualLink               0            NoVirtualNeighbor         0        
    IfDown                    0            PacketType                0        
    Passive Interface         0            TX                        0        
    ZeroRID                   0       

Summary of routes on the database, both on the FW and the router matches. Any clue how to fix this? Thanks. 

[_Val_]

Open a TAC request please

[kamilazat]

@dam_jim I know this was years ago, but do you remember what the issue was and how it got resolved?

[the_rock]

Considering this was in 2020, probably R80.xx

[AkosBakos]

TAC first, 

And I'm coriuos, what is the version and the JHF take?

A