Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Ivory

Cluster failure - routed cannot allocate memory

Hello. We have a 2 members cluster and each 3 or 4 days they swap. We have 10 VRF's set with OSPF and there is this particular one which has the most routes who is giving troubles.

This is the message i've got:

 

 

Spoiler

Mar 27 22:47:16 2020 CHCKPFW-1 routed[17209]: igmp_recv: packet from non-local neighbor 0.0.0.0
Mar 27 22:48:24 2020 CHCKPFW-1 routed[1193]: task_block_sbrk: sbrk(4096): Cannot allocate memory
Mar 27 22:48:24 2020 CHCKPFW-1 routed[1193]: Abort routed[1193] version routed-06.10.2014-23:11:13: Cannot allocate memory
Mar 27 22:48:24 2020 CHCKPFW-1 kernel: do_coredump: corename = |/etc/coredump/compress.sh /var/log/dump/usermode/routed.1193.core
Mar 27 22:48:24 2020 CHCKPFW-1 kernel: do_coredump: argv_arr[0] = /etc/coredump/compress.sh
Mar 27 22:48:24 2020 CHCKPFW-1 kernel: do_coredump: argv_arr[1] = /var/log/dump/usermode/routed.1193.core

Mar 27 22:49:21 2020 CHCKPFW-1 routed[17209]: igmp_recv: packet from non-local neighbor 0.0.0.0
 
 
 
And before this, all the times messages like this until it crashes:
 
 
Spoiler
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.3-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.5-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.6-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.7-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.8-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.9-10.106.18.122> seq 0x80000003 age 1 violates mi
nimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.10-10.106.18.122> seq 0x80000003 age 1 violates m
inimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
Mar 27 22:47:01 2020 CHCKPFW-1 routed[1193]: OSPF IO: neighbor 10.106.18.122(eth3-02.14) LSA <5-10.17.2.11-10.106.18.122> seq 0x80000003 age 1 violates m
inimum LSA spacing rule (LSA ignored).  Current seq 0x80000012 age 3600
 
show ospf command list lots of errors:
 
 
Spoiler
CHCKPFW-1:TACP-15> show ospf errors
show instance 4 ospf errors

Hello Protocol Errors

    Bad Size                  0            Network Mask Mismatch     0        
    Dead Interval Mismatch    0            Hello Duplicate Router ID 0        
    External Option Mismatch  0            NSSA Option Mismatch      0        
    Runt                      0            Hello Timer Mismatch      0        

Link State Update Errors

    Runt                      0            LSU Duplicate Router ID   0        
    LSU TooLow                5            BadCSum                   0        
    BadLSType                 0            ASEinStub                 0        
    Type7inNonNSSA            0            LSU TooNew                2200505  
    BadLSReq                  0            SeqNumWrap                0        
    Invalid SeqNum            0            SummaryinTotalStub        0        
    BadRouterLSASize          0            BadNetworkLSASize         0        
    BadSummaryLSASize         0            BadASELSASize             0        
    BadType7LSASize           0        

Link State ACK Errors

    LSAck Duplicate Router ID 0            LSAck TooLow              0        
    BadSize                   0            QuestionAck               102338311  
    BadLSType                 0        

Link State Request Errors

    LSR Duplicate Router ID   0            BadSize                   0        
    BadState                  0            Empty Request             0        

Database Description Errors

    ASEinStub                 0            Type7inNonNSSA            0        
    MTU                       0            BadLSType                 0        
    NotDuplicate              0            BadSize                   0        
    OptionsMismatch           0            DuplicateLSA              0        
    DD Duplicate Router ID    0            InitSet                   132      
    Runt                      0            MasterMismatch            0        
    SlaveSeq                  0            MasterSeq                 0        
    DD TooLow                 0        

Protocol Errors

    Bad Area ID               0            Area ID Mismatch          0        
    AuthCryptoSeq             0            AuthKey                   0        
    AuthKeyId                 0            AuthKeyTime               0        
    AuthKeyType               0            BadDestination            0        
    Checksum                  0            NoNeighbor                0        
    NoOspf                    0            Size                      0        
    Version                   0            NonLocal                  0        
    VirtualLink               0            NoVirtualNeighbor         0        
    IfDown                    0            PacketType                0        
    Passive Interface         0            TX                        0        
    ZeroRID                   0       
 
 
Summary of routes on the database, both on the FW and the router matches. Any clue how to fix this? Thanks. 
 

 

0 Kudos
1 Reply
Highlighted
Admin
Admin

Open a TAC request please

0 Kudos