suspicious interrupts values

[Wolfgang]

Hello CheckMates,

since two days we see this very low interrupts, normally all cores around 100.000

Looks like everything is running fine but I'm wondering.

Has anyone an idea ?

Wolfgang

[HeikoAnkenbrand]

Hi @Wolfgang,

Are the cores - which show few interrupts - used by SecureXL, CoreXL, NIC's or MQ?

Just do the following, and see what's going on:

# fw ctl affinity -l -r -v –a        -> Infos to FW CoreXL/ SecureXL / FW-Daemon

# cat  /proc/interrupts             -> Infos to NIC's

# cpmq  get                                -> Infos to MQ

I think there are no FW or NIC interrupts running.

[Wolfgang]

@HeikoAnkenbrand 

fw ctl affinity -l -r -v -a
CPU 0:
CPU 1: fw_5
mpdaemon fwd in.acapd rad dtpsd dtlsd rtmd vpnd pepd usrchkd pdpd wsdnsd in.asessiond in.ahttpd lpd in.aufpd cprid cpd
CPU 2: fw_3
mpdaemon fwd in.acapd rad dtpsd dtlsd rtmd vpnd pepd usrchkd pdpd wsdnsd in.asessiond in.ahttpd lpd in.aufpd cprid cpd
CPU 3: fw_1
mpdaemon fwd in.acapd rad dtpsd dtlsd rtmd vpnd pepd usrchkd pdpd wsdnsd in.asessiond in.ahttpd lpd in.aufpd cprid cpd
CPU 4:
CPU 5: fw_4
mpdaemon fwd in.acapd rad dtpsd dtlsd rtmd vpnd pepd usrchkd pdpd wsdnsd in.asessiond in.ahttpd lpd in.aufpd cprid cpd
CPU 6: fw_2
mpdaemon fwd in.acapd rad dtpsd dtlsd rtmd vpnd pepd usrchkd pdpd wsdnsd in.asessiond in.ahttpd lpd in.aufpd cprid cpd
CPU 7: fw_0
mpdaemon fwd in.acapd rad dtpsd dtlsd rtmd vpnd pepd usrchkd pdpd wsdnsd in.asessiond in.ahttpd lpd in.aufpd cprid cpd
All:
Interface eth4: has multi queue enabled

########################################################

cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
0: 58 0 0 0 0 0 0 0 IR-IO-APIC-edge timer
1: 3 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042
4: 796 0 0 0 0 0 0 0 IR-IO-APIC-edge serial
8: 1 0 0 0 0 0 0 0 IR-IO-APIC-edge rtc0
9: 1 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi acpi
12: 4 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042
14: 0 0 0 0 0 0 0 0 IR-IO-APIC-edge ide0
15: 0 0 0 0 0 0 0 0 IR-IO-APIC-edge ide1
18: 185 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi ehci_hcd:usb1, ehci_hcd:usb2, i801_smbus
19: 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi uhci_hcd:usb3, hpilo
25: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
26: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
28: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
29: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
30: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
31: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
33: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
34: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
35: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
36: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
37: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
38: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
39: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
40: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
41: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
43: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
44: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
46: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
47: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
48: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
49: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
51: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
52: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
53: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
54: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
55: 367011 0 0 0 0 0 0 0 IR-PCI-MSI-edge hpsa0-msix0
56: 2479457 0 0 0 0 0 0 0 IR-PCI-MSI-edge hpsa0-msix1
57: 1760457 0 0 0 0 0 0 0 IR-PCI-MSI-edge hpsa0-msix2
58: 1784373 0 0 0 0 0 0 0 IR-PCI-MSI-edge hpsa0-msix3
59: 141469 0 0 0 0 0 0 0 IR-PCI-MSI-edge hpsa0-msix4
60: 1637050 0 0 0 0 0 0 0 IR-PCI-MSI-edge hpsa0-msix5
61: 2005454 0 0 0 0 0 0 0 IR-PCI-MSI-edge hpsa0-msix6
62: 2130786 0 0 0 0 0 0 0 IR-PCI-MSI-edge hpsa0-msix7
64: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
66: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
67: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
68: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
69: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
70: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
71: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
72: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0
73: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1
74: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
76: 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
78: 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
80: 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
82: 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
83: 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
84: 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
85: 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
86: 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
87: 127606685 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4-TxRx-0
88: 2 0 0 0 306906605 0 0 0 IR-PCI-MSI-edge eth4-TxRx-1
91: 1821485890 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth12-TxRx-0
92: 1 0 0 0 1870868362 0 0 0 IR-PCI-MSI-edge eth12-TxRx-1
93: 2333541522 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth13-TxRx-0
94: 1 0 0 0 2608544858 0 0 0 IR-PCI-MSI-edge eth13-TxRx-1
95: 49196197 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-TxRx-0
96: 1 0 0 0 47379082 0 0 0 IR-PCI-MSI-edge eth6-TxRx-1
97: 2642507920 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-TxRx-0
98: 1 0 0 0 2490055713 0 0 0 IR-PCI-MSI-edge eth7-TxRx-1
99: 1671555075 0 1 0 0 0 0 0 IR-PCI-MSI-edge eth8-TxRx-0
100: 0 0 1 0 832205039 0 0 0 IR-PCI-MSI-edge eth8-TxRx-1
101: 4194613299 0 1 0 0 0 0 0 IR-PCI-MSI-edge eth9-TxRx-0
102: 0 0 1 0 3400540664 0 0 0 IR-PCI-MSI-edge eth9-TxRx-1
103: 488360252 0 1 0 0 0 0 0 IR-PCI-MSI-edge eth10-TxRx-0
104: 0 0 1 0 524962406 0 0 0 IR-PCI-MSI-edge eth10-TxRx-1
105: 368603030 0 1 0 0 0 0 0 IR-PCI-MSI-edge eth11-TxRx-0
106: 0 0 1 0 373062218 0 0 0 IR-PCI-MSI-edge eth11-TxRx-1
107: 15 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
108: 1 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6
109: 1 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7
110: 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge eth8
111: 0 0 3 0 0 0 0 0 IR-PCI-MSI-edge eth9
112: 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge eth10
113: 0 0 3 0 0 0 0 0 IR-PCI-MSI-edge eth11
114: 3 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth12
115: 2 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth13
NMI: 153950 259979 273430 260031 139620 325117 252609 247570 Non-maskable interrupts
LOC: 606960126 1496161743 1516712755 1515859544 598852372 1486343016 1519914383 1519267800 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 153950 259979 273430 260031 139620 325117 252609 247570 Performance monitoring interrupts
IWI: 10311987 1973570 1175500 1157627 9738005 1719067 1217420 1161513 IRQ work interrupts
RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries
RES: 489475300 146511799 137065572 144234929 74916815 131759505 144426018 146804668 Rescheduling interrupts
CAL: 2183 1972 1787693 1830594 2180 1985 1973624 1944668 Function call interrupts
TLB: 2113 1525917 1223944 1585061 2920 1561386 1438032 1382722 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 4846 4846 4846 4846 4846 4846 4846 4846 Machine check polls
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 Posted-interrupt notification event
PIW: 0 0 0 0 0 0 0 0 Posted-interrupt wakeup event

###########################################################

cpmq get
Current multiqueue status:
Total 8 cores. Multiqueue 2 cores
i/f type state mode cores
------------------------------------------------------------------------------------------------
eth4 ixgbe Up Auto 0,4
eth6 ixgbe Up Auto 0,4
eth7 ixgbe Up Auto 0,4
eth8 ixgbe Up Auto 0,4
eth9 ixgbe Up Auto 0,4
eth10 ixgbe Up Auto 0,4
eth11 ixgbe Up Auto 0,4
eth12 ixgbe Up Auto 0,4
eth13 ixgbe Up Auto 0,4

[Wolfgang]

strange, from time to time it looks like this:

[Muazzam]

This happened to one of my 13800 gateway. All of a sudden one day it starts showing almost similar behavior like yours. We did not find what was causing this but after the upgrade from R80.20 T47 to T103 this problem was over.

[HeikoAnkenbrand · Accepted solution]

Hi @Wolfgang,

The distribution looks good. 

CPU 0: Auto MQ
CPU 1: CoreXL fw_5
CPU 2: CoreXL fw_3
CPU 3: CoreXL fw_1
CPU 4: Auto MQ
CPU 5: CoreXL fw_4
CPU 6: CoreXL fw_2
CPU 7: CoreXL fw_0

I have a suspicion that cpview is reporting false values in this view. With R80.20/R80.30 many CPU interrupts are displayed equally. That can't really be right. 

Furthermore, I also don't believe that a CPU  can handle 4.xxx.xxx.xxx interrupts per second. The CPU clock should be slower than the interrupt value 🙂 So it must be a reporting issue. 

I have noticed the same behaviour with many firewalls. Maybe Check Point should take a look at this. Or you should open a TAC case.

[yo]

We have the same problem.

[FedericoMeiners]

@Wolfgang 

Happened to me in many customers running R80.20.

We opened a SR and the TAC told us it was only a cosmetic error.

[Timothy_Hall]

Seen this many times and it is just cosmetic.  I think it is some kind of counter overflowing, similarly to how very large numbers can sometimes be displayed as a negative number because the binary representation got so large it flipped the first/leading/leftmost bit from 0 to 1, and the counter was a signed value, where a leading bit of 1 indicates a negative number.  There is probably some significance to the number 4,294,967,276 when viewed in binary format, but it is Friday afternoon and I'm not in the mood to do binary math at the moment...another time perhaps.  😀

[HeikoAnkenbrand]

Hi @Timothy_Hall 

I did the math for us one Saturday night.

Linux 32-bit systeme modern int32_t or uint32_t with the max value unsigned 4.294.967.295 and signed from −2.147.483.648 to 2.147.483.647.

Is very near to this value 4,294,967,276. If we do the math, we might get the right interrupt value:

    4.294.967.295
  - 4,294,967,276
-----------------
                       19
                    =====

I think someone has programmed a typical int32_t vs. uint32_t overflow error when the mathematical signs change. 

Solution:

Add this code in cpinfo.

if( interrupt  > 2.147.483.647 ) {
  interrupt = 4.294.967.295 - interrupt;
}

What surprises me even more is that the same error with the same mathematical value (4,294,967,276) appears in several cpu lines. A small error has occurred during programming of this code.

@PhoneBoy Maybe you could forward this to TAC or R&D. 

PS:
But that's also how I messed up my grade in c++ programming during my studies.😁

[HeikoAnkenbrand]

Hi @Wolfgang,
Hi @Timothy_Hall, 

I have found a solution to the problem 🙂

You can use the following CLI command to display the interrupts per second:

# mpstat -P ALL

[Wolfgang]

That‘s great @HeikoAnkenbrand 

Wolfgang