cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post

Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

Hi there guys, I'm seeing this message  "simi_reorder_enqueue_packet" on /var/log/messages. Is this an indication traffic congestion? My network is  momentarily encountering intermittent application connectivity especially on VOIP. As usual, no drops are seen on tracker and zebug. 

Hope someone had encountered this.

16 Replies
Admin
Admin

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

It seems to suggest network congestion/fault of some sort from the few TAC cases opened on this error message.

It's probably worth opening one so we can investigate more closely.

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

I'll pass on my experience with this error in case it helps someone in the future.  Pushed out inline policy rules on Wednesday.  On Friday, DHCP broke.  It was going through the firewall, and we found the below in /var/log/messages and zdebug:

[Expert@CPFW2:0]# less /var/log/messages

Jan 18 12:41:38 2019 CPFW2 kernel: [SIM4];simi_reorder_enqueue_packet: reached the limit of maximum enqueued packets for conn:<*.*.*.*,67,*.*.*.*,67,17>, fw_key:<*.*.*.*,67,*.*.*.*,67,17> !

Before the inline policy, DHCP request were going through the firewall on a rule with DHCP specificed in the service, and replies were coming back on UDP 68 on a rule with Any as Dest and Service.  When that rule was changed to inline, I can look back in the logs and no longer see the UDP 68 replies.  Not sure why it didn't log on that Any rule because there was any Any allow inline app control rule at the bottom of it.  Anyway, after 1 and 1/2 days of UDP 68 traffic not getting through, this simi_reorder queue got full?

We made a return rule specifying udp 68, with Accept and no inline layers, and pushed it out.  That didn't fix it.  We had to uncheck UDP 67 and 68 from Sync, push policy, and force a failover.  That forced DHCP into a new connection, and then we could see logs from the UDP 68 DHCP reply traffic (from DHCP servers back to wireless controllers).  Don't know if that will permanently solve the issue or not.

Asking Check Point TAC for info about this queue, if there's a way to check it, and clear it out.  What it does, and why this happened.  I've got nothing back so far.

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

Hi there,

currently we have this issue too. It came up after a migration from R77.30 to R80.20 with latest ongoing jumbo. In our case its affecting CIFS transfers(but also other protocols). Instead of getting near line speed transfers (1GB/s) we have about 200Mbits/s , BUT: during high traffic CPU goes high on 2-3 cores to 100%, thus, IPS starts to bypass, etc. Currently a ticket is open, not solved actually...

0 Kudos

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

interesting: when the cpu load is high, there is a process called: "cphwd_q_init_ke", which consumes 100% on one core. Anyone aware about this one? The other "cpu core eaters" are fw_worker_x processes...

enanosca
Ivory

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

I;m also seeing this behavior but do not find any info on docs. were you able to find anything related?

PID PPID CMD %MEM %CPU
5153 1 [fw_worker_1] 0.0 59.8
5154 1 [fw_worker_2] 0.0 59.2
5152 1 [fw_worker_0] 0.0 59.2
5156 1 [fw_worker_4] 0.0 59.0
5157 1 [fw_worker_5] 0.0 58.8
5155 1 [fw_worker_3] 0.0 55.7
7059 1 [cphwd_q_init_ke] 0.0 41.9
7058 1 [cphwd_q_init_ke] 0.0 9.1

0 Kudos
ChammiK
Ivory

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

What does the cphwd_q_init_ke process do? I'm seeing this process going a little high at the moment.

0 Kudos

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

I have created a SR with this problem and got an answer:

Put “simi_reorder_hold_udp_on_f2v=0” in simkern.conf and reboot.

 

My concern is that no one can tell me what "simi_reorder_hold_udp_on_f2v" does and what it affects. I want to know the consequences of applying this parameter. I can’t finde anything in support center or on google.

 

Do any of you know?

0 Kudos
Highlighted
Admin
Admin

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

I'll see if I can get someone from R&D to comment.

Offhand, the name suggests it is something related to packet reordering of UDP packets prior to an acceleration decision being made.

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

I have been given this solution as well.  Have not implemented it.  I'll let you know if I have any issues.  I'll see if you find out what it's doing.

Admin
Admin

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

Looks like you got an answer in your SR. Smiley Happy

For everyone else reading along, I will explain.

  • Firewall and PPK (SecureXL) communication is asynchronous in R80.20 and above.
  • As a result, packets sometimes need to be held in PPK.
  • For UDP packets, it's generally ok to have some packets not arrive in the correct order.
  • This parameter tells PPK not to hold UDP packets.

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

Hi,

Any news on this? I just got the same message on a customer recently upgraded to R80.20. I had failed over to the other cluster member and resolved our problem, at least temporarily.

0 Kudos
Admin
Admin

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

If anyone has a TAC SR on this, please send to me in a private message.

0 Kudos

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

Hi Sir,

You may refer to this case. Thanks. 

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

0 Kudos
Admin
Admin

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

I said a private message Smiley Happy

In any case, this particular SR is waiting on you to provide debugs.

Did you happen to try the fix suggested in this thread?

phlrnnr
Copper

Re: Message seen on /var/log/messages - "simi_reorder_enqueue_packet"

It appears that Checkpoint has issued sk148432 for this issue.  The recommendation is to apply a hotfix.

Symptoms

  • The following drop is seen in /var/log/messages file:
    [kern];[tid_0];[SIM-];simi_reorder_enqueue_packet: reached the limit of maximum enqueued packets for conn

Cause

As part of the changes made in R80.20, SecureXL behave asynchronicaly with the Firewall. This means that Firewall and SecureXL might handle the connections separately. In order to avoid situation where SecureXL forward packets to the network before the FW finished the inspection, SecureXL will hold the packets in a queue, so we will have order for the packets.

This new mechanism is called: "simi_reorder".

In case that the queue is full, all the packets will be released and dropped. The main impact is for UDP packets, as TCP will never get the queue to be full.

Solution

Contact Check Point Support to get a Hotfix for this issue. 
A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix. 
For faster resolution and verification please collect CPinfo files from the Security Management and Security Gateways involved in the case.

There are 2 SIM module kernel parameters which control this bevaviour (please refer to sk147392, on how to set the SIM parameters) :

  1. simi_reorder_max_packets
  2. simi_reorder_hold_udp_on_f2v

simi_reorder_max_packets is controlling the maximum packets which the Q can hold - it is less recommended to change it, as it will consume additional memory, depending on how much the Q is increased.

simi_reorder_hold_udp_on_f2v controls the mechanism for UDP packets, it's value could be 0 or 1. 1 will enable the simi_reorder mechanism for UDP packets, 0 will disable it, so all the packets will be sent for the network, and might have some asynchronism between FW and SecureXL.

The hotfix is changing the mechanism behavior, in a way that when the queue is full, the packets will be released and not dropped.