Issue with SecureXL

[Tung_Nguyen_Son]

Dear all,

My customer firewall just hang. I see message log which have many log relevant with SecureXL.

Please help me to check this issue?

Aug 25 08:33:02 2020 FWHO-CORE-04 kernel: [SIM4];sim_db_save_conn: failed to save conn <10.63.90.87,51407,10.1.6.13,53,17>, no memory (-4)
Aug 25 08:33:02 2020 FWHO-CORE-04 kernel: [SIM4];sim_db_save_conn: failed to save conn <10.63.90.87,51407,10.2.6.11,53,17>, no memory (-4)
Aug 25 08:33:02 2020 FWHO-CORE-04 kernel: [SIM4];sim_db_save_conn: failed to save conn <10.63.220.45,54045,10.1.6.32,53,17>, no memory --4)
Aug 25 08:33:02 2020 FWHO-CORE-04 kernel: [SIM4];sim_db_save_conn: failed to save conn <10.63.220.45,54045,10.1.6.32,53,17>, no memory --4)
Aug 25 08:33:02 2020 FWHO-CORE-04 kernel: [SIM4];sim_db_save_conn: failed to save conn <10.0.67.119,59384,10.2.6.11,53,17>, no memory (-4)
Aug 25 08:33:02 2020 FWHO-CORE-04 kernel: [SIM4];sim_db_save_conn: failed to save conn <10.63.90.87,51407,10.2.6.13,53,17>, no memory (-4)

Aug 25 08:38:41 2020 FWHO-CORE-04 kernel: [SIM4];pkt_handle_f2v_if_needed: dropping packet (failed to send notification), conn: <10.1.12.118,45928,10.1.29.68,8443,6>
Aug 25 08:38:41 2020 FWHO-CORE-04 kernel: [SIM4];sim_mgr_host_send_message_start_cb: halloc (216 bytes) failed.
Aug 25 08:38:41 2020 FWHO-CORE-04 kernel: [SIM4];sim_mgr_nt_start_ex: cpaq_client_job_start failed type=ntConnUpdate.
Aug 25 08:38:41 2020 FWHO-CORE-04 kernel: [SIM4];sim_mgr_host_send_message_start_cb: halloc (216 bytes) failed.
Aug 25 08:38:41 2020 FWHO-CORE-04 kernel: [SIM4];sim_mgr_nt_start_ex: cpaq_client_job_start failed type=ntConnUpdate.
Aug 25 08:38:41 2020 FWHO-CORE-04 kernel: [SIM4];sim_mgr_host_send_message_start_cb: halloc (216 bytes) failed.
Aug 25 08:38:41 2020 FWHO-CORE-04 kernel: [SIM4];sim_mgr_nt_start_ex: cpaq_client_job_start failed type=ntConnUpdate.
Aug 25 08:38:41 2020 FWHO-CORE-04 kernel: [SIM4];sim_mgr_host_send_message_start_cb: halloc (216 bytes) failed.
Aug 25 08:38:41 2020 FWHO-CORE-04 kernel: [SIM4];sim_mgr_nt_start_ex: cpaq_client_job_start failed type=ntConnUpdate.
Aug 25 08:38:41 2020 FWHO-CORE-04 kernel: [SIM4];sim_mgr_host_send_message_start_cb: halloc (216 bytes) failed.

[G_W_Albrecht]

I can only see the rebbot:

Aug 25 08:42:10 2020 FWHO-CORE-04 xpand[22919]: tungns localhost t +volatile:clish:tungns:22028 t
Aug 25 08:42:10 2020 FWHO-CORE-04 clish[22028]: User tungns logged in with ReadWrite permission
Aug 25 08:42:17 2020 FWHO-CORE-04 clish[22028]: cmd in VS0 by tungns: Start executing : reboot (cmd md5: 2b755b30293f2f9f79ec1c86ab0943f2)
Aug 25 08:42:17 2020 FWHO-CORE-04 clish[22028]: cmd in VS0 by tungns: Processing : reboot (cmd md5: 2b755b30293f2f9f79ec1c86ab0943f2)
Aug 25 08:42:18 2020 FWHO-CORE-04 xpand[22919]: Final
Aug 25 08:42:19 2020 FWHO-CORE-04 logger: Set interfaces down before reboot
Aug 25 08:42:19 2020 FWHO-CORE-04 kernel: bonding: bond1: link status down for idle interface eth1-01, disabling it in 200 ms.
Aug 25 08:42:19 2020 FWHO-CORE-04 syslogd: sendto: Network is unreachable
Aug 25 08:42:19 2020 FWHO-CORE-04 syslogd: sendto: Network is unreachable

Could it be that your customer firewall is VSX ? I would rather contact TAC...

[_Val_]

Failover to the second cluster member. Reboot. If the problem persist, check that you are on the lates Jumbo.

Open a TAC case as soon as possible as well.

[dgr]

@Tung_Nguyen_Son Did you get anywhere with this? We're also seeing the same messages (apart from the sim_db_save_conn). After a while we start seeing packets being dropped and the problem gets progressively worse and we need to failover to standby and reboot. The issue reoccurs maybe every 3 months. TAC just told us to patch which hasn't helped. Our set up is VSX. Although we have two VSX clusters, same hardware, same software, same JHF and we only see this happening on one.

[lowstett · Accepted solution]

Hi,

I have had this exakt problem with a customer this morning. Same entries in the messages-logs as shown by @Tung_Nguyen_Son.
I also had to do what @dgr mentioned, failover and reboot the gateway.
Simultaneously to this, the memory on the gateway jumped up to over 90% used, from just 55% before, I am guessing that could be what is causing these log-entries.

In this environment we are running R80.30 JHF Take 155 in VSX with VSLS on 2 gateways. Only 1 of the VSX GWs have had the issue so far.
Planing to patch to JHF 217 on Monday.