R81.20 JHF 89 Fails (0-1-200008 & 0-1-2000107) on CP3200

[Perry_McGrew]

Updated our Mgt and GWs to JHF 89 due to some current issues with Identity Awareness.  After any JHF update, I do a test Policy install (no changes) to verify.   Policy installs failed on the 3200s.  Tried "fw fetch" also from a CP3200 which fails as well.  Ran the Policy.sh debug script on the Mgt server and sent it to TAC.   Tried running it on one of the CP3200 and it just ran forever....had to cancel it when "/" went from 68% to over 92%.  Canceling it automatically removes all the logs it created in /tmp. 

JHF 89 on the 5800 HA Cluster worked fine and I can install policy.  

Typically, these error codes are memory related.   TAC verified memory was available on a 3200.   Waiting for TAC to get back with diagnosis and hopefully a fix.  

[PhoneBoy]

What did fw fetch have to say as far as an error?

[Perry_McGrew]

I don't recall exactly.   But it seemed to track the same as the Policy push -- some sort of memory allocation issue.  fw fetch takes less resources than the Push from the Management.   

I let the Policy.sh debug script run on the CP3200 until the "/" mount point went to 92% and was still climbing.   I grep'd dmesg for ERROR and it was full of these types of messages:

Oct  7 13:08:46 2024 BVILLE-3200 kernel:[fw4_0];[71.245.91.32:56490 -> 1.1.1.1:53] [ERROR]: cmik_loader_fw_context_match_cb: match_cb for CMI APP 31 - DNS_DATA_SOURCE failed on context 201, executing context 366 and adding the app to apps in exception

Oct  7 13:08:46 2024 BVILLE-3200 kernel:[fw4_0];[71.245.91.32:57990 -> 68.237.161.12:53] [ERROR]: cmik_loader_fw_context_match_cb: match_cb for CMI APP 31 - DNS_DATA_SOURCE failed on context 201, executing context 366 and adding the app to apps in exception

Oct  7 13:26:42 2024 BVILLE-3200 kernel:[fw4_0];[192.168.8.117:60971 -> 192.168.20.254:443] [ERROR]: fwk_install_policy_app_load_prepare: fwk_atomic_load_prepare() failed, error: (14)

Oct  7 13:26:42 2024 BVILLE-3200 kernel:[fw4_0];[192.168.8.117:60971 -> 192.168.20.254:443] [ERROR]: install_policy_mgr_k_load_prepare: load_prepare failed for app: (FW), app_id: (1), app_position: (2)

Oct  7 13:29:47 2024 BVILLE-3200 kernel:[fw4_0];[192.168.20.243:57946 -> 142.250.65.219:443] [ERROR]: fwk_install_policy_app_load_prepare: fwk_atomic_load_prepare() failed, error: (14)

Oct  7 13:29:47 2024 BVILLE-3200 kernel:[fw4_0];[192.168.20.243:57946 -> 142.250.65.219:443] [ERROR]: install_policy_mgr_k_load_prepare: load_prepare failed for app: (FW), app_id: (1), app_position: (2)

TAC asked for a df -h  

Filesystem                       Size  Used Avail Use% Mounted on

/dev/mapper/vg_splat-lv_current   32G   16G   15G  52% /

I have plenty of free space in "/" .   Have a session with Tier 2 today.   

[Perry_McGrew]

Had session with TAC.  As noted, the issue is memory related -- or lack of memory needed for the install process.  Doing the "watch free -m" on the 3200, and can see the free memory values drop significantly.  When it went below 200MB, the policy would often fail. 

We kept repeating the Access Policy Install and eventually, got it to succeed.  TAC originally told me we needed to increase the RAM on the 3200 which looks to be 8 GB.  I told him I don't believe that one can add RAM to the 3200 -- which he came back and confirmed. 

Our CP3200 sites are very small -- just a few devices and employees.  The CP3200 are used for S2S VPN to our datacenter hosted apps.  

So the issue is technically resolved, but I have asked the tech to let DEV know about this as the CP3200's support R81.20.  Since these devices are fixed RAM, concerned that these JHFs improvements / fixes are going to continue to cause policy install failures going forward.  The answer can't be just to keep trying -- getting the 2000107 / 2000108 errors which say "Call Check Point Support".