Policy install failes several times until it is successful

[michael_labos]

When we try to install policy it fails several times until one of the tries is successful

when it fail, it fails with different error codes:

Usually: Policy installation failed on gateway. If the problem persists contact Check Point support (Error code: 0-2000195).

We restarted the appliance once but did not helped.

We are running a Checkpoint 6200 with R80.30

Update: the memory was installed today (8gb to 16gb upgrade) and policy install is working now.

Thanks to every taking their time to help.

[_Val_]

Please look into sk174391 and let us know if it helps. 

Anyway, a TAC case is highly recommended. An intermittent problem is usually related to some performance issues, so it may need to be reviewed by a support engineer.

[michael_labos]

Thank you for the answer, I looked at that SK before but couldn't understand how to see the debug output showing there to see if I get similar errors before changing any settings.

We also order RAM upgrade for the appliance so we are waiting to do that first before we contact support. 

[Chris_Atkinson · Accepted solution]

If the unit has only 8GB RAM it could potentially be a memory utilization issue.

The issue may temporarily be solved by a reboot but may ultimately need one or both of the following:

1. Memory upgrade (sizing/capacity issue)

2. Software/version upgrade (memory leak)

As Val has said TAC should be able to assist you in troubleshooting further.

[michael_labos]

Yes, we have the base unit with 8GB RAM, we will wait a few days that another 8GB will arrive to install it and see if that resolves the issue, if not that we will contact support.

[michael_labos]

The Memory upgrade did solve the issue, thank you.

[the_rock]

I had seen before where something like this could be caused by excessive amounts of revisions. So in smart console, under logs and settings, just find section for revisions and see number of them there. If there is lots, you may wish to delete some and try again.

[michael_labos]

I found Revisions under "Manage & Settings" they go back 2 and a half years back, so there are at least 600(20 per month) revisions there, I don't see an option in the GUI to delete in bulks only one by one

[the_rock]

There is, 100%. All you do, right click on one of them, actions -> purge and it will give you below warning:

 

Andy

[michael_labos]

Yes, I understand on how to delete a single one, I don't see an option to delete multiple revision in that list.

I also see now where it says how much there is, and there is around 1000

[the_rock]

Look at my screenshot, its pretty self explanatory. So, it shows you it would delete THAT revision and ANY revisions BELOW that one. So say you have 1000 revisions and you right click on number 10 from top, actions -> purge, it will delete that one and ANY below it, so you would be left with only 9 revisions. You understand? We can do remote session and I can show you if its confusing.

[michael_labos]

Sorry my mistake, I misunderstood the text from the screenshot, it works as you say.

I will give it a try to see if that helps.

[the_rock]

All good brother...and dont worry, CAPS LOCK were not meant as yelling or screaming, more sign to point out IMPORTANT things ; - )

[michael_labos]

Because it was my mistake I disserve the yelling 😛

[the_rock]

Its all good, its just my eastern European way, thats how we type in English hahaha. Anyway, lets get past that. Did it help?

[michael_labos]

I deleted/purged all pre 2022 revisions leaving around 150, did not helped.

[the_rock]

That sucks...ok, then I would follow what @Chris_Atkinson and @Timothy_Hall suggested, because it does make sense, for sure.

[Timothy_Hall · Accepted solution]

As Chris said this is almost always caused by a transient resource shortage on the gateway, usually memory.  The process of loading a new policy into the INSPECT code is very CPU and memory intensive on the gateway.  Best way to diagnose this is to run top and watch the waiting for I/O (wa) value, if it stays perennially high throughout the installation process the gateway is swapping and needs more memory.  Brief spikes of wa during this process are normal behavior.

[michael_labos]

I run top while installing policy and the only place where I see "wa" value is for each CPU in top, before installing the policy they idle at 0, when I install policy in it ranges between 1-7 with two or 3 jumps to 25-40 for less than a second.

[jberg712]

Is your Gateway and Management Standalone (i.e. together on the same 6200 appliance)?  Or do you have a distributed deployment where the management and gateway are separate?

For certain failures like this sometimes I will dig into the ELG logs to see if something can give me more detail.  I'll do an SSH session and run a 'tail -f' on the file to watch the log data in real-time and capture it in a putty session output or just analyze the log file.

There is an install_policy.elg log.  When you're in expert mode, you can run a 'tail -n 50 -f $FWDIR/log/install_policy.elg' and watch the log when installing policy... or just review the log itself.   It might have something useful or it might not but it should give you more detail into what's going on and maybe find something it's not happy about.