Policy installation failures with Error code: 0-2000183-27

[Vladimir]

I am working on the issue that one of my clients experiencing with their HA cluster. Policy installation on one of the gateways started failing with 0-2000183-27 error. 

The healthcheck.sh shows historical issues with smem, kmem (with incorrect notation about 2GB limit due to 32 bit kernel on 64 bit unit).

CPINFO fails with Could not verify CK: Internal Error.

Scheduled backups dropped from reasonable size to something like 60K around the time the problem manifested itself.

Obviously, I am far from finished troubleshooting, but wanted to check if the above mentioned symptoms ring a bell for anyone here.

Sanitized healthceck output is attached.

Thank you,

Vladimir 

[Jerry]

take 56 ... was very bugy mate, make it at least 112 and see how the healthcheck goes.

imho this is something with gaia itself but df seem ok to me, cpu wise - no issues,

very interesting scenario, wonder what if ... you do take 112 then 121 also making jumbo acumulator.

please let me know how it goes, I really wonder what a heck ...

--jerry--

[Vladimir]

I'll let you know how it goes. CP Pro Serve was not very helpful: reinstall from scratch. I do not mind, but would rather understand the cause of the issue or, at the very least, the conditions that make it possible.

Am actually not sure if the management requires upgrade to the same version first, before CXL members. Have to look it up. Will be grateful for any pointers.

Thnx,

Vladimir

[Jerry]

this is also interesting:

Installed Hotfixes:
This is Check Point CPinfo Build 914000182 for GAIA
[KAV]
   HOTFIX_R80_10
[IDA]
   HOTFIX_R80_10
[CPFC]
   HOTFIX_R80_10
[FW1]
   HOTFIX_R80_10
FW1 build number:
This is Check Point's software version R80.10 - Build 423
kernel: R80.10 - Build 031
[SecurePlatform]
   HOTFIX_R80_10_JUMBO_HF    Take: 56
[CPinfo]
   No hotfixes..
[DIAG]
   HOTFIX_R80_10
[PPACK]
   HOTFIX_R80_10
[CVPN]
   HOTFIX_R80_10
[CPUpdates]
   BUNDLE_R80_10_JUMBO_HF    Take: 56

# Hardware Platform Checks:
PL-10-00
Platform: PL-10-00
Model: Check Point 5400
Serial Number: 1620BA1083
CPU Model: Intel(R) Pentium(R) CPU G3420
CPU Frequency: 3192.849
Number of Cores: 2
CPU Hyperthreading: Disabled

*** I think this was installed as 32bit am I correct ? see below:

KMEM Warning:
 Kernel memory had 8 failures.
Presence of kmem failed allocations means that some applications did not get memory.
This is usually an indication of a memory problem; most commonly a memory shortage.
The natural limit is 2GB, since the Kernel is 32bit.).

[Vladimir]

The message bout 32 bit is misleading, as it is a 64 bit system.

It may have been installed as 32, but was changed to 64 before being deployed in production some 1/2 year ago.

[Jerry]

32 cannot be changed to 64 just like that !

afaik this is irreversible process and may lead to the issues indeed you’ve mentioned.

do find out what’s the story about the kernel version as it seem important in your case

(sorry got no access to the sk db just now)

Jerry

[Vladimir]

Hmm...:

[Jerry]

means it’s been installed as 64bit indeed

[Vladimir]

Point being that you can change the kernel bit settings after the fact, it simply requires a reboot to take effect and I personally prefer doing it before anything else on the appliance or VM is configured

See Setting Gaia kernel edition from 32-bit to 64-bit  sk94627.

That being said, I know that this particular unit is 64 bit and thus am puzzled by the message in the helthcheck.sh results, unless it meant to be a generic statement pointing to a high probability cause.