Entries in log/messages

[lucafabbri365]

Hello all,

I was checking /var/log/messages from the active node (Check Point R80.10 Cluster in Open Server - Take 154) and I found these entries; please can you help me to understand what they means and address the investigation:

1. kernel: [fw4_0];fwmultik_dispatch_inbound: instance mismatch (on connection <IP Address>(80) -> <IP Address> (9307) IPP 6): predefined says 2 lookup says 0)

2. kernel: [fw4_0];fwpslglue_do_log: Log buffer is full

3. kernel: [fw4_1];FW-1: Starting CUL mode because CPU usage (81%) on the local member increased above the configured threshold (80%).
4. kernel: [fw4_0];FW-1: SIM (SecureXL Implementation Module) SecureXL device detected.

kernel: [fw4_1];fwioctl: Policy has ended. Continuing extending dead timouts (fwha_cul_policy_done_time=8522491)

I also notices the $FWDIR/log/fw.log  is unreadable:

Is that a normal behavior ? I don't think so...

Any useful information is really appreciated.

Thank you very much,

Luca

[Matt_Ricketts]

SK113398 looks like the problem you may be having. If it is, this issue was fixed and is part of the JHF take 142 or higher for R80.10.

As for the unreable fw.log file, that is definitely not normal. If you are seeing logs appear in SmartLog, it could just be a view issue. You may try to do a manual log switch and see if that corrects it. If not, then I would call into TAC.

[lucafabbri365]

Hello Matt,

thank you for your reply, as reported initially, we already have Take 154 installed:

# cpinfo -y all

This is Check Point CPinfo Build 914000190 for GAIA
[IDA]
HOTFIX_R80_10

[CPFC]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 154

[FW1]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 154

FW1 build number:
This is Check Point's software version R80.10 - Build 124
kernel: R80.10 - Build 001

[SecurePlatform]
HOTFIX_R80_10_JUMBO_HF Take: 154

[CPinfo]
No hotfixes..

[PPACK]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 154

[DIAG]
HOTFIX_R80_10

[CVPN]
HOTFIX_R80_10
HOTFIX_R80_10_JUMBO_HF Take: 154

[CPUpdates]
BUNDLE_CPINFO Take: 0
BUNDLE_R80_10_JUMBO_HF_SC Take: 83
BUNDLE_R80_10_JUMBO_HF Take: 154

Regarding fw.log...I'll try your solution.

Thank you,

Luca

[Dmitry_Krupnik]

Hello Luca,

The fw.log is binary file and can be opened by CPLogFilePrint only. For it just perform the “CPLogFilePrint $FWDIR/log/fw.log” command under expert mode.

Tell me please, which CPUs usage do you see on the members? You can see this information in the CPView.

Regards,
Dmitry Krupnik

[lucafabbri365]

Hello Dmitry,

first of all, the "CPLogFilePrint $FWDIR/log/fw.log" worked fine, thank you.

Now, regarding the point 3: kernel: [fw4_1];FW-1: Starting CUL mode because CPU usage (81%) on the local member increased above the configured threshold (80%).

The entry appears when there is an high CPU usage (one of three fw_worker_<nn> process consumes about 70%, 80%, 85% and 90% percentage).

Casually I reproduced the issue: I was downloading a Windows 10 iso file from my client PC, then I noticed a 57% CPU usage (see screenshot below).

We have HTTPs Inspection and Application & URL Filtering enabled; Application & URL Filtering is enabled for ALL; however HTTPs Inspection is in ByPass for ALL, with the exception of my client computer where it was set to Inspect (because I'm testing HTTPs inspection before applying it to all users).

How is possible a simple file download from one client will use a lot of that CPU ? Do you think it is due to limited CPU license we are just using in our Check Point ?

Regarding the other points, I'll give you more information asap.

Thank you,

Luca

[Dmitry_Krupnik]

Hello Luca,

According to logs:

1) "fwmultik_dispatch_inbound: instance mismatch". I have asked the RnD owners about this log. If you see any impact, I suggest to open request in the TAC

2) "fwpslglue_do_log: Log buffer is full" First of all make sure, that logging works in the default mode, perform the "fw ctl debug 0" command under expert mode. After it take a look the sk52100

3) "Starting CUL mode because CPU usage (81%)". This log means, that Cluster Under Load (CUL) mechanism works as expected. Question, that important to understand - why your appliance has high CPU load and how often it happens. If I understand correctly generally the CPU usage of your GW exists in the allowable range, but during some downloads the usage greatly increases.  I will strive to discover this question internally, but won't be able to provide ETA. If you see any impact, I suggest to open request in the TAC.

4) "SIM (SecureXL Implementation Module)" - for me it looks like general SecureXL log, I don't think that you need to worry about this message.

Regards,

Dmitry

[PhoneBoy]

When was CPLogFilePrint added?

Didn't realize that command was present.

[Dmitry_Krupnik]

 This tool exist since R80.10. I am discussing with RnD creating SK, which will describe it.

[PhoneBoy]

Excellent, looking forward to seeing it

CPLogFilePrint seems to provide more details than fw log, which has been around since at least FireWall-1 version 2