r81.20 OPEN management server log issues

[vishnukanth]

Hi everyone,

I have integrated my 9000 gateways with the OPEN Management Server R81.20 and initially verified that traffic logs were appearing correctly.

After some time, when I checked again, there were no traffic logs visible on the Management Server.

I have already performed the following checks:

Verified using tcpdump on the Management Server and confirmed that the gateways are sending logs to the Management Server.

1 Verified all Management Server processes.

2 Performed cpstop / cpstart.

3 Rebooted the Management Server.

However, the issue still persists and the logs are not visible in SmartConsole.

Could anyone kindly help me identify what else could be causing this issue?

BR,
vishnu

[emmap]

Is the date and time and timezone right on your server and your gateways?

Is the gateway reporting any issues? Check with 'cpstat fw -f log_connection'

Is the management server configured to index the logs? 

If you click the hamburger menu next to the search query line and manually open the fw.log file, do you see traffic logs?

[vishnukanth]

Hello emmap,

1 the date and time and timezone is correct on the mgmt server

2 management server configured to index the logs is enabled (log indexing is enabled)

3 verified the cpstat fw -f log_connection in the GW and it shows disconnected to the mgmt server ip and opening the fw.log file shows empty.

I think that i need to T-shoot with GW is not connected with the MGMT SERVER & I have attached the snaps for your reference

BR,
vishnu

[emmap]

If this is a new setup, do an 'install database' on your management server from the main menu in smart console. Otherwise yes troubleshoot the connectivity. And make sure the date and time are right on the gateways too.

[vishnukanth]

sure emmap! will check the connectivity and let you know

BR,

vishnu

[Martijn]

Hi,

There can be many reasons why log is not send/received.

Did you check the gateway is sending the log to the SmartCenter with tcpdump on the gateway and SmartCenter
Are Implied Rules enabled?
Can you check cpview the correct log-server is listed? cpview -> advanced -> logging?
Enough free disk-space on the SmartCenter? Correct time on SmartCenter and gateway?
Can you check the Master-file ($FWDIR/conf/masters) to see if the correct server is listed?

Some times a policy install will start logging from the gateway to the SmartCenter again.
Performing a fw logswitch on gateway and SmartCenter may re-initialize the log-processes.

sk40090 - Troubleshooting Check Point logging issues when Security Management Server / Log Server is...

Martijn 

[vishnukanth]

Hi Martijn,

Yes, I have verified this using the **tcpdump** command on the Management Server. When I initiate traffic from the gateway to **8.8.8.8**, I can see that the gateway is sending the logs to the Management Server. Also, the **implied rules are enabled by default**.

I have attached screenshots from **cpview → Advanced → Logging** for your reference. There is **no Log Exporter configuration** on this setup.

This is a **new server**, and we have **sufficient disk space available**.

I also checked the **masters file**, and everything looks correct. Screenshots are attached for reference.

After integrating the gateway with the Management Server, I was able to see traffic logs initially. However, after about a week, I am no longer able to see any logs in SmartConsole. I have tried installing the policy and database multiple times, but the issue still persists.

Additionally, I have gone through **SK40090** and verified all the points mentioned there. The only pending step is enabling debug, which I plan to check as a last step.

Please let me know if you need any additional information or specific outputs from my side.

Thanks & regards,

vishnu

[Don_Paterson]

Try this on the log server (expert mode)

fw log -ftn

CTLR + C to stop

If you see output from the command it means that the fwd process on the log server (SMS) is receiving and processing income/received logs. 

The commands evstop and evstart will stop and start the log server processes on the log server (SMS)

Doing the tcpdump (or cppcap) for just port 257 will show logs being sent/received. 

Did NAT get changed or configured on the management server object?

R82 has a few new NAT settings for management. 

[Don_Paterson]

From the screenshot the gateway reports that it is logging locally. 

That implies a network problem between the SG and the SMS. 

The SG needs port 257 destination port access to the log server. 

Could something be blocking that?

The fwd process exists on both SG and SMS and handles logs. 

If needed it can be stopped and started independently. 

To Stop

Management Server / Security Gateway:

cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd"

To Start

Management Server / Security Gateway:

cpwd_admin start -name FWD -path "$FWDIR/bin/fw" -command "fwd"

Thats from Sk97638

A reboot of the firewall is a valid troubleshooting step too. 

[Don_Paterson]

https://community.checkpoint.com/t5/Security-Gateways/R8x-Ports-Used-for-Communication-by-Various-Ch...

Sometimes a 3rd party or externally managed SG/firewall is introduced between the SG and the SMS and the Check Point specific ports are blocked. 

What you are seeing is a symptom of that scenario. 

There are a lot of Check Point ports to consider. 

Some are used every day (logs, monitoring and policy installation) and others every 2 to 3 years (automatic certificate renewal).

They must all be allowed or the product cannot work as it was designed to. 

[Don_Paterson]

When you said  "I can see that the gateway is sending the logs to the Management Server.", how did you test that?

Here is a valid packet capture test to check if log traffic is arriving to the Log server:

cppcap -i eth0 -f "port 257"

Replace eth0 with the management port.

In the output you should see the log server as the destination and port 257.

That together with output from fw log -ftn (showing the gateway details) should confirm a working logging solution

cppcap -i eth0 -d in -f "port 257 and host <mgmt IP of SG>"

-d in will show only inbound and and host w.x.y.z will filter for just the log traffic from the target SG IP address.

This procedure could be useful for when the connectivity is restored:

List log files stored on the SG (run on the SG in expert mode):

fw lslogs

Switch the current Active log file:

fw logswitch

To pull the logs into the log server (run this on the Log server in expert mode):

WARNING: If there are a lot of logs stored on the SG, or just a few large log files then it can take a while for all of the logs to copy .

fw fetchlogs <gateway name (SmartConsole name)>

All logs will be moved to the Log server.

The log files are renamed and the gateway name is added to the beginning of each file name (with two underscores __)

You can automate that log fetch with an extra SmartConsole configuration (on the gateway/cluster object).

This will take care of any logs stored locally on the gateway being transferred automatically to the Log server.

If you search the logs for type:Control then you will see System Monitor messages showing the gateway starting to log to the Log server again.

More > Sys Message: started loggin CN=<gw SIC cert< 

[Don_Paterson]

That is the right SK to share 👍

netstat is important for the testing of connectivity.

I will suggest they update the SK to include cppcap along with or instead of tcpdump.

The associated SK at the bottom is good too, Doesn't make sense for it to be Advanced level(?).

https://support.checkpoint.com/results/sk/sk98317 

This thread caught my attention because I am running through the new R82 CCTA courseware and lab steps right now.

Heavy focus on packet captures in this one, with lots of new content to cover cppcap, tcpdump and fw monitor and Wireshark packet capture analysis.

Logging troubleshooting still has a full chapter/module in the new version.

https://training-certifications.checkpoint.com/

[vishnukanth]

Hi don paterson,

thanks for your T-shoot steps let me check this and i will share you the outputs.

thanks,

Vishnu