Solved: SG logging behavior

Knut99

Hello,

I am very new with Checkpoint's Security Gateways so pardon my lack of general knowledge about the product.

I have to debug a communication issue between our network and one of our client and I am a bit lost about how the Security Gateway log (or don't log in this case) certain events.

First a bit of information and topology:

We have a VSX deployment (R81.10 JHF T94) consisting of 3 VS connected by 2 VSW on top of 2 OpenServers.

This particular client is connected through a private routed link connected to one of the VS (Let's call it VS2).

Our users Vlan are, on the other hand, connected to another VS, we call it VS1 here. So in order to reach the customer infrastructure we have to go through VS1 and VS2.

As for filtering, we are using a Layer that is present in both security policies (VS1 & 2).

Finally, VS2 is source nating our outgoing traffic with its interface IP because we do have overlapping subnets with this client.

Now for the issue:

The client expose a webservice in his network that we cannot reach (timeout).

I cannot see any firewall or nat log regarding this traffic on VS2.

What I know so far:

From the smart console I can see that traffic is logged and accepted by VS1 but the is no log from VS2.

Client's network team confirmed me that they see my packets coming in with the nated source IP.

My questions:

How do we view the nat logs on Checkpoint Firewalls ?

Why can't I see any firewall logs on VS2 ? (We are using the same accept rule on both VS1 and 2 through the use of a layer; and yes, the rule is set to log traffic.)

To be clear, the issue is most probably located on the client side but I'd like to take this oppportunity to learn more about Checkpoint's GW.

Thanks for your help.

Fabien

Knut99

Hello,

I am a bit embarrassed to report here the reason for this logging issue...

It appears that an "inter-VS" rule I didn't know about, exists in both policies. IT allows traffic from "the other VS" without logging it.

So origin traffic was allowed (and logged) in VS1 by the "correct" rule in the layer, but was allowed (and not logged) by this second rule on VS2.

Apparently it was done in order to save some disk space on the logging node.

We are going to need more disk space to us allow to log everything ... 🙂

Sorry for wasting your time!

View solution in original post

PhoneBoy

The very same log that shows the Accept action should also show the NAT that was done (if any) in the full log card.
Simply click on the relevant log line in SmartConsole or SmartView.

Did you do a tcpdump from VS2 to verify traffic is being received?

Knut99

Hello and thank you @PhoneBoy and @the_rock for your help.

VS1 log doesn't show any NATing as the NAT happen on the VS2 side. So that is expected.

However I don't understand why I have no logs from VS2.

Since I started this thread, I got the confirmation that the connectivity issue was indeed on the client side (server misconfiguration). Now connectivity is restored but the logging behavior on my GWs is still the same : no logs from VS2 while traffic is flowing through it.

I've attached a screenshot of the log view in the smart console when I was successfully contacting the remote server (10.204.22.128) with this simple curl command:

curl -vvv http://10.204.22.128
* Trying 10.204.22.128:80...
* Connected to 10.204.22.128 (10.204.22.128) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.204.22.128
> User-Agent: curl/8.0.1
> Accept: */*

As you can see on the screenshot only VS1 is logging traffic.

The access rule on both VS is the same as it is the same layer that is deployed on both GW's policies.

I've attached a quick graph I made of the network physical topology to make things easier to understand.

I hope all of that makes sens.

Cheers,

Fabien

the_rock

Forgive me if this statement will sound dumb (apologies), as I have not dealt with VSX in forever, so not sure if these commands work on VS level, but if they do, it would be good to compare.

See if you can run below on both VS1 and VS2 level and if if it works, what it actually "spits out".

Andy

example from my lab (regular fw R81.20 jumbo 65)

[Expert@CP-gw:0]# cd $FWDIR/log
[Expert@CP-gw:0]# ls -lh fw.log
-rw-rw---- 1 admin root 8.2K Jun 10 00:00 fw.log
[Expert@CP-gw:0]#

Output should ALWAYS show size 8.2K, which means its NOT logging locally, but rather sending to mgmt server. Any other value would indicate the issue.

PhoneBoy

Before doing any further troubleshooting, you need to see if the traffic actually being received on VS2 first.
The only way to do that is fw monitor/tcpdump or possibly fw ctl zdebug.

the_rock

Super valid point, for sure. I assumed based on all @Knut99 that was the case, but better be 100% sure.

Andy

Knut99

Hi @PhoneBoy and @the_rock

Thanks again for your help!

Here is the ouput of fw monitor on the first VSX cluster member (VS1 is currently running on this node):

VS id 4 correspond to VS1 here

[Expert@FWVSXHA1:0]# fw monitor -co 20 -F "10.32.10.247,0,10.204.22.128,0,0" -F "10.204.22.128,0,10.32.10.247,0,0"
[...]
[vs_4][ppak_0] bond50.860:i[44]: 10.32.10.247 -> 10.204.22.128 (TCP) len=52 id=35474
TCP: 52503 -> 443 .S.... seq=cc817471 ack=00000000
[vs_4][fw_0] bond50.860:i[44]: 10.32.10.247 -> 10.204.22.128 (TCP) len=52 id=35474
TCP: 52503 -> 443 .S.... seq=cc817471 ack=00000000
[vs_4][ppak_0] bond50.860:i[44]: 10.32.10.247 -> 10.204.22.128 (TCP) len=52 id=35474
TCP: 52503 -> 443 .S.... seq=cc817471 ack=00000000
[vs_4][ppak_0] bond50.860:I[44]: 10.32.10.247 -> 10.204.22.128 (TCP) len=52 id=35474
TCP: 52503 -> 443 .S.... seq=cc817471 ack=00000000
[vs_4][ppak_0] bond20.810:o[44]: 10.32.10.247 -> 10.204.22.128 (TCP) len=52 id=35474
TCP: 52503 -> 443 .S.... seq=cc817471 ack=00000000
[vs_4][ppak_0] bond20.810:O[44]: 10.32.10.247 -> 10.204.22.128 (TCP) len=52 id=35474
TCP: 52503 -> 443 .S.... seq=cc817471 ack=00000000

And here is the ouput of fw monitor on the second VSX cluster member (VS2 is currently running on this node):

VS id 5 correspond to VS2 here

[Expert@FWVSXHA2:0]# fw monitor -co 20 -F "10.32.10.247,0,10.204.22.128,0,0" -F "10.204.22.128,0,10.32.10.247,0,0"
[...]
[vs_5][ppak_0] wrp320:i[44]: 10.32.10.247 -> 10.204.22.128 (ICMP) len=60 id=35461
ICMP: type=8 code=0 echo request id=14691 seq=17873
[vs_5][ppak_0] wrp320:i[44]: 10.32.10.247 -> 10.204.22.128 (ICMP) len=60 id=35462
ICMP: type=8 code=0 echo request id=14691 seq=17874
[vs_5][ppak_0] wrp320:i[44]: 10.32.10.247 -> 10.204.22.128 (ICMP) len=60 id=35463
ICMP: type=8 code=0 echo request id=14691 seq=17875
[vs_5][ppak_0] wrp320:i[44]: 10.32.10.247 -> 10.204.22.128 (ICMP) len=60 id=35464
ICMP: type=8 code=0 echo request id=14691 seq=17876
[vs_5][ppak_0] wrp320:i[44]: 10.32.10.247 -> 10.204.22.128 (ICMP) len=60 id=35465
ICMP: type=8 code=0 echo request id=14691 seq=17877
[vs_5][ppak_0] wrp320:i[44]: 10.32.10.247 -> 10.204.22.128 (TCP) len=52 id=35469
TCP: 33874 -> 443 .S.... seq=adef9269 ack=00000000
[vs_5][ppak_0] wrp320:i[44]: 10.32.10.247 -> 10.204.22.128 (TCP) len=52 id=35469
TCP: 33874 -> 443 .S.... seq=adef9269 ack=00000000
[vs_5][ppak_0] wrp320:I[44]: 10.32.10.247 -> 10.204.22.128 (TCP) len=52 id=35469
TCP: 33874 -> 443 .S.... seq=adef9269 ack=00000000
[vs_5][ppak_0] bond40.510:o[44]: 10.32.10.247 -> 10.204.22.128 (TCP) len=52 id=35469
TCP: 33874 -> 443 .S.... seq=adef9269 ack=00000000
[vs_5][ppak_0] bond40.510:O[44]: 10.16.224.1 -> 10.204.22.128 (TCP) len=52 id=35469
TCP: 13226 -> 443 .S.... seq=adef9269 ack=00000000

So traffic is flowing through VS1 and VS2 and source nating is working.

However from the smart console I still can only see logs from VS1. See attached screenshot form the smartconsole.

the_rock

Hey @Knut99

Were you able to check what I mentioned Monday? As I stated, its possible this does NOT apply to VS itself, but worth double checking.

Andy

Knut99

Hey @the_rock

I believe this command is relevant per VSX cluster member, not per VS.

Here are te results for both cluster member:

VS1

[Expert@FWVSXHA1:0]# cd $FWDIR/log
[Expert@FWVSXHA1:0]# ls -lh fw.log
-rw-rw---- 1 admin root 8.2K Mar 25 00:00 fw.log

VS2

[Expert@FWVSXHA2:0]# cd $FWDIR/logs
[Expert@FWVSXHA2:0]# ls -lh fw.log
-rw-rw---- 1 admin root 8.2K Mar 24 00:00 fw.log

Thnaks for helping me out!

the_rock

Ok, thats perfect, tells us 100% its configured to log to mgmt, since file is NOT growing locally.

Sorry if I missed this before, but did logging ever work right for this vs?

Andy

Knut99

From what I can see yes, logging is working for this VS, at least I can see traffic from this VS in the smart console if I don't add my filters (src and dst ips).

the_rock

Are you allowed to do remote? If yes, be free to message me directly and I can send you zoom invite.

Just working on bunch of labs today, so I got time.

Btw, I know this might be a long shot, but see if you can open an "old school" SV tracker as per below and see what happens.

Andy

Apologies, had biking accident, so sorry for any spelling mistakes lol

Knut99

Just tried with Tracker (didn't know this tool), but it shows the same logs than the smart console.

I may have time to reach out to you later in the day. Thanks for the suggestion.

the_rock

Yes, no problem. Its how people would check logs before R80, via smart view tracker. Do reach out, I can do any time between 1-4 pm EST (GMT-4)

Knut99

Hello @the_rock

Sorry I couldn't find the time to reach out last time.

Would you be available for a remote session sometime this week?

Thanks,

Fabien

the_rock

Sure, let me know.

the_rock

Hey @Knut99

No need to apologize mate, we are always here to help each other, so you came to the right place. I agree with Phoneboy, probably if you do basic capture, it would help us.

So, I will give you some examples.

Say client IP is 10.10.10.10, you can run this from expert ->

tvpdump -enni any host 10.10.10.10 and port 443 (assuming thats the port)

You can also do fw monitor as well, like below

fw monitor -e "accept host(10.10.10.10) and port(443);"

Now, say src behind the fw is 192.168.2.2 trying to reach the client, you can do this:

fw monitor -F "192.168.2.2,0,10.10.10.10,443,0" -F "10.10.10.10,0,192.168.2.2,443,)

Idea is this "srcip,srcport,dstip,dstport,protocol"

So really what matters is dst port, src port does not matter, protocol can be any (or 0 that is)

Btw, you can also refer to this site, super useful, that my colleague made ages ago.

Best,

Andy

https://tcpdump101.com/#

Knut99

Hello,

I am a bit embarrassed to report here the reason for this logging issue...

It appears that an "inter-VS" rule I didn't know about, exists in both policies. IT allows traffic from "the other VS" without logging it.

So origin traffic was allowed (and logged) in VS1 by the "correct" rule in the layer, but was allowed (and not logged) by this second rule on VS2.

Apparently it was done in order to save some disk space on the logging node.

We are going to need more disk space to us allow to log everything ... 🙂

Sorry for wasting your time!

the_rock

Dont feel bad man, it happens, glad it solved!

Andy

Are you a member of CheckMates?

SG logging behavior