Constant monitoring with 'zdebug drop'

[Matlu]

Hi, Mates.

A query, the command

#fw ctl zdebug drop | grep 'Destination IP'

can be left 'running' in the background in order to 'monitor' a particular IP and know if during the day, the command 'registers' something relevant?

We do this in order to capture traffic at the exact moment, because unfortunately when we apply the command in real time everything works fine, but just when we stop testing, the connection between 1 source and 1 destination begins to fail and we have no records of those precise moments.

Is this possible?

Greetings.

[the_rock]

I let it run in my lab before for a week, never had a problem.

Andy

[Matlu]

Hey, Buddy.

I think my query was not well explained.

I want to leave ‘running’ the zdebug drop for example x 1 week.
So, if for example I connect via ssh and applied the command, it is simply not to interrupt the command with Ctrl+C?

If I close the terminal (Putty, SecureCRT, etc), I can be sure that the command is still running?

[the_rock]

Thats exactly what I did.

[Matlu]

Is it possible to send the output of the command #fw ctl zdebug drop | grep -IP DESTINATION- to a particular ‘path’ of a vsenv (For example in /var/tmp/) and tell it to save the result in a file ending in .txt extension?

[the_rock]

Yup...example

fw ctl zdebug + drop | grep 1.1.1.1 > /var/log/zdebug.txt

[Matlu]

Hi,
The #fw ctl zdebug drop
is 100% reliable for troubleshooting?
The messages that appear here for a particular IP that you are having problems with, for example related to TCP RETRANSMISSIONS, is it reliable to rely on this command or are there other testing options?

[the_rock]

Personally, I would say its reliable, yes, but you would need to use it along with other things, such as tcpdump, logs, fw monitor, etc

Andy

[Matlu]

Hi,
2 inquiries, please.

- To capture traffic in Troubleshooting with the FW Monitor is it ‘indispensable’ to turn off the SXL?

- In a VSX environment, can I capture traffic with the FW Monitor and export the output to a pcap file, in a particular vsenv?

Thanks for the replies.

[the_rock]

1) You can, but most people dont, unless its really complicated issue or involves kernel debug (in my experience)

2) Its been some timeI dealt with VSX, but I recall you can do so in VS itself

example -> vsenv 1 -> fw monitor -e "accept host(1.2.3.4) and port(444);" -o /var/log/testfwmonitor.out

Andy

[Matlu]

I want to capture TCP traffic with #FW Monitor and #fw ctl zdebug....
In an environment of a TCP communication between 1 source and 1 destination on a particular port, as we are currently reporting a problem with TCP retransmissions.

So I wanted to be sure if it is necessary to turn off the SXL, because to be honest, many times one forgets to 'turn it on' again. 😏😅

[the_rock]

One time when I was helping a lady with some CP upgrades and they would always turn off sxl and she says "Dont worry Andy, I always remember to turn it back on after the upgrade", there comes 6 am, I say you forgot something, she goes "What, should we go for a breakfast?"...Im thinking, YES, PLUS turn back on sxl haha 🙂

No shame in writting it on big piece of paper or whiteboard, all good!

Andy

[Timothy_Hall]

There are very few situations where you must use fw monitor -e (which can only capture traffic in F2F/slowpath) on a modern Check Point gateway.  If all you need to see is whether traffic was received and then whether it was transmitted (basically just capture points i and O), use cppcap.  If you need to see all 4 capture points iIoO (say for detailed NAT troubleshooting, or to see if Gaia is dropping something instead of the Check Point code), use fw monitor -F.  Both these tools will give you a complete capture regardless of the state of SecureXL.  SecureXL should really not be disabled for any reason via fwaccel off on a modern Check Point gateway.

If you attempt to run fw ctl zdebug + drop and fw monitor -F simulteneously, be aware that you need to run them in the proper order to keep them from interfering with each other.  See here: Debug Filter Battle -- fw monitor -F vs. fw ctl zdebug + drop

[the_rock]

On another note @Matlu , what is the EXACT issue you have here? You said lots of tcp retransmissions?

Andy

[Matlu]

It is a problem related to “TCP Retransmissions”.

I have a communication between 1 source and 1 destination going through an S2S VPN.

Everything works fine, but for certain very “intermittent” periods of time, the source that is at the other end (not mine), “complains” of a “slow and intermittent” experience in its application.

The connectivity between the source and the destination is through a specific port (Port: 7003).

When these negative experiences occur, the remote end of the VPN, makes captures on their side, and what they observe are constant retransmissions in the connectivity, but on our side the #fw ctl zdebug and #tcpdump commands (which are the ones we have tested so far), do not show us anything relevant.

Apparently for our CP, everything “flows” normally, without interruptions.

That's why my questions were focused on diagnostic commands that can help me to rule out errors in the CP.

Is it more useful to rely on CPPCAP than TCPDUMP, if what I want is to “analyze” the traffic from a pcap file to see it in Wireshark?

[the_rock]

Just be CAREFUL in production environment, maybe dont leave it running for too too long (example, say for 24 or 48 hours). Just curious though, is it affecting traffic extensively or happens just randomly?

Andy

[Bob_Zimmerman]

Note that filtering it with grep potentially presents a lot of load to the firewall. It's going through the extra debug code for every single drop, then you're discarding 99.9% of the drops with the grep.

Also, the output is buffered, so you won't see output into the file immediately. Instead, a chunk of output will accumulate for a while, then once enough is collected, it will all be handed to grep at once. On a firewall which isn't dropping much, this could result in 10+ drops all showing up at once.

I would do this like so:

fw ctl debug 0
fw ctl debug -buf 32000
fw ctl debug -m fw + drop
fw ctl debug -F "0,0,0,0,0" -F "0,0,0,0,0" ...
# Replace 0,0,0,0,0 with a filter, like used for fw monitor.
# Be sure to filter for both directions in case a reply is dropped.
nohup fw ctl kdebug -T -f 2>&1 >/var/log/<ticket number>_drop.txt &

The '-T' adds timestamps to the debug output.

Might take a little trial and error. The -F filter may need to be applied along with one of the commands. nohup allows the process it runs to survive a SIGHUP, which is the signal a closing shell sends to processes it is running. Normal output will go to /var/log/<ticket number>_drop.txt, and errors will go to nohup.out in the directory where you ran the command.

The ticket number is important so you can tell what exactly that file is for when you look at the system again in six months. To kill the command once you have the data you want, you would use:

ps -ef | grep "kdebug" | grep -v "grep" | awk '{print $2}' | xargs -L 1 kill;fw ctl debug 0

As for how to watch this and send an email when a drop is added, I would have to think about it for a while.

[Timothy_Hall]

For long-term monitoring with fw ctl zdebug + drop (the "+" ensures you see drops by SecureXL which are rather rare) I wouldn't recommend piping the full output to a grep like that.  While the overwhelming majority of packets are accepted and don't show up in the zdebug output, the commencement of a DoS attack or even port scan will cause the amount of output to increase rapidly and potentially impact the performance (or worse) of the firewall.

What I would recommend is something like this in one window:

fw ctl zdebug + drop > output.txt 

Then after a few seconds (long enough for the zdebug to fully initialize) do this in another window:

fw ctl set str simple_debug_filter_daddr_1 DESTINATION_IP

Then after hitting CNTRL+C to stop the zdebug in the first window, now run this command in the second window just in case:

fw ctl debug 0 (to ensure your filter has been unset, the zdebug being stopped should do this automatically though)

The second command will filter the output inside the debug itself and substantially reduce the chance of a DoS flooding the output.  Unfortunately you can't run that fw ctl set command before starting the zdebug as it will just get reset by the zdebug when it starts (and usually when it ends).

If anyone knows how to add/set simple_debug_filter_daddr_1 to the zdebug as part of invoking the zdebug in a single command I'd be all ears.

[Bob_Zimmerman]

For a zdebug in particular, it's pretty simple to add the simple filters. I demonstrate how to do so (and how to add multiple flags) here:

fw ctl zdebug -T -F "10.0.1.253,0,192.168.144.120,0,0" -m fw xlate drop

[Matlu]

Hello,

How would you apply the filter, if you have a connection with the following data:

SRC IP: 10.150.1.8
DST IP: 10.252.30.195
DST PORT: 7003

The objective would be that the zdebug tells me if the CP is really “dumping” some kind of traffic.

[Bob_Zimmerman]

Same as you would write it for fw monitor:

-F "10.150.1.8,0,10.252.30.195,7003,0" \
-F "10.252.30.195,7003,10.150.1.8,0,0"

I would probably leave the port off, and might even separate them like this to catch all drops to or from either address on any port:

-F "10.150.1.8,0,0,0,0" \
-F "0,0,10.150.1.8,0,0" \
-F "10.252.30.195,0,0,0,0" \
-F "0,0,10.252.30.195,0,0"

Capture filters should generally be broad so you can catch things which you don't expect.

[Matlu]

Hello.

Would it look like this?

fw ctl zdebug -T -F ”10.150.1.8,0,10.252.30.195,7003,0” \ -F “10.252.30.195,7003,10.150.1.8,0,0,0” -m fw xlate drop

Should the final part of the command “-m fw xlate drop” .... be “respected” or can that part be “omitted”?

Thanks.

[Bob_Zimmerman]

The backslash is only needed to spread a command out over multiple lines. I use it to keep things readable.

# This command:

fw ctl zdebug -T -F "10.150.1.8,0,10.252.30.195,7003,0" -F "10.252.30.195,7003,10.150.1.8,0,0" -m fw xlate drop

# is the same as this command:

fw ctl zdebug -T \
-F "10.150.1.8,0,10.252.30.195,7003,0" \
-F "10.252.30.195,7003,10.150.1.8,0,0" \
-m fw xlate drop

As you can see, the lower form with the backslashes is a lot simpler. Smaller parts to understand. I even noticed a typo in one of your filter expressions (an extra ",0" at the end) because the two expressions were not the same length.

The '-m ...' part is telling it what modules and flags you want debug information from. If you don't want to debug NAT, you don't need xlate. It sounds like you may need a + in there after the 'fw'.

I would really run the debug like I specified in my first message, with nohup.