Using Wireshark to capture packets on Firewall using standard out.

[KlowikiOne]

Wireshark Live Capture From Check Point Firewall directly to your PC

   Note: If a Jump box is required in your environment you need to run command from the jump box.

Standard Warning applies, Do not run a capture of any kind in this manner on an loaded firewall. No CPU over 70% should be safe however after-hours is best. Theoretically this type of capture will reduce CPU utilization by relieving the need to write a file. No download needed. The capture will automatically close when the capture is stopped.  Ensure you have the bandwidth available to copy traffic. Filters also work in the command line.

ssh admin@192.168.199.22 "tcpdump -s0 -w - -i eth0" | "c:/program files\wireshark\wireshark.exe" -k -i -

I tried many different commands and command shells before I got the traffic redirects to work.

Cygwin  did not work

Mobaxterm did not work

Openssh in Cygwin did not work

Almost every search in google for using stdout to push firewall traffic to the PC for Wireshark to use stdin to see live traffic in real time did not work. I found no references to use the good old command window and openssh.

Openssh as the command line that will open an ssh session over port 22 to the firewall. Tcpdump to redirect the filtered traffic over the tcp encrypted ssh tunnel session on port 22 back to the workstation in raw format. Wireshark to receive the stdout via stdin and display to the screen in real time.  Amazing stuff.

[PhoneBoy]

This is quite clever and well done on figuring it out.
You mind if I move this to a more public space?

[KlowikiOne]

I do not mind.  Time for public critique.  🙂

[PhoneBoy]

Done, moved out to Security Gateways.

[Danny]

Thanks for this!

[the_rock]

Good tip 👍

[Timothy_Hall]

Yep a neat trick and well done!  Here is how I got to the same place awhile back through a different means but I found a number of caveats (the "timeout" one is more precautionary than anything as I saw some stuck tcpdump processes); here is the relevant lecture content along with the lab exercise from my Max Capture: Know Your Packets self-guided video:

[KlowikiOne]

I added a ram disk as a destination for debugs and realized a 60% reduction in CPU utilization on SND cores compared to the same debug writing to disk. This was in a production environment at 3am in the morning but still worked well. Might be worth further testing for packet captures. 

Process Example is using the ram disk as a destination for tcpdump.

Process

#

#/*Make sure there is enough space. Memory swap does still work to disk if you go over the amount available. Ramfs does not support swap. */

#

#free -g

#

              total        used        free      shared  buff/cache   available

Mem:             15           8           0           1           6           5

Swap:            31           1          29

#

#/*  Create the location the new disk will mount.  */

#mkdir /mnt/rmdisk

#

#/*  Mount the ram as tmpfs and specify amount of ram to use and where to mount it. */

#

#mount -t tmpfs -o size=4g tmpfs /mnt/rmdisk

#

#/* You can cd into the directory and use it as a normal disk. Remember this is volatile memory and will be erased on reboot or dismount.  For Example: tcpdump. */

#

#tcpdump -enni any -w /mnt/rmdisk/dumpme.pcap

I am also investigating using tar files for tcpdump destinations for reducing CPU utilization. The whole Idea is to minimize the hardware interrupt for DMA for transfers from RAM to Disk. The process is slower than compressing data via cpu and a tar file. Sounded like a good idea anyway.

Thank you everyone for your replies and kudos. 

[Timothy_Hall]

One other way to reduce CPU load I was investigating for live Wireshark captures was using cppcap instead of tcpdump for these live captures, as cppcap doesn't seem to cause nearly as much overhead as tcpdump which is the tool's main intent.  However I confirmed with the tool's author @Aviad_Hadarian that cppcap does not have the option to output raw packets to the stdout, but if it did I think that would be very helpful to reduce live capture overhead.  Might it be possible for cppcap to write its raw packet output to a named pipe (FIFO) via -o instead of a regular file, then have some intermediate process such as dd read that named pipe, dump it to its stdout and directly into Wireshark's stdin?

Also apparently having stuck tcpdump processes executing in the background running up CPU load on the Dispatcher cores has occurred often enough to cause the following recent SK to be created: sk172886: Multiple ksoftirqd processes show high CPU use

[Bob_Zimmerman]

Some OpenSSH options can help this work through jump servers in the path. Specifically, check out the ProxyJump option in ssh_config(5). It uses port forwarding on a jump server to let the client connect "directly" to the end destination. Notably, this doesn't expose any key material to the jump server in the path (unlike SSH agent forwarding).

A few other options can get similar results.

[KlowikiOne]

Do you know of a way to start multiple tcpdumps in a single session? I would love to see checkpoint permit "screen" to operate.  I would also like to see and updated ethtool. 

[the_rock]

@KlowikiOne Well, since you are a CP employee, your best bet for those things is to verify with R&D. Personally, I never heard of a way to start multiple tcpdumps in a single session.

[Timothy_Hall]

If you need to capture on all interfaces with tcpdump, just use an interface name of "any" but be careful as that can impact the performance of the firewall.  You could also use cppcap with the "-I (interface)" option that will capture traffic on all interfaces except the named (interface) as opposed to using "-i" for the interface specification.  ethtool was updated in Gaia 3.10.  All this was covered in my 'Max Capture: Know Your Packets" course.

[KlowikiOne]

I avoid using the any interface, however I do like cppcap option to capture on any interface except flag. 

[the_rock]

Thats good point, BUT, sometimes, if you dont know what interface to use, you have to use any...

[KlowikiOne]

#ip route get <dest.ip>

Output will give you interface name to use in tcpdump. 

[the_rock]

That is true, good point : - )

Andy

[the_rock]

As seen from lab example:

[Expert@CP-GW:0]# ip r g 8.8.8.8
8.8.8.8 via 172.16.10.1 dev eth0 src 172.16.10.249
cache
[Expert@CP-GW:0]# tcpdump -i eth0 host 8.8.8.8
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:31:48.563712 IP CP-GW.57316 > dns.google.domain: 14292+ A? www.8983bd2f5e976cffe087-a8239a8de08e774770d9a0a128aff634.ssl.cf1.rackcdn.com. (95)
15:31:48.563938 IP CP-GW.57316 > dns.google.domain: 19780+ A? www.sigcheck.checkpoint.com. (45)
15:31:48.564119 IP CP-GW.57316 > dns.google.domain: 64267+ A? www.whitelist-cidr.portal.checkpoint.com. (58)
15:31:48.564269 IP CP-GW.57316 > dns.google.domain: 10469+ A? www.dl.dropboxusercontent.com. (47)
15:31:48.564429 IP CP-GW.57316 > dns.google.domain: 588+ A? www.office.live.com. (37)

[the_rock]

Check out site my colleague made while back on capturing with all major vendors, hope it helps you.

Andy

https://tcpdump101.com/#

[KlowikiOne]

I actually use this site to formulate the command line to use with tcpdump and cppcap.  I find having customers use this is a helpful learning tool.  

[the_rock]

My colleague made that site in order to actually help anyone struggling with packet captures and debugs...we all been there mate, whether we want to admit ot ir not 😂

[Bob_Zimmerman]

There's not a good way to. Multiple SSH sessions are basically free, though (no more overhead than a single connection carrying all the same data).

[KlowikiOne]

I am going to try to use cprid_util to spawn a tcpdump with a flag to close out the capture.  

[Bob_Zimmerman]

I ended up writing a small script to handle stuff like captures on a bunch of interfaces. It can even leave them running after you log off:

### To start:
ticket="6-1234567890"
filter="(host 192.0.2.38 or host 203.0.113.54) and (host 1.1.1.1 or host 8.8.8.8 or host 4.2.2.2)"

dateString="$(date --rfc-3339=seconds | tr ' ' 'T' | tr -d ':')"
allAddresses="$(<<<"${filter}" egrep -o "([0-9]{1,3}\.){3}[0-9]{1,3}")"
allInterfaces="$(<<<"${allAddresses}" xargs -L 1 ip route get | grep dev | awk '{print $(NF-2)}')"
for interface in $(<<<"${allInterfaces}" sort | uniq);do
filename="/var/log/${ticket}_${dateString}_$(hostname)_${interface}.pcap"
nohup tcpdump -i "${interface}" "${filter}" -s 128 -C 100 -W 3 -w "${filename}" &
done


### To kill:
ps -ef | grep "6-1234567890" | grep -v grep | awk '{print $2}' | xargs -L 1 kill

You need to update the ticket number and filter in the "To start" section, and the ticket in the "To kill" section. The ticket can be whatever you want. I use a Check Point ticket here, but in reality, I use one of my company's internal incident tickets almost every time.

You write the filter one time, then it figures out all the interfaces which lead to addresses in the filter, and starts the capture on all of them. As written, the captures are limited to 200-300 MB per interface (a rolling set of three 100 MB captures), and they go into /var/log named like ${ticket}_${dateString}_$(hostname)_${interface}.pcap0. When running captures on many firewalls (how many issues only involve one), I typically generate the date string one time, then copy that exact date to the other firewalls instead of calculating a new date string. The names are unique enough I can collect all the files afterwards and toss them in one directory for the ticket without worry of overlap, and if I find a capture some time later, I can look into the ticket to see if it still matters.

I write my filters very narrowly. I put all the possible versions of an address (e.g, before NAT and after NAT) in one parenthetical expression, then 'and' them together in the combinations to describe each flow I want to catch. Keep the filters narrow, and the captures present minimal additional load to the system.

I originally wrote this to catch an issue which was hard to predict, but easy to detect. We left captures running for several days, and eventually caught it.

[the_rock]

Thats awesome!

[KlowikiOne]

You are amazing at writing scripts.  Genius level.  

I will adopt what you have shared into many of my scripts.  I will also add credits and site links within what is shared so others will know where the additions originate.  Thank you for sharing!

[Bob_Zimmerman]

So the cool trick with this one is that it takes the filter and extracts everything which looks like an IP address, uses 'ip route get' to get the interface leading to each, then dedups the list and runs the capture on each unique interface. This means it can catch complex traffic flows like client on interface A to load balancer VIP on interface B, then load balancer source NAT on B to backing servers on interface C all in one capture with accurate timing information.

I recently used it to track a connection through three firewalls, three sets of load balancers, and five total layers of NAT. Write the filter once, use it everywhere.

[HeikoAnkenbrand]

Very nice solution!

If you add "not port 22", you will no longer see your own ssh packages.

ssh admin@1.1.1.1 "tcpdump -s0 -w - -i eth0 not port 22" | "c:/program files\wireshark\wireshark.exe" -k -i -