Who rated this post

I ended up writing a small script to handle stuff like captures on a bunch of interfaces. It can even leave them running after you log off:

### To start:
ticket="6-1234567890"
filter="(host 192.0.2.38 or host 203.0.113.54) and (host 1.1.1.1 or host 8.8.8.8 or host 4.2.2.2)"

dateString="$(date --rfc-3339=seconds | tr ' ' 'T' | tr -d ':')"
allAddresses="$(<<<"${filter}" egrep -o "([0-9]{1,3}\.){3}[0-9]{1,3}")"
allInterfaces="$(<<<"${allAddresses}" xargs -L 1 ip route get | grep dev | awk '{print $(NF-2)}')"
for interface in $(<<<"${allInterfaces}" sort | uniq);do
filename="/var/log/${ticket}_${dateString}_$(hostname)_${interface}.pcap"
nohup tcpdump -i "${interface}" "${filter}" -s 128 -C 100 -W 3 -w "${filename}" &
done


### To kill:
ps -ef | grep "6-1234567890" | grep -v grep | awk '{print $2}' | xargs -L 1 kill

You need to update the ticket number and filter in the "To start" section, and the ticket in the "To kill" section. The ticket can be whatever you want. I use a Check Point ticket here, but in reality, I use one of my company's internal incident tickets almost every time.

You write the filter one time, then it figures out all the interfaces which lead to addresses in the filter, and starts the capture on all of them. As written, the captures are limited to 200-300 MB per interface (a rolling set of three 100 MB captures), and they go into /var/log named like ${ticket}_${dateString}_$(hostname)_${interface}.pcap0. When running captures on many firewalls (how many issues only involve one), I typically generate the date string one time, then copy that exact date to the other firewalls instead of calculating a new date string. The names are unique enough I can collect all the files afterwards and toss them in one directory for the ticket without worry of overlap, and if I find a capture some time later, I can look into the ticket to see if it still matters.

I write my filters very narrowly. I put all the possible versions of an address (e.g, before NAT and after NAT) in one parenthetical expression, then 'and' them together in the combinations to describe each flow I want to catch. Keep the filters narrow, and the captures present minimal additional load to the system.

I originally wrote this to catch an issue which was hard to predict, but easy to detect. We left captures running for several days, and eventually caught it.