NAT table (fwx_alloc) top users

[Kaspars_Zibarts]

Certainly this has been done before but I somehow didn't find any good reference to get expanded summary of NAT statistics, like top NAT IPs that are being used or top destinations. So whilst troubleshooting my hide NAT failures with O365 I wrote this little bash script to show:

• top NAT IPs used on GW
• top destinations in NAT table
• top client (original src) IPs in the NAT table
• top NAT pools where pool = 3-tuple "protocol + NAT IP + dest IP" as in SK156852

Here's the screenshot, possible issues will be highlighted in red (normally exceeding 40000, but you can change it as you wish)

And the code:

#!/bin/bash

# Script to interpret fwx_alloc table top users
# Only interpreting rows that start with TCP or UDP <00000006 or <00000011
# For VSX set to correct environment manually
# NAT pool does not take into considertaion dst port as per SK156852

topcount=4  # Set how many top users to display
redthreshold=40000  # Highlight to show high usage
RED='\033[0;31m'
GRN='\033[0;32m'
CYN='\033[0;36m'
NC='\033[0m' # No Color

fw tab -t fwx_alloc -u > nat_table.raw

# Get top NAT IPs
echo -e "${GRN}"
echo -e "==== TOP NAT IP ADDRESSES ====${NC}"
echo -e "------------------------------"
cat nat_table.raw | sed 's/[><,;]//g' | egrep "^00000006|^00000011" | awk '{print $2}' | sort | uniq -c | sort -r | head -$topcount | while read line; do

  count=`echo "$line" | awk '{print $1}'`
  if [ $count -gt $redthreshold ]; then count="${RED}${count}${NC}"; fi
  ipaddr=`printf '%d.%d.%d.%d\n' $(echo $line | awk '{print $2}' | sed 's/../0x& /g')`
  while [ ${#ipaddr} -lt 20 ]; do ipaddr="$ipaddr "; done
  echo -e "  $ipaddr $count"
done
echo

# Get top destination IPs
echo -e "${GRN}"
echo -e "==== TOP DST IP ADDRESSES ====${NC}"
echo -e "------------------------------"
cat nat_table.raw | sed 's/[><,;]//g' | egrep "^00000006|^00000011" | awk '{print $4}' | sort | uniq -c | sort -r | head -$topcount | while read line; do

  count=`echo "$line" | awk '{print $1}'`
  ipaddr=`printf '%d.%d.%d.%d\n' $(echo $line | awk '{print $2}' | sed 's/../0x& /g')`
  while [ ${#ipaddr} -lt 20 ]; do ipaddr="$ipaddr "; done
  echo "  $ipaddr $count"
done
echo

# Get top destination IPs
echo -e "${GRN}"
echo -e "==== TOP SRC IP ADDRESSES ====${NC}"
echo -e "------------------------------"
cat nat_table.raw | sed 's/[><,;]//g' | egrep "^00000006|^00000011" | awk '{print $5}' | sort | uniq -c | sort -r | head -$topcount | while read line; do

  count=`echo "$line" | awk '{print $1}'`
  ipaddr=`printf '%d.%d.%d.%d\n' $(echo $line | awk '{print $2}' | sed 's/../0x& /g')`
  while [ ${#ipaddr} -lt 20 ]; do ipaddr="$ipaddr "; done
  echo "  $ipaddr $count"
done
echo


# Get top NAT pools as per sk156852
echo -e "${GRN}"
echo -e "=== TOP NAT POOLS SK156852 ===${NC}"
echo -e "------------------------------"
echo
cat nat_table.raw | sed 's/[><,;]//g' | egrep "^00000006|^00000011" | awk '{print $1" "$2" "$4}' | sort -k1 -k2 -k3 | uniq -c |  sort -r | head -$topcount | while read line; do

  count=`echo "$line" | awk '{print $1}'`
  if [ $count -gt $redthreshold ]; then count="${RED}${count}"; fi
  
  proto="TCP"
  if [ `echo $line | awk '{print $2}' | grep -c "00000006"` -eq 0 ]; then proto="UDP"; fi
  
  
  natIPhex=`echo $line | awk '{print $3}'`
  dstIPhex=`echo $line | awk '{print $4}'`
  natIP=`printf '%d.%d.%d.%d\n' $(echo $line | awk '{print $3}' | sed 's/../0x& /g')`
  dstIP=`printf '%d.%d.%d.%d\n' $(echo $line | awk '{print $4}' | sed 's/../0x& /g')`
  while [ ${#natIP} -lt 15 ]; do natIP="$natIP "; done
  while [ ${#dstIP} -lt 15 ]; do dstIP="$dstIP "; done
  
  echo -e "${CYN}  $proto $natIP > $dstIP  Total: $count${NC}"
  echo "  --------------------------------------------------"
  
  cat nat_table.raw | sed 's/[,;<>]//g' | egrep "^00000006|^00000011" | grep "$natIPhex" | grep "$dstIPhex" | awk '{print $5}' | sort | uniq -c | sort -r | head -$topcount | while read line2; do
    count=`echo "$line2" | awk '{print $1}'`
    ipaddr=`printf '%d.%d.%d.%d\n' $(echo $line2 | awk '{print $2}' | sed 's/../0x& /g')`
    while [ ${#ipaddr} -lt 20 ]; do ipaddr="$ipaddr "; done
    echo "    $ipaddr $count"    
  done
  echo
  
done

[RickHoppe]

Great script! This script saves a lot of time. No more manually interpreting and converting the fwx_alloc table 🙂

BTW, when looking at SK156852 I noticed that NAT statistics are now added to CPView in R80.40.

[Kaspars_Zibarts]

thanks! Indeed - I was stuck with R80.30... so had n choice 🙂

[pjoseph]

This is an excellent script. Is there a version that works for R81.20? I believe fwx_alloc table no longer exists post R80.30.

[Kaspars_Zibarts]

Hi! The short answer is that NAT stats are available in CPView (see SK156852) and tables changed after R80.40 🙂 so I didn't bother updating script

Where did the fwx_alloc table go in R80.40 and above?

In R80.40 we introduced GNAT, which does not use the fwx_alloc table for allocating ports anymore. Instead, we use a global table (fwx_alloc_global) that is shared between all instances

[pjoseph]

Yes, I was aware of the tab in CPView. However, it only shows the top 2 NAT pools. We recently started hitting the NAT port exhaustion issue so we wanted a way to proactively monitor atleast the top 4 NAT pools which your script does beautifully.

[Kaspars_Zibarts]

Here's quick fix for versions R80.40 and higher. I'm afraid I don't have a luxury of large environments to test it properly, but I believe it should work 

#!/bin/bash

# Script to interpret fwx_alloc_global (R80.40+) table top users
# Only interpreting rows that start with TCP or UDP <00000006 or <00000011
# For VSX set to correct environment manually
# NAT pool does not take into considertaion dst port as per SK156852

topcount=4  # Set how many top users to display
redthreshold=40000  # Highlight to show high usage
RED='\033[0;31m'
GRN='\033[0;32m'
CYN='\033[0;36m'
NC='\033[0m' # No Color

fw tab -t fwx_alloc_global -u > nat_table.raw

# Get top NAT IPs
echo -e "${GRN}"
echo -e "==== TOP NAT IP ADDRESSES ====${NC}"
echo -e "------------------------------"
cat nat_table.raw | sed 's/[><,;]//g' | egrep "^00000006|^00000011" | awk '{print $2}' | sort | uniq -c | sort -r | head -$topcount | while read line; do

  count=`echo "$line" | awk '{print $1}'`
  if [ $count -gt $redthreshold ]; then count="${RED}${count}${NC}"; fi
  ipaddr=`printf '%d.%d.%d.%d\n' $(echo $line | awk '{print $2}' | sed 's/../0x& /g')`
  while [ ${#ipaddr} -lt 20 ]; do ipaddr="$ipaddr "; done
  echo -e "  $ipaddr $count"
done
echo

# Get top destination IPs
echo -e "${GRN}"
echo -e "==== TOP DST IP ADDRESSES ====${NC}"
echo -e "------------------------------"
cat nat_table.raw | sed 's/[><,;]//g' | egrep "^00000006|^00000011" | awk '{print $4}' | sort | uniq -c | sort -r | head -$topcount | while read line; do

  count=`echo "$line" | awk '{print $1}'`
  ipaddr=`printf '%d.%d.%d.%d\n' $(echo $line | awk '{print $2}' | sed 's/../0x& /g')`
  while [ ${#ipaddr} -lt 20 ]; do ipaddr="$ipaddr "; done
  echo "  $ipaddr $count"
done
echo

# Get top destination IPs
echo -e "${GRN}"
echo -e "==== TOP SRC IP ADDRESSES ====${NC}"
echo -e "------------------------------"
cat nat_table.raw | sed 's/[><,;]//g' | egrep "^00000006|^00000011" | awk '{print $6}' | sort | uniq -c | sort -r | head -$topcount | while read line; do

  count=`echo "$line" | awk '{print $1}'`
  ipaddr=`printf '%d.%d.%d.%d\n' $(echo $line | awk '{print $2}' | sed 's/../0x& /g')`
  while [ ${#ipaddr} -lt 20 ]; do ipaddr="$ipaddr "; done
  echo "  $ipaddr $count"
done
echo


# Get top NAT pools as per sk156852
echo -e "${GRN}"
echo -e "=== TOP NAT POOLS SK156852 ===${NC}"
echo -e "------------------------------"
echo
cat nat_table.raw | sed 's/[><,;]//g' | egrep "^00000006|^00000011" | awk '{print $1" "$2" "$4}' | sort -k1 -k2 -k3 | uniq -c |  sort -r | head -$topcount | while read line; do

  count=`echo "$line" | awk '{print $1}'`
  if [ $count -gt $redthreshold ]; then count="${RED}${count}"; fi

  proto="TCP"
  if [ `echo $line | awk '{print $2}' | grep -c "00000006"` -eq 0 ]; then proto="UDP"; fi


  natIPhex=`echo $line | awk '{print $3}'`
  dstIPhex=`echo $line | awk '{print $4}'`
  natIP=`printf '%d.%d.%d.%d\n' $(echo $line | awk '{print $3}' | sed 's/../0x& /g')`
  dstIP=`printf '%d.%d.%d.%d\n' $(echo $line | awk '{print $4}' | sed 's/../0x& /g')`
  while [ ${#natIP} -lt 15 ]; do natIP="$natIP "; done
  while [ ${#dstIP} -lt 15 ]; do dstIP="$dstIP "; done

  echo -e "${CYN}  $proto $natIP > $dstIP  Total: $count${NC}"
  echo "  --------------------------------------------------"

  cat nat_table.raw | sed 's/[,;<>]//g' | egrep "^00000006|^00000011" | grep "$natIPhex" | grep "$dstIPhex" | awk '{print $6}' | sort | uniq -c | sort -r | head -$topcount | while read line2; do
    count=`echo "$line2" | awk '{print $1}'`
    ipaddr=`printf '%d.%d.%d.%d\n' $(echo $line2 | awk '{print $2}' | sed 's/../0x& /g')`
    while [ ${#ipaddr} -lt 20 ]; do ipaddr="$ipaddr "; done
    echo "    $ipaddr $count"
  done
  echo

done

[pjoseph]

This is awesome! I can confirm it works. Tested it on R80.40 and R81.20.

[Matlu]

Hi @Kaspars_Zibarts 
In a VSX environment, can your script be used?
The script must be “loaded” in the VS0? Or does it have to be loaded independently on each VS?
Thanks for your comments.

[Kaspars_Zibarts]

Yes, you select required VS (vsenv X) and then run the script 🙂