I'm really hoping this will help some out there maybe. And if so/not, maybe others will be able to help me by improving on this or showing me better ways to do things.
A little history and back story for this effort first.
We have 150+ clusters and growing all over the world. (Currently 77.30 but about to upgrade to 80.10) We have Tufin for orchestration and we use Splunk for logs and everything Splunk is supposed to do/help with. The big push over the past several years has been automation. I started hacking scripts together to do various tasks for our teams here. We have an ever growing issue with policy size specifically with one cluster which we have been struggling to keep below 10k rules. Verification of that policy takes (currently) 50 mins. (side note, with R80 in my vm lab it takes 15 mins) In our efforts to keep track of this I created a quick and dirt policy verification script. It's purpose was to verify every policy in every domain and time how long it takes. That evolved into much more and now I have 3 main components to my daily scripts. policy-verify.sh, gp-assign.sh, and MDS-Functions.sh.
gp-assign.sh assigns global policy out to a domain. if you pass -install it will install all policies for that domain.
policy-verify will do a few things which have shrunk a bit with R80 but the main thing is a policy verify for every policy in the given DMS and time how long it takes. It will also check for special characters using CP's rulename_check program (we were having that issue quite a bit). We have a couple other things it does specific to our environment which I cleaned out of the script I'm posting here.
MDS-Functions.sh the the guts of our efforts. all of our cron jobs call this file. Everything is region based. Our Domains all have the region in their name (for LAC/EMEA/APAC currently, our NA is the odd man out and still needs to be renamed) allowing for the script to use grep's to do work based on regions.
what you can pass to MDS-Functions:
- VERIFY-EMEA - verify all EMEA policies.
- VERIFY-NA - is actually everything but EMEA as that is primarily how we have our 2 MDS's split.
- EMEA/NA/LAC/APAC - any of these will install policies for any domains in given region
- GPONLY-EMEA/NA/LAC/APAC - assing global policy for given region.
The MDS-Functions script will then dump results to logger so they can get shipped to Splunk. The raw log of the effort is copied over to /var/log/Policy-Installs/ so we have some troubleshooting info if something odd happens. There is an email sent out at the beginning and end of our policy installs to notify folks.
We then use Splunk to alert on errors which is nice for trending and other things we can do with the data. We do a ton in Splunk so I have other stats scripts I run daily and every 15mins collecting info for trending and alerting. I can post some of those if folks are interested.
All of these files go into /sbin. If you put them elsewhere you will need to update the scripts.
I believe I have this all working for R80. Been working on the updates and cleanup for a while.
Version 3.0.1 - Removed rule name check bits and fixed issue with results manipulation
#!/bin/bash
#
# MDS-Functions.sh
#
# Main script for performing automated global policy assignments
# policy verifications and installs
#
# by Ivan Moore
#
# 1.0.0 - Initial - ugly version by Ivan
# 2.0.0 - mass cleanup
# 3.0.0 - R80 conversion
# 3.0.1 - Fix results manipulation
#
VERSION=3.0.0
# Get some variables setup
date=`date '+%B %e %Y'`
TODATE=$(date +%Y-%m-%d)
FROMSMTP="ivan.moore@firstdata.com"
TOSMTP="ivan.moore@firstdata.com"
TNAME="$$"
COLUMNS=237
MYNAME=`hostname`
export COLUMNS=237
# Load Check Point profile so we can make use of Check Point commands
. /etc/profile.d/CP.sh
function email_header {
cat >$PWD/$atype-Report-$TNAME.txt <<- HEADER
To: $NOTIF
From: ivan.moore@firstdata.com@firstdata.com
Subject: $email_subject $date
$body
HEADER
}
email_results () {
cat $1 | msmtp --from=$FROMSMTP -t $TOSMTP -C /etc/msmtp.conf
}
gp_assign () {
# gp_assign SLEEP -install
SLEEP=$1
INSTALL=$2
NEGATE=" "
FOCUS=" "
case $atype in
GP-EMEA) FOCUS="EMEA" ;;
GP-APAC) FOCUS="APAC\|CDC" ;;
GP-NA) FOCUS="EMEA\|APAC\|CDC\|LAC"; NEGATE="-v" ;;
GP-LAC) FOCUS="LAC" ;;
NA) NEGATE="-v"; FOCUS="LAC\|EMEA\|APAC\|CDC-FDR\|GLOBAL-VSX" ;;
EMEA) FOCUS="EMEA\|GLOBAL-VSX" ;;
APAC) FOCUS="APAC\|CDC-FDR" ;;
LAC) FOCUS="LAC" ;;
esac
if [ $FOCUS ];
then
COMPLETE=(`mgmt_cli show domains --root true --format json | grep name | grep -v "System Data" | sed -e 's/\"name\" : \"//' | sed -e 's/\",//' |sed 's/[][*]\|[[:space:]]//g' | grep $NEGATE $FOCUS`)
else
COMPLETE=(`mgmt_cli show domains --root true --format json | grep name | grep -v "System Data" | sed -e 's/\"name\" : \"//' | sed -e 's/\",//' |sed 's/[][*]\|[[:space:]]//g'`)
fi
if [ $INSTALL ];
then
if [ -e $NOINSTALL ]
then
NOTIF="TOSMTP"
email_subject="Nightly $LOCATION Policy Installation Cancelled"
body="Nightly $LOCATION Installation Cancelled - Remove $NOINSTALL on $MYNAME to Allow Installation"
email_header
email_results $Rep
rm $Rep
exit 0
else
if grep --quiet "$atype-$TODATE" /etc/MDS-Skip-Dates.conf ; then
NOTIF="TOSMTP"
email_subject="Nightly $LOCATION Policy Installation Skipped"
body="Nightly $LOCATION Installation Skipped due to today being listed in /etc/MDS-Skip-Dates.conf for $atype"
email_header
email_results $Rep
rm $Rep
exit 0
else
NOTIF="$ALERT"
email_subject="Nightly $LOCATION Policy Installation Beginning for"
body="The nightly Standard Firewall Policy installation for all $LOCATION Devices has started. This is an automated process that is actively monitored by the Ops team. If there are any concerns or issues, please reach out to the On Call"
email_header
email_results $Rep
echo " " >>$Rep
rm $Rep
fi
fi
NOTIF="TOSMTP"
email_subject="Nightly $LOCATION Policy Installation Complete Report for"
body="Below is the complete Installation report for the $LOCATION policy push."
email_header
fi
if [ $INSTALL ];
then
SKIP=(`cat /etc/MDS-Skip-Domains.conf | grep -v "#"`)
for i in "${SKIP[@]}"; do
COMPLETE=(${COMPLETE[@]//*$i*})
done
else
SKIP=(Test)
for i in "${SKIP[@]}"; do
COMPLETE=(${COMPLETE[@]//*$i*})
done
fi
if [ $INSTALL ];
then
g=5
echo "Policy Installation Starting" | logger -t "Policy_Installation" >/dev/null
else
g=1
echo "GP Assignment Starting" | logger -t "GP_Assignment" >/dev/null
fi
for((i=0; i < ${#COMPLETE[@]}; i+=g))
do
GP=`echo "${COMPLETE[@]:i:g}" | sed -e "s/ /,/g"`
if [ $INSTALL ];
then
echo "Starting install on $GP ..." >>$Log
echo "Starting install on $GP ..." | logger -t "Policy_Installation" >/dev/null
else
echo "Starting Global Policy Assignment on $GP ..." >>$Log
echo "Starting Global Policy Assignment on $GP ..." | logger -t "GP_Assignment" >/dev/null
fi
echo " " >>$Log
/sbin/gp-assign.sh $GP $INSTALL >>$Log 2>&1
echo " " >>$Log
sleep $SLEEP >>$Log
done
if [ $INSTALL ];
then
# Results manipulation
for POLICY in `cat $Log | grep '"task-name" : "Policy installation -' | sed -e 's/\"task-name\" : \"Policy installation - //' | sed -e 's/,//' | tr -d '"' |sed 's/[][*]\|[[:space:]]//g'`;
do
STATUS=`cat $Log | grep "\"task-name\" : \"Policy installation - $POLICY" -A 1 | grep '"status" : ' | sed -e 's/\"status\" : \"//' | sed -e 's/,//' | tr -d '"' |sed 's/[][*]\|[[:space:]]//g'`
if [ $STATUS == "failed" ];
then
PERRORS=`cat $Log |grep "\- $POLICY\" failed" -A 200 | grep '"type" : "err"' -A 1 | grep -v '"type" : "err"\|--\|"stage"' | sed -e 's/\"message\" : //' | sed -e 's/^[ \t]*//'`
eval 'for PERROR in '$PERRORS';
do
echo "$POLICY - $PERROR" | logger -t "Policy_Installation" >/dev/null
done'
else
echo "$POLICY installation $STATUS" | logger -t "Policy_Installation" >/dev/null
fi
done
cp $Log $PLog
echo "Policy Installation Complete" | logger -t "Policy_Installation" >/dev/null
rm -f $Log 2>&1
NOTIF="$ALERT"
email_subject="Nightly $LOCATION Policy Installation Completed for"
body="OPS has completed the Standard Firewall Policy Change for all $LOCATION Devices. Any failed policy pushes may be re-attempted manually. If there are any concerns or issues, please reach out to the On Call."
email_header
echo " " >>$Rep
email_results $Rep
else
for((i=0; i < ${#COMPLETE[@]}; i+=g))
do
DOMAIN=`echo "${COMPLETE[@]:i:g}" | sed -e "s/ /,/g"`
STATUS=`cat $Log | grep "\"task-name\" : \"Assign 'Global' to '$DOMAIN'" -A 1 | grep '"status" : ' | sed -e 's/\"status\" : \"//' | sed -e 's/,//' | tr -d '"' |sed 's/[][*]\|[[:space:]]//g'`
echo "GP Assignment to $DOMAIN $STATUS" | logger -t "GP_Assignment" >/dev/null
if [ $STATUS == "failed" ];
then
PERRORS=`cat $Log |grep "\"task-name\" : \"Assign 'Global' to '$DOMAIN'" -A 50 | grep "comments" | grep -v "\"comments\" : \"\"," | sed -e 's/\"comments\" : \"//' | sed -e 's/,//' | sed -e 's/^[ \t]*//'`
eval 'for PERROR in '$PERRORS';
do
echo "$DOMAIN - $PERROR" | logger -t "GP_Assignment" >/dev/null
done'
fi
done
cp $Log $PLog
echo "GP Assignment Complete" | logger -t "GP_Assignment" >/dev/null
fi
rm -f $Rep $Log 2>&1
}
policy_verify () {
if [ $atype == "VEREMEA" ];
then
DOMAINS=`mdsstat | grep -v grep | grep CMA | grep -v Test | grep EMEA | awk -F "|" '{print $3}' | sort -u`
fi
if [ $atype == "VERNA" ];
then
DOMAINS=`mdsstat | grep -v grep | grep CMA | grep -v Test | grep -v EMEA | awk -F "|" '{print $3}' | sort -u`
fi
for DMS in `echo $DOMAINS`
do
/sbin/policy-verify.sh $DMS $TNAME
cat $PWD/$DMS-verify-$TNAME.txt >> $Rep
rm $PWD/$DMS-verify-$TNAME.txt >& /dev/null
cat $PWD/$DMS-verify-failed-$TNAME.txt >> $Log
rm $PWD/$DMS-verify-failed-$TNAME.txt >& /dev/null
cat $PWD/$DMS-verify-syslog-$TNAME.txt >> $Sys
rm $PWD/$DMS-verify-syslog-$TNAME.txt >& /dev/null
done
mdsenv
echo " " >> $Rep
echo "------------------------" >> $Rep
echo "Out of State Check" >> $Rep
echo "------------------------" >> $Rep
echo " " >> $Rep
for DOMAIN in `mdsstat | grep -v grep | grep CMA | awk -F "|" '{print $3}' | sort -u`
do
mdsenv $DOMAIN
if grep -q ":fw_allow_out_of_state_tcp (1)" $FWDIR/conf/objects_5_0.C
then
echo "TCP OoS packets allowed for $DOMAIN" >> $Rep
echo "TCP OoS Packets - $DOMAIN: Allowed " >> $Sys
fi
if grep -q ":fw_drop_out_of_state_udp (false)" $FWDIR/conf/objects_5_0.C
then
echo "UDP OoS Packets - $DOMAIN: Allowed" >> $Sys
echo "UDP OoS packets allowed for $DOMAIN" >> $Rep
fi
if grep -q ":fw_drop_out_of_state_icmp (false)" $FWDIR/conf/objects_5_0.C
then
echo "ICMP OoS packets allowed for $DOMAIN" >> $Rep
echo "ICMP OoS Packets - $DOMAIN: Allowed" >> $Sys
fi
if grep -q ":fw_drop_out_of_state_sctp (false)" $FWDIR/conf/objects_5_0.C
then
echo "SCTP OoS Packets - $DOMAIN: Allowed" >> $Sys
echo "SCTP OoS packets allowed for $DOMAIN" >> $Rep
fi
done
echo " " >> $Rep
echo "------------------------" >> $Rep
echo "Detailed Failure Reports" >> $Rep
echo "------------------------" >> $Rep
echo " " >> $Rep
cat $Log >> $Rep
cp $Log $PLog
rm $Log >& /dev/null
echo "Policy Verification Complete" >> $Sys
cat $Sys 2>&1 | logger -t "Policy_Verification" >/dev/null
rm $Sys >& /dev/null
}
USAGE='Usage:\tMDS-Functions.sh [ GPONLY-NA | GPONLY-EMEA | GPONLY-APAC | GPONLY-LAC | VERIFY-NA | VERIFY-EMEA | NA | APAC | LAC | EMEA ]\n '
if [ $# == 0 ] ; then
echo -e $USAGE
exit 1;
fi
ASSIGNTYPE=$@
# if necessary create a working directory, then change to it as the PWD (present working directory)
my_temp="/var/log/tmp/MDS-Nightly"
[[ -d $my_temp ]] || mkdir $my_temp
cd $my_temp
PDIR="/var/log/Policy-Installs/"
[[ -d $PDIR ]] || mkdir $PDIR
case $ASSIGNTYPE in
GPONLY-NA)
atype="GP-NA"
Log="$PWD/$atype-Log-$TNAME.txt"
Rep="$PWD/$atype-Report-$TNAME.txt"
Sys="$PWD/$atype-Syslog-$TNAME.txt"
PLog="$PDIR/$TODATE-$atype.txt"
gp_assign "20"
;;
GPONLY-EMEA)
atype="GP-EMEA"
Log="$PWD/$atype-Log-$TNAME.txt"
Rep="$PWD/$atype-Report-$TNAME.txt"
Sys="$PWD/$atype-Syslog-$TNAME.txt"
PLog="$PDIR/$TODATE-$atype.txt"
gp_assign "20"
;;
GPONLY-APAC)
atype="GP-APAC"
Log="$PWD/$atype-Log-$TNAME.txt"
Rep="$PWD/$atype-Report-$TNAME.txt"
Sys="$PWD/$atype-Syslog-$TNAME.txt"
PLog="$PDIR/$TODATE-$atype.txt"
gp_assign "20"
;;
GPONLY-LAC)
atype="GP-LAC"
Log="$PWD/$atype-Log-$TNAME.txt"
Rep="$PWD/$atype-Report-$TNAME.txt"
Sys="$PWD/$atype-Syslog-$TNAME.txt"
PLog="$PDIR/$TODATE-$atype.txt"
gp_assign "20"
;;
NA)
atype="NA"
Log="$PWD/$atype-Log-$TNAME.txt"
Rep="$PWD/$atype-Report-$TNAME.txt"
Sys="$PWD/$atype-Syslog-$TNAME.txt"
PLog="$PDIR/$TODATE-$atype.txt"
LOCATION="NA"
NOTIF="$TOSMTP"
ALERT="$TOSMTP"
ALERT="$TOSMTP"
NOINSTALL="/NO-$atype"
gp_assign "20" "-install"
;;
APAC)
atype="APAC"
Log="$PWD/$atype-Log-$TNAME.txt"
Rep="$PWD/$atype-Report-$TNAME.txt"
Sys="$PWD/$atype-Syslog-$TNAME.txt"
PLog="$PDIR/$TODATE-$atype.txt"
LOCATION="APAC"
NOTIF="$TOSMTP"
ALERT="$TOSMTP"
NOINSTALL="/NO-$atype"
gp_assign "20" "-install"
;;
LAC)
atype="LAC"
Log="$PWD/$atype-Log-$TNAME.txt"
Rep="$PWD/$atype-Report-$TNAME.txt"
Sys="$PWD/$atype-Syslog-$TNAME.txt"
PLog="$PDIR/$TODATE-$atype.txt"
LOCATION="LAC"
NOTIF="$TOSMTP"
ALERT="$TOSMTP"
NOINSTALL="/NO-$atype"
gp_assign "20" "-install"
;;
EMEA)
atype="EMEA"
Log="$PWD/$atype-Log-$TNAME.txt"
Rep="$PWD/$atype-Report-$TNAME.txt"
Sys="$PWD/$atype-Syslog-$TNAME.txt"
PLog="$PDIR/$TODATE-$atype.txt"
LOCATION="EMEA"
NOTIF="$TOSMTP"
ALERT="$TOSMTP"
NOINSTALL="/NO-$atype"
gp_assign "20" "-install"
;;
VERIFY-NA)
atype="VERNA"
Log="$PWD/$atype-Log-$TNAME.txt"
Rep="$PWD/$atype-Report-$TNAME.txt"
Sys="$PWD/$atype-Syslog-$TNAME.txt"
policy_verify
rm $Rep $Log >& /dev/null
;;
VERIFY-EMEA)
atype="VEREMEA"
Log="$PWD/$atype-Log-$TNAME.txt"
Rep="$PWD/$atype-Report-$TNAME.txt"
Sys="$PWD/$atype-Syslog-$TNAME.txt"
policy_verify
rm $Rep $Log >& /dev/null
;;
*)
echo $USAGE
echo "I don't understand $ASSIGNTYPE, exiting."
exit 1
;;
esac
#!/bin/bash
# policy-verify.sh
#
# Policy Verification Script
# created by Ivan Moore
#
# This script will identify all of the policies within the
# provided Domain and run a policy verification against them
# dumping the output to $TDIR/$DMS-verify.txt
# It will also use the rulename_check program from CP against each policy
# to check for Special Characters.
#
# 3.0.0 - R80 conversion
# 3.0.1 - Remove rulecheck portion as it is not needed anymore
#
VERSION=3.0.1
date=`date '+%B %e %Y'`
sdate=`date '+%Y%m%d'`
COLUMNS=237
export COLUMNS=237
. /etc/profile.d/CP.sh
TDIR=/var/log/tmp/MDS-Nightly
USAGE='Usage:\tpolicy-verify.sh [DMS]\n '
if [ $# == 0 ] ; then
echo -e $USAGE
exit 1;
fi
DMS=$1
if [ -z "$2" ]
then
TNAME="$$"
else
TNAME=$2
fi
# Switch to requested Domain environment
if ! mdsenv $DMS >& /dev/null ;
then
echo "That is not a proper DMS name or IP"
exit 1
fi
echo "DMS: $DMS " >> $TDIR/$DMS-verify-$TNAME.txt
CMAIP=`mdsstat | grep $DMS | awk '{print $5}'`
echo "Policy Verifications:" >> $TDIR/$DMS-verify-$TNAME.txt
echo " " >> $TDIR/$DMS-verify-$TNAME.txt
for POLICY in $(mgmt_cli show-packages -d $CMAIP --root true --format text | grep -B1 package | grep name | awk '{print $2}' | tr -d '"' |sed 's/[][*]\|[[:space:]]//g')
do
start=`date +%s`
echo -n "Policy: $POLICY - " >> $TDIR/$DMS-verify-$TNAME.txt
echo -n "Policy Verification - $DMS - $POLICY: " >> $TDIR/$DMS-verify-syslog-$TNAME.txt
fwm verify $POLICY >>$TDIR/$POLICY-verify-$TNAME.txt 2>&1
end=`date +%s`
runtime=$((end-start))
if grep -q "Rules Verified OK!" $TDIR/$POLICY-verify-$TNAME.txt
then
echo "Success!" >> $TDIR/$DMS-verify-$TNAME.txt
echo "Success" >> $TDIR/$DMS-verify-syslog-$TNAME.txt
echo "Elapsed Time: $((runtime /60)) minutes and $(($runtime %60)) seconds" >> $TDIR/$DMS-verify-$TNAME.txt
echo "Policy Verification Time - $DMS - $POLICY: $((runtime /60)):$(($runtime %60))" >> $TDIR/$DMS-verify-syslog-$TNAME.txt
echo " " >> $TDIR/$DMS-verify-$TNAME.txt
else
echo "Failed!" >> $TDIR/$DMS-verify-$TNAME.txt
echo "Failed" >> $TDIR/$DMS-verify-syslog-$TNAME.txt
touch $TDIR/$DMS-verify-failed-$TNAME.txt
echo "Policy Verification Time - $DMS - $POLICY: $((runtime /60)):$(($runtime %60))" >> $TDIR/$DMS-verify-syslog-$TNAME.txt
echo "Elapsed Time: $((runtime /60)) minutes and $(($runtime %60)) seconds" >> $TDIR/$DMS-verify-$TNAME.txt
echo " " >> $TDIR/$DMS-verify-$TNAME.txt
echo "See Failure Report at bottom for details" >> $TDIR/$DMS-verify-$TNAME.txt
echo " " >> $TDIR/$DMS-verify-$TNAME.txt
echo "DMS: $DMS" >>$TDIR/$DMS-verify-failed-$TNAME.txt
echo "Policy: $POLICY" >>$TDIR/$DMS-verify-failed-$TNAME.txt
echo " " >>$TDIR/$DMS-verify-failed-$TNAME.txt
cat $TDIR/$POLICY-verify-$TNAME.txt >>$TDIR/$DMS-verify-failed-$TNAME.txt
echo " ">>$TDIR/$DMS-verify-failed-$TNAME.txt
echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=">>$TDIR/$DMS-verify-failed-$TNAME.txt
echo " ">>$TDIR/$DMS-verify-failed-$TNAME.txt
fi
rm $TDIR/$POLICY-verify-$TNAME.txt
done
echo "------------------------" >> $TDIR/$DMS-verify-$TNAME.txt
#!/bin/bash
#
# gp-assign.sh
#
# Global Policy Assignment and Installation Script
# created by Ivan Moore
#
# This Script will do re-assign of a Global Policy to a Domain
# if -install is passed it will also install policies within that Domain
# It also times the process
#
VERSION=2.0.0
date=`date '+%B %e %Y'`
. /etc/profile.d/CP.sh
# Name of the Global Domain
GPOL_NAME="Global"
USAGE='Usage: gp-assign.sh (Domain) [-install]\t\n '
if [ $# == 0 ] ; then
echo -e $USAGE
exit 1;
fi
DOMAINS=$1
for DOMAIN in `echo $DOMAINS | sed -e 's/,/ /g'`
do
echo "DOMAIN = $DOMAIN"
if [ "$2" == "-install" ] ; then
start=`date +%s`
for POLICY in $(mgmt_cli show-packages -d $DOMAIN --root true --format text | grep -B1 package | grep name | awk '{print $2}' | tr -d '"' |sed 's/[][*]\|[[:space:]]//g')
do
echo "Policy = $POLICY"
mgmt_cli install-policy policy-package $POLICY -d $DOMAIN access true --root true --format json
done
end=`date +%s`
runtime=$((end-start))
echo "Elapsed time for $DOMAIN: $((runtime /60)) minutes and $(($runtime %60)) seconds"
echo " "
else
if [ -z "`mgmt_cli show domain name ${DOMAIN} --format json --root true | jq -r '."global-domain-assignments"[]'`" ]; then
echo "Domain \"${DOMAIN}\" has no Global Domain assigned, skipping it."
else
echo "Reassigning Global Domain \"${GPOL_NAME}\" to Domain \"${DOMAIN}\"..."
mgmt_cli assign-global-assignment global-domains ${GPOL_NAME} dependent-domains ${DOMAIN} --root true --format json
fi
fi
done
