Nightly Automation Scripts

Ivan_Moore — Thu, 15 Mar 2018 00:38:35 GMT

I'm really hoping this will help some out there maybe. And if so/not, maybe others will be able to help me by improving on this or showing me better ways to do things.

A little history and back story for this effort first.

We have 150+ clusters and growing all over the world. (Currently 77.30 but about to upgrade to 80.10) We have Tufin for orchestration and we use Splunk for logs and everything Splunk is supposed to do/help with. The big push over the past several years has been automation. I started hacking scripts together to do various tasks for our teams here. We have an ever growing issue with policy size specifically with one cluster which we have been struggling to keep below 10k rules. Verification of that policy takes (currently) 50 mins. (side note, with R80 in my vm lab it takes 15 mins) In our efforts to keep track of this I created a quick and dirt policy verification script. It's purpose was to verify every policy in every domain and time how long it takes. That evolved into much more and now I have 3 main components to my daily scripts. policy-verify.sh, gp-assign.sh, and MDS-Functions.sh.

gp-assign.sh assigns global policy out to a domain. if you pass -install it will install all policies for that domain.

policy-verify will do a few things which have shrunk a bit with R80 but the main thing is a policy verify for every policy in the given DMS and time how long it takes. It will also check for special characters using CP's rulename_check program (we were having that issue quite a bit). We have a couple other things it does specific to our environment which I cleaned out of the script I'm posting here.

MDS-Functions.sh the the guts of our efforts. all of our cron jobs call this file. Everything is region based. Our Domains all have the region in their name (for LAC/EMEA/APAC currently, our NA is the odd man out and still needs to be renamed) allowing for the script to use grep's to do work based on regions.

what you can pass to MDS-Functions:

VERIFY-EMEA - verify all EMEA policies.
VERIFY-NA - is actually everything but EMEA as that is primarily how we have our 2 MDS's split.
EMEA/NA/LAC/APAC - any of these will install policies for any domains in given region
GPONLY-EMEA/NA/LAC/APAC - assing global policy for given region.

The MDS-Functions script will then dump results to logger so they can get shipped to Splunk. The raw log of the effort is copied over to /var/log/Policy-Installs/ so we have some troubleshooting info if something odd happens. There is an email sent out at the beginning and end of our policy installs to notify folks.

We then use Splunk to alert on errors which is nice for trending and other things we can do with the data. We do a ton in Splunk so I have other stats scripts I run daily and every 15mins collecting info for trending and alerting. I can post some of those if folks are interested.

All of these files go into /sbin. If you put them elsewhere you will need to update the scripts.

I believe I have this all working for R80. Been working on the updates and cleanup for a while.

Version 3.0.1 - Removed rule name check bits and fixed issue with results manipulation

#!/bin/bash
#
# MDS-Functions.sh
#
# Main script for performing automated global policy assignments
# policy verifications and installs
#
# by Ivan Moore
#
# 1.0.0 - Initial - ugly version by Ivan
# 2.0.0 - mass cleanup
# 3.0.0 - R80 conversion
# 3.0.1 - Fix results manipulation 
#

VERSION=3.0.0

# Get some variables setup
date=`date '+%B %e %Y'`
TODATE=$(date +%Y-%m-%d)
FROMSMTP="ivan.moore@firstdata.com"
TOSMTP="ivan.moore@firstdata.com"
TNAME="$$"
COLUMNS=237
MYNAME=`hostname`
export COLUMNS=237

# Load Check Point profile so we can make use of Check Point commands
. /etc/profile.d/CP.sh


function email_header {
        cat >$PWD/$atype-Report-$TNAME.txt <<- HEADER
                To: $NOTIF
                From: ivan.moore@firstdata.com@firstdata.com
                Subject: $email_subject $date

$body

HEADER
}

email_results () {
  cat $1 | msmtp --from=$FROMSMTP -t $TOSMTP -C /etc/msmtp.conf
}

gp_assign () {
# gp_assign SLEEP -install
  SLEEP=$1
  INSTALL=$2

NEGATE=" "
FOCUS=" "
case $atype in
  GP-EMEA) FOCUS="EMEA" ;;
  GP-APAC) FOCUS="APAC\|CDC" ;;
  GP-NA) FOCUS="EMEA\|APAC\|CDC\|LAC"; NEGATE="-v" ;;
  GP-LAC) FOCUS="LAC" ;;
  NA) NEGATE="-v"; FOCUS="LAC\|EMEA\|APAC\|CDC-FDR\|GLOBAL-VSX" ;;
  EMEA) FOCUS="EMEA\|GLOBAL-VSX" ;;
  APAC) FOCUS="APAC\|CDC-FDR" ;;
  LAC) FOCUS="LAC" ;;
esac

if [ $FOCUS ];
then
  COMPLETE=(`mgmt_cli show domains --root true --format json | grep name | grep -v "System Data" | sed -e 's/\"name\" : \"//' | sed -e 's/\",//' |sed 's/[][*]\|[[:space:]]//g' | grep $NEGATE $FOCUS`)
else
  COMPLETE=(`mgmt_cli show domains --root true --format json | grep name | grep -v "System Data" | sed -e 's/\"name\" : \"//' | sed -e 's/\",//' |sed 's/[][*]\|[[:space:]]//g'`)
fi


  if [ $INSTALL ];
  then

    if [ -e $NOINSTALL ]
    then
      NOTIF="TOSMTP"
      email_subject="Nightly $LOCATION Policy Installation Cancelled"
      body="Nightly $LOCATION Installation Cancelled - Remove $NOINSTALL on $MYNAME to Allow Installation"
      email_header
      email_results $Rep
      rm $Rep
      exit 0
    else
      if grep --quiet "$atype-$TODATE" /etc/MDS-Skip-Dates.conf ; then
        NOTIF="TOSMTP"
        email_subject="Nightly $LOCATION Policy Installation Skipped"
        body="Nightly $LOCATION Installation Skipped due to today being listed in /etc/MDS-Skip-Dates.conf for $atype"
        email_header
        email_results $Rep
        rm $Rep
        exit 0
      else
        NOTIF="$ALERT"
        email_subject="Nightly $LOCATION Policy Installation Beginning for"
        body="The nightly Standard Firewall Policy installation for all $LOCATION Devices has started. This is an automated process that is actively monitored by the Ops team.  If there are any concerns or issues, please reach out to the On Call"
        email_header
        email_results $Rep
        echo " " >>$Rep
        rm $Rep
      fi
    fi

    NOTIF="TOSMTP"
    email_subject="Nightly $LOCATION Policy Installation Complete Report for"
    body="Below is the complete Installation report for the $LOCATION policy push."
    email_header
  fi


  if [ $INSTALL ];
  then
    SKIP=(`cat /etc/MDS-Skip-Domains.conf | grep -v "#"`)
    for i in "${SKIP[@]}"; do
      COMPLETE=(${COMPLETE[@]//*$i*})
    done
  else
    SKIP=(Test)
    for i in "${SKIP[@]}"; do
      COMPLETE=(${COMPLETE[@]//*$i*})
    done
  fi

  if [ $INSTALL ];
  then
    g=5
    echo "Policy Installation Starting"  | logger -t "Policy_Installation" >/dev/null
  else
    g=1
    echo "GP Assignment Starting"  | logger -t "GP_Assignment" >/dev/null
  fi

  for((i=0; i < ${#COMPLETE[@]}; i+=g))
  do
    GP=`echo "${COMPLETE[@]:i:g}" | sed -e "s/ /,/g"`
    if [ $INSTALL ];
    then
      echo "Starting install on $GP ..." >>$Log
      echo "Starting install on $GP ..." | logger -t "Policy_Installation" >/dev/null
    else
      echo "Starting Global Policy Assignment on $GP ..." >>$Log
      echo "Starting Global Policy Assignment on $GP ..." | logger -t "GP_Assignment" >/dev/null
    fi
    echo " " >>$Log
    /sbin/gp-assign.sh $GP $INSTALL >>$Log 2>&1
    echo " " >>$Log
    sleep $SLEEP >>$Log
  done

  if [ $INSTALL ];
  then

    # Results manipulation
    for POLICY in `cat $Log | grep '"task-name" : "Policy installation -' | sed -e 's/\"task-name\" : \"Policy installation - //' | sed -e 's/,//' | tr -d '"' |sed 's/[][*]\|[[:space:]]//g'`;
    do
      STATUS=`cat $Log | grep "\"task-name\" : \"Policy installation - $POLICY" -A 1 | grep '"status" : ' | sed -e 's/\"status\" : \"//' | sed -e 's/,//' | tr -d '"' |sed 's/[][*]\|[[:space:]]//g'`

      if [ $STATUS == "failed" ];
        then
          PERRORS=`cat $Log |grep "\- $POLICY\"  failed" -A 200 | grep '"type" : "err"' -A 1 | grep -v '"type" : "err"\|--\|"stage"' | sed -e 's/\"message\" : //' | sed -e 's/^[ \t]*//'`
          eval 'for PERROR in '$PERRORS';
          do
            echo "$POLICY - $PERROR"  | logger -t "Policy_Installation" >/dev/null
          done'
        else
          echo "$POLICY installation $STATUS"  | logger -t "Policy_Installation" >/dev/null
        fi

    done

    cp $Log $PLog
    echo "Policy Installation Complete" | logger -t "Policy_Installation" >/dev/null
    rm -f $Log 2>&1

    NOTIF="$ALERT"
    email_subject="Nightly $LOCATION Policy Installation Completed for"
    body="OPS has completed the Standard Firewall Policy Change for all $LOCATION Devices.  Any failed policy pushes may be re-attempted manually.  If there are any concerns or issues, please reach out to the On Call."
    email_header
    echo " " >>$Rep
    email_results $Rep
  else

    for((i=0; i < ${#COMPLETE[@]}; i+=g))
    do
      DOMAIN=`echo "${COMPLETE[@]:i:g}" | sed -e "s/ /,/g"`
      STATUS=`cat $Log | grep "\"task-name\" : \"Assign 'Global' to '$DOMAIN'" -A 1 | grep '"status" : ' | sed -e 's/\"status\" : \"//' | sed -e 's/,//' | tr -d '"' |sed 's/[][*]\|[[:space:]]//g'`

      echo "GP Assignment to $DOMAIN $STATUS" | logger -t "GP_Assignment" >/dev/null
      if [ $STATUS == "failed" ];
        then
          PERRORS=`cat $Log  |grep "\"task-name\" : \"Assign 'Global' to '$DOMAIN'" -A 50  | grep "comments" | grep -v "\"comments\" : \"\"," | sed -e 's/\"comments\" : \"//' | sed -e 's/,//'  | sed -e 's/^[ \t]*//'`
          eval 'for PERROR in '$PERRORS';
          do
            echo "$DOMAIN - $PERROR" | logger -t "GP_Assignment" >/dev/null
          done'
        fi
    done
    cp $Log $PLog
    echo "GP Assignment Complete" | logger -t "GP_Assignment" >/dev/null
  fi

  rm -f $Rep $Log 2>&1
}

policy_verify () {

  if [ $atype == "VEREMEA" ];
  then
    DOMAINS=`mdsstat | grep -v grep | grep CMA | grep -v Test | grep EMEA | awk -F "|" '{print $3}' | sort -u`
  fi
  if [ $atype == "VERNA" ];
  then
    DOMAINS=`mdsstat | grep -v grep | grep CMA | grep -v Test | grep -v EMEA | awk -F "|" '{print $3}' | sort -u`
  fi

  for DMS in `echo $DOMAINS`
  do
    /sbin/policy-verify.sh $DMS $TNAME
    cat $PWD/$DMS-verify-$TNAME.txt >> $Rep
    rm $PWD/$DMS-verify-$TNAME.txt >& /dev/null
    cat $PWD/$DMS-verify-failed-$TNAME.txt >> $Log
    rm $PWD/$DMS-verify-failed-$TNAME.txt >& /dev/null
    cat $PWD/$DMS-verify-syslog-$TNAME.txt >> $Sys
    rm $PWD/$DMS-verify-syslog-$TNAME.txt >& /dev/null
  done

  mdsenv
  echo " " >> $Rep

  echo "------------------------" >> $Rep
  echo "Out of State Check" >> $Rep
  echo "------------------------" >> $Rep
  echo " " >> $Rep

  for DOMAIN in `mdsstat | grep -v grep | grep CMA  | awk -F "|" '{print $3}' | sort -u`
  do
    mdsenv $DOMAIN
    if grep -q ":fw_allow_out_of_state_tcp (1)" $FWDIR/conf/objects_5_0.C
    then
      echo "TCP OoS packets allowed for $DOMAIN" >> $Rep
      echo "TCP OoS Packets - $DOMAIN: Allowed " >> $Sys
    fi
    if grep -q ":fw_drop_out_of_state_udp (false)" $FWDIR/conf/objects_5_0.C
    then
      echo "UDP OoS Packets - $DOMAIN: Allowed" >> $Sys
      echo "UDP OoS packets allowed for $DOMAIN" >> $Rep
    fi
    if grep -q ":fw_drop_out_of_state_icmp (false)" $FWDIR/conf/objects_5_0.C
    then
      echo "ICMP OoS packets allowed for $DOMAIN" >> $Rep
      echo "ICMP OoS Packets - $DOMAIN: Allowed" >> $Sys
    fi
    if grep -q ":fw_drop_out_of_state_sctp (false)" $FWDIR/conf/objects_5_0.C
    then
      echo "SCTP OoS Packets - $DOMAIN: Allowed" >> $Sys
      echo "SCTP OoS packets allowed for $DOMAIN" >> $Rep
    fi
  done

  echo " " >> $Rep

  echo "------------------------" >> $Rep
  echo "Detailed Failure Reports" >> $Rep
  echo "------------------------" >> $Rep
  echo " " >> $Rep
  cat $Log >> $Rep
  cp $Log $PLog

  rm $Log >& /dev/null

  echo "Policy Verification Complete" >> $Sys
  cat $Sys 2>&1 | logger -t "Policy_Verification" >/dev/null
  rm $Sys >& /dev/null

}



USAGE='Usage:\tMDS-Functions.sh [ GPONLY-NA | GPONLY-EMEA | GPONLY-APAC | GPONLY-LAC | VERIFY-NA | VERIFY-EMEA | NA | APAC | LAC | EMEA ]\n '

if [ $# == 0 ] ; then
    echo -e $USAGE
    exit 1;
fi

ASSIGNTYPE=$@

# if necessary create a working directory, then change to it as the PWD (present working directory)
my_temp="/var/log/tmp/MDS-Nightly"
[[ -d $my_temp ]] || mkdir $my_temp
cd $my_temp

PDIR="/var/log/Policy-Installs/"
[[ -d $PDIR ]] || mkdir $PDIR


case $ASSIGNTYPE in
        GPONLY-NA)
                atype="GP-NA"
                Log="$PWD/$atype-Log-$TNAME.txt"
                Rep="$PWD/$atype-Report-$TNAME.txt"
                Sys="$PWD/$atype-Syslog-$TNAME.txt"
                PLog="$PDIR/$TODATE-$atype.txt"
                gp_assign "20"
                ;;
        GPONLY-EMEA)
                atype="GP-EMEA"
                Log="$PWD/$atype-Log-$TNAME.txt"
                Rep="$PWD/$atype-Report-$TNAME.txt"
                Sys="$PWD/$atype-Syslog-$TNAME.txt"
                PLog="$PDIR/$TODATE-$atype.txt"
                gp_assign "20"
                ;;
        GPONLY-APAC)
                atype="GP-APAC"
                Log="$PWD/$atype-Log-$TNAME.txt"
                Rep="$PWD/$atype-Report-$TNAME.txt"
                Sys="$PWD/$atype-Syslog-$TNAME.txt"
                PLog="$PDIR/$TODATE-$atype.txt"
                gp_assign "20"
                ;;
        GPONLY-LAC)
                atype="GP-LAC"
                Log="$PWD/$atype-Log-$TNAME.txt"
                Rep="$PWD/$atype-Report-$TNAME.txt"
                Sys="$PWD/$atype-Syslog-$TNAME.txt"
                PLog="$PDIR/$TODATE-$atype.txt"
                gp_assign "20"
                ;;
        NA)
                atype="NA"
                Log="$PWD/$atype-Log-$TNAME.txt"
                Rep="$PWD/$atype-Report-$TNAME.txt"
                Sys="$PWD/$atype-Syslog-$TNAME.txt"
                PLog="$PDIR/$TODATE-$atype.txt"
                LOCATION="NA"
                NOTIF="$TOSMTP"
                ALERT="$TOSMTP"
                ALERT="$TOSMTP"
                NOINSTALL="/NO-$atype"
                gp_assign "20" "-install"
                ;;
        APAC)
                atype="APAC"
                Log="$PWD/$atype-Log-$TNAME.txt"
                Rep="$PWD/$atype-Report-$TNAME.txt"
                Sys="$PWD/$atype-Syslog-$TNAME.txt"
                PLog="$PDIR/$TODATE-$atype.txt"
                LOCATION="APAC"
                NOTIF="$TOSMTP"
                ALERT="$TOSMTP"
                NOINSTALL="/NO-$atype"
                gp_assign "20" "-install"
                ;;
        LAC)
                atype="LAC"
                Log="$PWD/$atype-Log-$TNAME.txt"
                Rep="$PWD/$atype-Report-$TNAME.txt"
                Sys="$PWD/$atype-Syslog-$TNAME.txt"
                PLog="$PDIR/$TODATE-$atype.txt"
                LOCATION="LAC"
                NOTIF="$TOSMTP"
                ALERT="$TOSMTP"
                NOINSTALL="/NO-$atype"
                gp_assign "20" "-install"
                ;;
        EMEA)
                atype="EMEA"
                Log="$PWD/$atype-Log-$TNAME.txt"
                Rep="$PWD/$atype-Report-$TNAME.txt"
                Sys="$PWD/$atype-Syslog-$TNAME.txt"
                PLog="$PDIR/$TODATE-$atype.txt"
                LOCATION="EMEA"
                NOTIF="$TOSMTP"
                ALERT="$TOSMTP"
                NOINSTALL="/NO-$atype"
                gp_assign "20" "-install"
                ;;
        VERIFY-NA)
                atype="VERNA"
                Log="$PWD/$atype-Log-$TNAME.txt"
                Rep="$PWD/$atype-Report-$TNAME.txt"
                Sys="$PWD/$atype-Syslog-$TNAME.txt"
                policy_verify
                rm $Rep $Log >& /dev/null
                ;;
        VERIFY-EMEA)
                atype="VEREMEA"
                Log="$PWD/$atype-Log-$TNAME.txt"
                Rep="$PWD/$atype-Report-$TNAME.txt"
                Sys="$PWD/$atype-Syslog-$TNAME.txt"
                policy_verify
                rm $Rep $Log >& /dev/null
                ;;
        *)
                echo $USAGE
                echo "I don't understand $ASSIGNTYPE, exiting."
                exit 1
                ;;
esac

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

#!/bin/bash
# policy-verify.sh
#
# Policy Verification Script
# created by Ivan Moore
#
# This script will identify all of the policies within the
# provided Domain and run a policy verification against them
# dumping the output to $TDIR/$DMS-verify.txt
# It will also use the rulename_check program from CP against each policy
# to check for Special Characters.  
#
# 3.0.0 - R80 conversion
# 3.0.1 - Remove rulecheck portion as it is not needed anymore
#
VERSION=3.0.1
date=`date '+%B %e %Y'`
sdate=`date '+%Y%m%d'`
COLUMNS=237
export COLUMNS=237
. /etc/profile.d/CP.sh

TDIR=/var/log/tmp/MDS-Nightly

USAGE='Usage:\tpolicy-verify.sh [DMS]\n '


if [ $# == 0 ] ; then
    echo -e $USAGE
    exit 1;
fi

DMS=$1
if [ -z "$2" ]
then
    TNAME="$$"
else
    TNAME=$2
fi

# Switch to requested Domain environment
if ! mdsenv $DMS >& /dev/null ;
then
  echo "That is not a proper DMS name or IP"
  exit 1
fi

echo "DMS: $DMS " >> $TDIR/$DMS-verify-$TNAME.txt
CMAIP=`mdsstat | grep $DMS | awk '{print $5}'`

echo "Policy Verifications:" >> $TDIR/$DMS-verify-$TNAME.txt
echo " " >> $TDIR/$DMS-verify-$TNAME.txt

for POLICY in $(mgmt_cli show-packages -d $CMAIP --root true --format text | grep -B1 package | grep name | awk '{print $2}' | tr -d '"' |sed 's/[][*]\|[[:space:]]//g')
        do
        start=`date +%s`
        echo -n "Policy:  $POLICY  -  " >> $TDIR/$DMS-verify-$TNAME.txt
        echo -n "Policy Verification - $DMS - $POLICY: " >> $TDIR/$DMS-verify-syslog-$TNAME.txt
        fwm verify $POLICY >>$TDIR/$POLICY-verify-$TNAME.txt  2>&1
        end=`date +%s`
        runtime=$((end-start))
        if grep -q "Rules Verified OK!" $TDIR/$POLICY-verify-$TNAME.txt
        then
          echo "Success!" >> $TDIR/$DMS-verify-$TNAME.txt
          echo "Success" >> $TDIR/$DMS-verify-syslog-$TNAME.txt
          echo "Elapsed Time: $((runtime /60)) minutes and $(($runtime %60)) seconds" >> $TDIR/$DMS-verify-$TNAME.txt
          echo "Policy Verification Time - $DMS - $POLICY: $((runtime /60)):$(($runtime %60))" >> $TDIR/$DMS-verify-syslog-$TNAME.txt
          echo " " >> $TDIR/$DMS-verify-$TNAME.txt
        else
          echo "Failed!" >> $TDIR/$DMS-verify-$TNAME.txt
          echo "Failed" >> $TDIR/$DMS-verify-syslog-$TNAME.txt
          touch $TDIR/$DMS-verify-failed-$TNAME.txt
          echo "Policy Verification Time - $DMS - $POLICY: $((runtime /60)):$(($runtime %60))" >> $TDIR/$DMS-verify-syslog-$TNAME.txt
          echo "Elapsed Time: $((runtime /60)) minutes and $(($runtime %60)) seconds" >> $TDIR/$DMS-verify-$TNAME.txt
          echo " " >> $TDIR/$DMS-verify-$TNAME.txt
          echo "See Failure Report at bottom for details" >> $TDIR/$DMS-verify-$TNAME.txt
          echo " " >> $TDIR/$DMS-verify-$TNAME.txt
          echo "DMS: $DMS" >>$TDIR/$DMS-verify-failed-$TNAME.txt
          echo "Policy: $POLICY" >>$TDIR/$DMS-verify-failed-$TNAME.txt
          echo " " >>$TDIR/$DMS-verify-failed-$TNAME.txt
          cat $TDIR/$POLICY-verify-$TNAME.txt >>$TDIR/$DMS-verify-failed-$TNAME.txt
          echo " ">>$TDIR/$DMS-verify-failed-$TNAME.txt
          echo "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=">>$TDIR/$DMS-verify-failed-$TNAME.txt
          echo " ">>$TDIR/$DMS-verify-failed-$TNAME.txt
        fi
        rm $TDIR/$POLICY-verify-$TNAME.txt
done

echo "------------------------" >> $TDIR/$DMS-verify-$TNAME.txt

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

#!/bin/bash
#
# gp-assign.sh
#
# Global Policy Assignment and Installation Script
# created by Ivan Moore
#
# This Script will do re-assign of a Global Policy to a Domain
# if -install is passed it will also install policies within that Domain
# It also times the process 
#
VERSION=2.0.0
date=`date '+%B %e %Y'`
. /etc/profile.d/CP.sh

# Name of the Global Domain
GPOL_NAME="Global"


USAGE='Usage:  gp-assign.sh (Domain) [-install]\t\n '

if [ $# == 0 ] ; then
    echo -e $USAGE
    exit 1;
fi

DOMAINS=$1
for DOMAIN in `echo $DOMAINS | sed -e 's/,/ /g'`
do
  echo "DOMAIN = $DOMAIN"
  if [ "$2" == "-install" ] ; then
    start=`date +%s`
    for POLICY in $(mgmt_cli show-packages -d $DOMAIN --root true --format text | grep -B1 package | grep name | awk '{print $2}' | tr -d '"' |sed 's/[][*]\|[[:space:]]//g')
        do
        echo "Policy = $POLICY"
        mgmt_cli install-policy policy-package $POLICY -d $DOMAIN access true --root true --format json
        done
    end=`date +%s`
    runtime=$((end-start))
    echo "Elapsed time for $DOMAIN: $((runtime /60)) minutes and $(($runtime %60)) seconds"
    echo " "
  else
    if [ -z "`mgmt_cli show domain name ${DOMAIN} --format json --root true | jq -r '."global-domain-assignments"[]'`" ]; then
        echo "Domain \"${DOMAIN}\" has no Global Domain assigned, skipping it."
    else
        echo "Reassigning Global Domain \"${GPOL_NAME}\" to Domain \"${DOMAIN}\"..."
        mgmt_cli assign-global-assignment global-domains ${GPOL_NAME} dependent-domains ${DOMAIN} --root true --format json
    fi
  fi
done

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

topic Re: Nightly Automation Scripts in API / CLI Discussion

Nightly Automation Scripts

Re: Nightly Automation Scripts

Re: Nightly Automation Scripts

Re: Nightly Automation Scripts

Re: Nightly Automation Scripts

Re: Nightly Automation Scripts

Re: Nightly Automation Scripts