Pre/Post Change Script

Ivan_Moore — Fri, 16 Mar 2018 17:34:23 GMT

I've had this rolling around for a while and finally have some time to work on it. This community is the perfect place to drop this for expansion, correction, improvement,etc...

The idea is a pre/post change script that can be run via whatever method (manually via cli or via CDT, etc..) that would run various checks on the system and do some sort of compare afterwards. There would have to be some sort of logic to help the engineer spot issues or potential issues which is part of what I haven't started working on.

Phase 1 is identifying useful information that we need to be looking at and collecting it. See below. Calls out the ping-gateways.sh script which I will post below as well. I really need to add to the list of things to check to include CoreXL information, be VSX aware, etc...

The main thing I'm hoping to get for this phase is ideas for what to look for and the best way to collect it in a format that will make phase 2 easier.

Phase 2 is manipulating the data and comparing pre/post and presenting it in a decent manner to help speed up and improve the validation efforts. (Not started)

#!/bin/bash
#
# checkup.sh
#
# Initially created by Ivan Moore
#
# Pre/Post Change Validation script
#
# 1.0.0 Initial version which just collects data
#
VERSION=1.0.0
MYNAME=`hostname`
date=`date '+%d%b%Y'`
time=`date '+%X'`
. /etc/profile.d/CP.sh
TLOG="/var/log/tmp/$MYNAME/$MYNAME-$time.txt"
USAGE='Usage:\tcheckup.sh\n '


collect () {
  echo Running $1
  echo "#################" >>$TLOG
  echo "Running $1" >>$TLOG
  echo "#################" >>$TLOG
  echo "#" >>$TLOG
  echo " " >>$TLOG
  $1 >>$TLOG
  echo " " >>$TLOG
}


#######
#
echo "Setting up some temp space and our log file"
#

mkdir /var/log/tmp/$MYNAME >/dev/null 2>&1
cd /var/log/tmp/$MYNAME

echo "############# Validation Log ############" >$TLOG
echo "#############    $date   ############" >>$TLOG
echo "#############    $time    ############" >>$TLOG
echo "#########################################" >>$TLOG
echo "" >>$TLOG

######
#
#
#

collect "cphaprob -a if"
collect "cphaprob -l list"
collect "cphaprob stat"
collect "fwaccel stats"
collect "fwaccel stats -s"
collect "fwaccel stat"
collect "ping-gateways.sh"
collect "netstat -i"

dmesg | tail >>$TLOG

echo "Running ethtool and checking bonds"
for IF in `ifconfig -a | grep -v grep | grep HWaddr | awk '{print $1}' | grep -v bond`
  do
    echo -n "$IF: ">>$TLOG;
    ethtool -S "$IF" >>$TLOG
done

for i in `ifconfig | grep bond | awk ' { print $1 }' | grep -v "\."`; do \
  cphaconf show_bond $i >>$TLOG;\
done

collect "cpview -p"

#######
echo " "
echo " "
echo "Some bits to look at."
echo " "
echo "### Interface Errors - If Any"
echo " "
netstat -i | grep -v "\." | awk '{ print $1 "\t\t" $5 "\t" $6 "\t" $7 "\t" $9 "\t" $10 "\t" $11 }'
echo " "
echo "Cluster State: " `cphaprob stat | grep "(local)" |  awk {' print $5'}`
echo "Currenet # of Connections: " `fw tab -t connections -s | grep "connections" | awk {' print $4 '}`

echo " "
echo "Full log file can be found here:"
echo "    $TLOG"
echo " "
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

#!/bin/bash
#
#  ping-gateways.sh
#
# [Ivan Moore]
#
# Determine all next hop gateways from routing table and ping
# to make sure we can reach them.  If no ping response is received
# check ARP table to see if we have L2 just in case the device is
# not allowed to respond.
#
# 11/19/2015

VERSION=1.0.0

# Program name: ping script to ping next-hop addresses
date
ip route | awk '/via/ { print $3 }' | sort -u | while read output
do
    ping -c 1 -w 1 "$output" > /dev/null
    if [ $? -eq 0 ]; then
      echo "node $output is up"
    else
      echo " "
      echo "node $output is down"
      echo "Checking for an ARP entry in case PING is disabled:"
      arp -a $output &
      TASK_PID=$!
      sleep 20
      kill $TASK_PID >/dev/null 2>&1
      echo " "
    fi
done
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Pre/Post Change Script

KernelGordon — Tue, 20 Mar 2018 19:45:16 GMT

I like the idea of the project. Are you planning on putting this on GitHub or just adding to the project and updating the post here?

Re: Pre/Post Change Script

Ivan_Moore — Tue, 20 Mar 2018 20:57:49 GMT

I hadn't given it that much thought yet. I guess whichever makes more sense. This was the initial to get it out there. Now I'm committed.

topic Re: Pre/Post Change Script in API / CLI Discussion

Pre/Post Change Script

Re: Pre/Post Change Script

Re: Pre/Post Change Script