topic Re: Old Management Server Logs Transfer using Cron Jobs in Firewall and Security Management

Old Management Server Logs Transfer using Cron Jobs

Silvan_Nyambu — Mon, 06 May 2019 10:02:42 GMT

Hello Experts,

I wanted to setup scheduled log transfer on R77.30 or R80.20 management server using cron jobs if possible. The script should be able to check for three months old log files, transfer them to an offsite SFTP/FTP storage box and after the transfer is complete the files are deleted from the management server to create space.

Currently I mostly use WinSCP to copy the files manually and I was looking for a way that I can use to automate the process.

Does anyone have an Idea of how I can use cron jobs to achieve this? Kindly advise.

Thanks in advance.

Re: Old Management Server Logs Transfer using Cron Jobs

Nick_Doropoulos — Mon, 06 May 2019 11:36:07 GMT

Hi Silvan,

The first thing you would need to do is to identify the exact log files big enough to consume excessive amounts of disk space. The reason for that is that you don't want to accidentally delete any files you might need. You can identify the correct files by starting with df -h and then carry on analysing each partition recursively as outlined on sk60080. For the purpose of clarity, we will assume that the logs you want to delete are inside the /var/log directory (though again you would need to be as specific as possible).

The script you could start with could have the following form:

==========Script===============

#!/bin/bash

#Find log files inside the /var/log directory that are at least three months old and store those files in a #text file.
find /var/log -type f -mtime +90 2> /dev/null > log_files_to_transfer.txt

#Send all of the files found to the remote log server via sftp.
for file in log_files_to_transfer.txt; do
echo put $file | sftp -b- offsite-storage-box:path
done

#Find log files inside the /var/log directory that are at least three months old and delete them.
find /var/log -type f -mtime +90 2> /dev/null -exec rm -f {} \;
exit 0

==============================

Below are some instructions on how to use the above script:

1) Store the script inside the /bin/sh directory.

2) Make the script executable:

chmod u+x /bin/sh/log_transfer.sh

3) Edit the crontab file:

- crontab -e
- Press Shift+G to get to the bottom of the file followed by pressing o to move down one more line and get into insert mode as well.

4) To set up the cron job, type the following:

0 22 * * * /bin/sh/log_transfer.sh

5) To exit the crontab file:

- Press Escape
- Type :wq to save and exit

The purpose of this script is to find logs inside the /var/log directory older than 30 days, sftp them to a remote server and then delete the said logs. It would be run every day at 10pm (outside production hours). You can obviously adjust it to another frequency if you wish.

****Again, please note that the above script is just a template to start from as you need to be very specific as to which logs to delete. You don't want to delete any files that might prove to be critical.

I hope this helps.

Re: Old Management Server Logs Transfer using Cron Jobs

Silvan_Nyambu — Mon, 06 May 2019 14:18:30 GMT

Thanks Nick,

What we wanted to copy are the checkpoint log files. Below sample files to be exact.

-rw-rw---- 1 admin root 614M Dec 14 2016 2016-12-14_143512.log
-rw-rw---- 1 admin root 157 Dec 14 2016 2016-12-14_143512.log_stats
-rw-rw---- 1 admin root 80 Nov 26 2016 2016-12-14_143512.logaccount_ptr
-rw-rw---- 1 admin root 20M Dec 14 2016 2016-12-14_143512.loginitial_ptr
-rw-rw---- 1 admin root 22M Dec 14 2016 2016-12-14_143512.logptr
-rw-rw---- 1 admin root 4.9M Dec 14 2016 2016-12-14_173332.log
-rw-rw---- 1 admin root 80 Dec 14 2016 2016-12-14_173332.logaccount_ptr
-rw-rw---- 1 admin root 152K Dec 14 2016 2016-12-14_173332.loginitial_ptr
-rw-rw---- 1 admin root 165K Dec 14 2016 2016-12-14_173332.logptr