I wrote following python script that you run on management. It will request a start and end date (can be just 1 day) and a name for an archive file. It will go through and export all logs in the given range (without name/port resolution) into /var/log/tmp and then create a tarball under the same based on archive name provided. Once launched you can do the following:
1.Run the program you want to run: <python script name>
2.Hit Ctrl+Z
3.Type: disown -h %(job number displayed in “Stopped” message of Ctrl+Z)
4.Type: bg 1(job number displayed in “Stopped” message of Ctrl+Z)
5.Logout
Note: you must logout - you can log back in and monitor the job using 'ps aux | grep fwm' which should show which log file is currently being processed, however, if you stay logged in the job will finish prematurely.
Takes roughly around 15 to 20 minutes per 2 gb log file
-------------------------------------------------------------------------------
#!/usr/bin/python3
import time
from datetime import date, timedelta
import datetime
import fnmatch
import os
import subprocess
# Set static variables
path = os.environ["FWDIR"]
timestr = time.strftime("%Y-%m-%d")
#Get begin date
date_entry = input('Enter first date in range YYYY-MM-DD format: ')
year, month, day = map(int, date_entry.split('-'))
Startdate = datetime.date(year, month, day)
#Get end date
date_entry = input('Enter last date in range YYYY-MM-DD format: ')
year, month, day = map(int, date_entry.split('-'))
Enddate= datetime.date(year, month, day)
#Filename for tarball
archive = input('Please enter name for tarball: ')
#compare beginging date from end date and print current
delta = timedelta(days=1)
# Process log files
while Startdate <= Enddate:
date = (Startdate.strftime("%Y-%m-%d"))
range = f'{date}*.log'
for file in os.listdir('%s/log/' % path):
if fnmatch.fnmatch(file, range):
subprocess.call(['fwm logexport -n -p -i %s -o /var/log/tmp/%s.txt' % (file, file)],shell=True)
Startdate += delta
#Create Tarball of exported logs
print("All log files processed")
print("zipping up logs")
subprocess.call(['tar -cvzf /var/log/tmp/%s-%s-".tgz" /var/log/tmp/*.txt' %(archive, timestr)],shell=True)
print("tarball completed")
