1

I have a folder structure in /var/log like this:

.
├── customers
│   ├── core00001
│   │   ├── 2016.07.21
│   │   │   ├── apache.log
│   │   │   └── error.log
│   │   └── 2016.07.22
│   │       ├── apache.log
│   │       └── error.log
│   ├── core00002
│   │   ├── 2016.07.21
│   │   │   ├── apache.log
│   │   │   └── error.log
│   │   └── 2016.07.22
│   │       ├── apache.log
│   │       └── error.log
│   ├── dashboard001
│   │   ├── 2016.07.21
│   │   │   ├── dash.log
│   │   │   └── error.log
│   │   └── 2016.07.22
│   │       ├── dash.log
│   │       └── error.log
│   └── dashboard002
│       ├── 2016.07.21
│       │   ├── dash.log
│       │   └── error.log
│       └── 2016.07.22
│           ├── dash.log
│           └── error.log
└── servers
    ├── server01
    │   ├── 2016.07.21
    │   │   ├── access.log
    │   │   └── system.log
    │   └── 2016.07.22
    │       ├── access.log
    │       └── system.log
    └── server02
        ├── 2016.07.21
        │   ├── access.log
        │   └── system.log
        └── 2016.07.22
            ├── access.log
            └── system.log

With these folders I need to do some sort of log rotation. It's not real log rotation, but I need to zip and delete files after a certain amount of days. I thought of making a config file in which I can put the amount of days after a file needs to be zipped and the amount of days after a file and empty folder needs to be deleted.

The cores and servers can increase and log file names can be added with each server. For this reason I thought of using a config file with the bash script, so I can add names in here.

My first idea of the config file is like this:

Zip;Delete;Main;Sub;App

7;365;/customers;core*;/apache.log
7;365;/customers;core*;/error.log

7;180;/customers;dash*;/dash.log
7;180;/customers;dash*;/error.log

28;365;/servers;server*;/access.log
14;365;/servers;server*;/error.log

My problem lies in that I don't know how to accomplish this without any help.

I already tried to create a loop where the config file will be read, but then only 1 line will be read and the script stops.

#!/bin/bash
configfile="/etc/customlogrotate/logrotation.conf"
logbasefolder="/var/log"

echo " " > log.txt
echo " " >> log.txt
echo "Starting logrotation script" >> log.txt

while IFS=';' read -r daysafterzip daystosave sectionfolder logfilename
do
echo "$daysafterzip $daystosave $logbasefolder$sectionfolder$logfilename" >> log.txt

cd $logbasefolder

done < "$configfile"

Edit: Thanks for the replies. I finally managed to get it working with the $logfile variable.

I now use this script (without the logfile variable, because everything works for now):

#!/bin/bash
# This file is managed by Ansible - Contact SysAdmin for changes!!
# file: roles/syslog-ng/templates/logrotation.sh.j2

configfile="/etc/customlogrotate/logrotation.conf"
logbasefolder="/var/log"

# Move to logbasefolder
cd $logbasefolder

# Loop all options from the configfile
while IFS=';' read -r daystozip daystodelete sectionfolder subsectionfolder logfilename
do

        datetozip=`date -d "$daystozip days ago" +%Y%m%d`
        datetodelete=`date -d "$daystodelete days ago" +%Y%m%d`

        cd $logbasefolder$sectionfolder

        for hostfolder in $subsectionfolder/
        do
                cd $hostfolder

                # loop trough all date folder in hostfolder
                for datefolder in [0-9][0-9][0-9][0-9].[0-9][0-9].[0-9][0-9]
                do
                        # remove dashes from datefolder
                        datefoldershort=`echo "$datefolder" | tr -d .`

                        # check if logfile is to be deleted based on date
                        if test "$datefoldershort" -lt "$datetodelete"
                        then
                                cd $datefolder

                                # search folder for logfilename and delete it
                                for filetodelete in $logfilename.gz
                                do
                                        #Put script for deletion here    
                                done

                                # move back one folder to continue the list
                                cd ..

                        # check if logfile is to be zipped based on date
                        elif test "$datefoldershort" -lt "$datetozip"
                        then
                                cd $datefolder
                                # search folder for logfilename and zip it
                                for filetozip in $logfilename
                                do
                                        gzip ${logfilename:1}
                                done
                               # move back one folder to continue the list
                                cd ..
                        fi
                done

                # move back one folder to continue the list
                cd ..
        done
done < "$configfile"

With this configfile:

11;14;/customers;core*;/apache.log
10;13;/customers;core*;/custom.log
9;12;/customers;core*;/system.log
11;14;/customers;srv*;/error.log
8;9;/customers;srv*;/system.log

Thanks again!!

0

The script is working as you expect, but it changes the directory where it writes log.txt.

Initially, the script writes to log.txt in the current directory, but the script changes its directory. The cd $logbasefolder command changes the current directory so the script begins writing log.txt in one directory, but after the cd command, it writes log.txt to $logbasefolder instead.

Fix the problem by using a variable for writeable path to log.txt ($PWD or $HOME are possibilities), or use pushd and popd to swap directories between the initial directory and $logbasefolder. For example:

#!/bin/bash
configfile="/etc/customlogrotate/logrotation.conf"
logbasefolder="/var/log"
logfile="$PWD/log.txt"

echo " " > $logfile
echo " " >> $logfile
echo "Starting logrotation script" >> $logfile

while IFS=';' read -r daysafterzip daystosave sectionfolder logfilename
do

echo "$daysafterzip $daystosave $logbasefolder$sectionfolder$logfilename" >> $logfile

# alternative method to $logfile variable:
# pushd $logbasefolder > /dev/null
#  zip and delete files ...
# popd > /dev/null
cd $logbasefolder   # remove with pushd and popd

done < "$configfile"
0

I think that you might have to go about this a little differently (but I might be wrong). From what I can tell you might need something you can automate (i.e. use something like crontab). That means:

  • either, you write a script which traverses all folders below "/var/log/", then reads the name of each file, checks the archiving schedule for that file type, then checks when this particular file has been archived last, and proceed accordingly;
  • or, you make your life a little easier by creating a number of scripts dedicated to a specific purpose, and set up cronjobs for each.

Personally I'd much prefer the second option. It's much more modular and once you have put it in place it's probably also easier to maintain. The whole thing would then look something like this:

  • Script1: zip all customers/*/*log ; set cron to run this every 7 days
  • Script2: zip all server/*/access.log ; set cron to run every 28 days
  • Script3: zip all server/*/error.log ; cron every 14 days
  • Script4: delete all files which need deleting after 365 days ; set cron accordingly
  • Script5: delete all files which need deleting after 180 days; set cron accordingly

Sounds more tedious and complicated than it should be to actually set up, is modular, easy to maintain and adjust.

Hope this helps. JW

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.