0

I need to transfer a couple of log-files each hour to a logstorage which all users can reach, and I use rsync for do that, but the problem comes when the log-file is rotated and automatically zipped, rsync detects there is a new file and it transfered it too, that means on the destination server I have:

example.log.2014-01-17--00

and

example.log.2014-01-17--00.gz

which is the same logfile but zipped with gzip -9.

this is my rsync line, it is included into a script:

rsync -rvzt --include "*/" --include "$pattern" --exclude "*" $directory loguser@123.456.789.000:/home/loguser/logstorage/$subdir/$basefilename

it is executed each hour by a crontab, and it transfer all files matching a $pattern and $directory included in a file which contains paths and patterns I want to transfer.

So my question is if there is an rsync option that I have not seen, or if there is some posibility to do that and how.

Thanks in advance.

  • @Andryoid - Reading the --include/--exclude sequence: get all subfolders */, only with pattern $pattern, but then exclude everything *? Did I interpret the intent, and if so, that doesn't sound right. Wouldn't it be, get all subfolders */, only with pattern $pattern, but then strip off files with .gz, which would be: --exclude "*.gz". Without def's for $subdir, $basefilename, and example path - pretty much a complete example, that's what I can think of off the top of my head. – Billy McCloskey Jan 17 '14 at 14:33
0

Have you considered using rsyslog forwarding [1]?

Also, from [2], it looks like adding --exclude '*.gz' should fix your issue. Why do you --exclude '*'? My reading of that would be that every file is excluded from the sync.

[1] http://www.rsyslog.com/storing-and-forwarding-remote-messages/

[2] http://www.thegeekstuff.com/2011/01/rsync-exclude-files-and-folders/

0

Okay, I will explain a little bit more about my environment and what I'm looking for:

I have an environment wich so much log-files, those log-files are rotated depending on its weight, so not all log-files are rotated, I have also a lot of different paths where those logfiles resides, and those log-files are pattern-like named.

This is an example of a file containing paths, names, ZIP and DEL times:

# all entries in this file are logfile directories.
# Report: NO # Specify if you want to receive result by mail, options (YES/NO)
# These files will be zipped after <ZIP> days and deleted after <DEL> days
# If <DEL> is not specified, no files will be deleted
# Format: <DIR> <FILEPATTERN> <ageInDaysBeforeBZIP> [<ageInDaysBeforeDELETE>]
# Example: /home/loguser/logstorage       "PATTERN"       <ZIP>    <DEL>
#
/home/user1/www/logs/                     "example.log*"    3        30

In this case the script will read and will take paths into variable $directory, will take pattern to variable $pattern and ZIP & DEL will be used for zip log-files older than 3 days and delete old log-files older than 30 days, but this is another part of the script.

So what I need to do is transfer current log-files hourly basis into destination server but i cannot use --delete, because in destination server I need to keep log-files for 90 days.

So I have created this script which is running hourly basis and transfering matching pattern files to destination server.

On both servers are another cronjob to "keep the room tidy", running once a day and zipping log-files older than 1 day and deleting those what are older than 90 days, but it cannot do its job because if it try to gzip -9 example.log.2014-01-17--12 can't do that, file exists because was zipped on source and transfered to destination zipped.

I'm realizing if I add, or modify --exclude "*.gz" instead of --exclude "*" zipped files will not be transfered, anyway it will be zipped on destination and problem solved I think, but I need to be sure that there are the same files with the same md5sum, because there are so much important log-files.

Woow, I hope your understand and I'm very very grateful for your time and cooperation, I hope this to help some more sysadmin with the same problem or similar.

If you need some more explanation please let me know.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.