logrotate tweaks to enhance rsync backups

logrotate, by default (at least on CentOS as of 3, 4, and 5.x), rotates logs by incrementing all numbers on previous logs by one, and moving the current log to log.0

This is all fine and good if you like your logs to all be rotated in order, and have a certain number backed up:

secure –> secure.0.gz –> secure.1.gz –> secure.2.gz …

the next ‘logrotate’ will move secure.0.gz to secure.1.gz, secure.1.gz to secure.2.gz, secure.2.gz to secure.3.gz …

This is very problematic when you use rsync to back up files, and when you include logs in your backups.  The issue is that rsync will see all rotated files as “brand new” files, and thus have to download all of them again and again, wasting time, disk I/O, bandwidth, etc.

To alleviate this issue, you can add ‘dateext’ to /etc/logrotate.conf – which will now back up all logs with a date extension, rather than just an incrementing number.  This causes rotated logs to be static, and rsync will not have to download them again and again and again based on their file name changing and their ‘contents’ differing.

From the logrotate manpage:

Archive old versions of log files adding a daily extension like YYYYMMDD instead of  simply adding a number.

Now your logs look like this:

secure, secure-20091120.gz, secure-20091020.gz, secure-20090920.gz …

the next ‘logrotate’ will only add a new file ‘secure-20091220.gz’ and leave all the others the same.

This means rsync will only have to download a single new file (and the partial “secure” log file which is currently being written to by syslog) rather than having to download the entire list of rotated logs, saving tons of time on backups when you’ve got lots of logs.

Further Reading

A good article about the problems related to “hindsight log rotation schemes”, although with the addition (in 2005?) of the logrotate ‘dateext’ feature, their suggestion to use syslog-ng is not the only good option.  Since a lot of the popular server distros (Redhat/CentOS/Debian) ship with syslogd/rsyslogd and logrotate, most people won’t replace their syslog with syslog-ng just for time-based log rotation.


Leave a Reply

Your email address will not be published. Required fields are marked *