0

I'm using logrotate with the copytruncate option. This works well, creating a sparse file that starts with an increasing number of "virtual" null blocks that take up no space on disk.

The problem is with the copied files: While they take up little room on disk, trying to examine them using less takes forever, since the "virtual" null blocks are expanded to actual nulls. I'd really like to eliminate the initial sparse null blocks from the start of the copied file.

Here's what I know so far: ls -ls and du can tell me how much of the file is "real". And I think dd can be used to make a copy without the leading empty blocks. But I'm having trouble putting it all together into something I can put into the postrotate section of my logrotate.conf file.

I've found methods that use tr or sed to delete the nulls, but that requires expanding the file (making the virtual nulls physical), and over time the file can grow to over a terabyte! I need a more 'surgical' approach that works without expanding the file. It should require only messing with the inodes, since that's where the sparse blocks live (not in the actual allocated area).

Of course, the "real" fix is to make the generating program use SIGHUP to reopen its output file, but that's not possible in this case.

What's the simplest and fastest way to directly remove leading null blocks from a sparse file?


Addendum: Here's how to make your own sparse file to play with:

$ dd if=/dev/zero of=sparse.txt bs=1 count=0 seek=8G
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000226785 s, 0.0 kB/s

$ echo 'Hello, World!' >>sparse.txt

$ ls -ls sparse.txt
4 -rwxrwxrwx 1 me me 8589934606 Nov  6 10:20 sparse.txt

$ ls -lsh sparse.txt 
4.0K -rwxrwxrwx 1 me me 8.1G Nov  6 10:20 sparse.txt

That "huge" file takes up almost no room on disk. Now try less sparse.txt. You'll have to wade through 8G of nulls to get to the characters at the end. Even tail -n 1 sparse.txt takes a good while.

7
  • I've never heard of a program that modifies a file by messing with the inodes. There's no system call for doing this, it would have to be done by modifying the raw disk, and this would only be safe if you unmounted the filesystem first.
    – Barmar
    Nov 6, 2014 at 19:54
  • What you could do is figure out the number of null blocks there are by comparing the file's disk space with its length. Then use dd with the seek=n option to skip over the null blocks. This uses lseek, so it doesn't need to read the virtual blocks.
    – Barmar
    Nov 6, 2014 at 19:56
  • @barmar I tried, but failed: Can you share an example that works with any file? I'm using stat -c "%o %B %b %s" to get the info I think I need.
    – BobC
    Nov 6, 2014 at 21:27
  • This seems to be more complicated than I thought. stat %b sparse.txt is reporting 32 blocks for the 8.1GB file. I think it must be including indirect blocks. If I create a file with 16K or 32K nulls at the beginning, they both say 8 blocks.
    – Barmar
    Nov 6, 2014 at 22:40
  • And both 1M and 2M say 16 blocks.
    – Barmar
    Nov 6, 2014 at 22:41

3 Answers 3

1

First of all I would like to thank both of you.

I did test @Michael's solution using the following logrotate configuration and it works as promised. But we still have the primary log file and sparse file.

  access_logs.conf: |
    /var/log/access_v2.log {
        daily
        missingok
        rotate 7
        compress
        delaycompress
        dateformat -%Y%m%d_%H%M%S
        create
        su root root
        firstaction
            rm -rf /var/log/access_v2.log
            sh /etc/logrotate-truncat/truncat.sh /var/log/access.log >> /var/log/access_v2.log
            > /var/log/access.log
         endscript
    }

Here's another take on this. If the log file in question (here /var/log/access.log) can get recreated if we delete it, then we can explicitly implement what copytrucate does (copy the file and then truncate it) within firstaction section but in copydelete fashion (copy the file and then delete it). The file /var/log/access.log will get created automatically and it will not be a sparse file. Hence, no need to clean the sparse file.

Sample

  access_logs.conf: |
    /var/log/access_v2.log {
        daily
        missingok
        rotate 7
        compress
        delaycompress
        dateformat -%Y%m%d_%H%M%S
        create
        su root root
        firstaction
            rm -rf /var/log/access_v2.log
            cp /var/log/access.log /var/log/access_v2.log
            rm -rf /var/log/access.log
         endscript
    }
0

Here's my first seems-to-work attempt, using stat and dd, that works only for leading-sparse files:

#! /bin/bash
for f in $@; do
  echo -n "$f : "
  fields=( `stat -c "%o %B %b %s" $f` )
  xfer_block_size=${fields[0]}
  alloc_block_size=${fields[1]}
  blocks_alloc=${fields[2]}
  size_bytes=${fields[3]}

  bytes_alloc=$(( $blocks_alloc * $alloc_block_size ))

  alloc_in_xfer_blocks=$(( ($bytes_alloc + ($xfer_block_size - 1))/$xfer_block_size ))
  size_in_xfer_blocks=$(( ($size_bytes + ($xfer_block_size - 1))/$xfer_block_size ))
  null_xfer_blocks=$(( $size_in_xfer_blocks - $alloc_in_xfer_blocks ))
  null_xfer_bytes=$(( $null_xfer_blocks * $xfer_block_size ))
  non_null_bytes=$(( $size_bytes - $null_xfer_bytes ))

  if [ "$non_null_bytes" -gt "0" -a "$non_null_bytes" -lt "$size_bytes" ]; then
    cmd="dd if=$f of=$f.new bs=1 skip=$null_xfer_bytes count=$non_null_bytes"
    echo $cmd
    exec $cmd
  else
    echo "Nothing to do: File is not sparse."
  fi
done

What do you think?

5
  • I just realized that the above script only handles complete blocks of leading nulls. It should be a simple tweak to make it correctly handle any number of initial null bytes.
    – BobC
    Nov 7, 2014 at 15:19
  • Yeah, you just have to round down to a multiple of the block size.
    – Barmar
    Nov 7, 2014 at 16:12
  • Notice the rounding in setting *_in_xfer_blocks. But that still doesn't reveal how many nulls there may be in the first physical block, since the file may start with a partial block of nulls. So the "best" solution may be to create the file as above, then read the first block and count the leading nulls, and copy again to get rid of them.
    – BobC
    Nov 7, 2014 at 21:52
  • Ooooh! Just figured it out. At least in ext3 (and presumably in other filesystems), the inode for a sparse hole has resolution of bytes, so the first byte of the first physical block will also be the first byte past the hole. So the above code may actually be correct under all circumstances. Testing needed!
    – BobC
    Nov 7, 2014 at 21:59
  • I didn't think you were worried too much about a small run of nulls at the beginning; the important thing is not to seek past the run of nulls. I didn't notice the - 1 in the calculations.
    – Barmar
    Nov 7, 2014 at 23:17
0

I created an account here so I could thank @BobC for his answer (and his question.) It was the catalyst I needed to solve our longstanding issue with Solr logs.

I modified BobC's script to optimize it a bit for the logrotate use case (using the $xfer_block_size for ibs, and an arbitrarily large (8M) obs, followed by a tr -d "\000" to eliminate the remaining nulls) and then used it in the firstaction section of my logrotate config.

My solution is slightly hacky, I guess, but it's much better than having to bounce critical production services when an 80+ GB log file threatens to fill up the disk...

This is what I ended up with:

#! /bin/bash
# truncat.sh
# Adapted from @BobC's script http://superuser.com/a/836950/539429
#
# Efficiently cat log files that have been previously truncated.  
# They are sparse -- many null blocks before the interesting content.
# This script skips the null blocks in bulk (except for the last) 
# and then uses tr to filter the remaining nulls.
#
for f in $@; do
  fields=( `stat -c "%o %B %b %s" $f` )
  xfer_block_size=${fields[0]}
  alloc_block_size=${fields[1]}
  blocks_alloc=${fields[2]}
  size_bytes=${fields[3]}

  bytes_alloc=$(( $blocks_alloc * $alloc_block_size ))

  alloc_in_xfer_blocks=$(( ($bytes_alloc + ($xfer_block_size - 1))/$xfer_block_size ))
  size_in_xfer_blocks=$(( ($size_bytes + ($xfer_block_size - 1))/$xfer_block_size ))
  null_xfer_blocks=$(( $size_in_xfer_blocks - $alloc_in_xfer_blocks ))
  null_xfer_bytes=$(( $null_xfer_blocks * $xfer_block_size ))
  non_null_bytes=$(( $size_bytes - $null_xfer_bytes ))

  if [ "$non_null_bytes" -gt "0" -a "$non_null_bytes" -lt "$size_bytes" ]; then
    cmd="dd if=$f ibs=$xfer_block_size obs=8M skip=$null_xfer_blocks "
    $cmd | tr -d "\000"
  else
    cat $f
  fi
done

Using larger blocks makes dd orders of magnitude faster. dd makes a first cut, then tr trims the rest of the nulls. As a point of reference, for an 87 GiB sparse file (containing 392 MiB data):

# ls -l 2015_10_12-025600113.start.log
-rw-r--r-- 1 solr solr 93153627360 Dec 31 10:34 2015_10_12-025600113.start.log
# du -shx 2015_10_12-025600113.start.log
392M    2015_10_12-025600113.start.log
#
# time truncat.sh 2015_10_12-025600113.start.log > test1
93275+1 records in
45+1 records out
382055799 bytes (382 MB) copied, 1.53881 seconds, 248 MB/s

real    0m1.545s
user    0m0.677s
sys 0m1.076s

# time cp --sparse=always 2015_10_12-025600113.start.log test2

real    1m37.057s
user    0m8.309s
sys 1m18.926s

# ls -l test1 test2
-rw-r--r-- 1 root root   381670701 Dec 31 10:07 test1
-rw-r--r-- 1 root root 93129872210 Dec 31 10:11 test2
# du -shx test1 test2
365M    test1
369M    test2

When I let logrotate process this using copytruncate, it took most of an hour, and resulted in a fully-materialized non-sparse file -- which then took over an hour to gzip.

Here's my final logrotate solution:

/var/log/solr/rotated.start.log {
    rotate 14
    daily
    missingok
    dateext
    compress
    create
    firstaction
        # this actually does the rotation.  At this point we expect 
        # an empty rotated.start.log file.
        rm -f /var/log/solr/rotated.start.log
        # Now, cat the contents of the log file (skipping leading nulls) 
        # onto the new rotated.start.log
        for i in /var/log/solr/20[0-9][0-9]_*.start.log ; do
           /usr/local/bin/truncat.sh $i >> /var/log/solr/rotated.start.log
           > $i  # truncate the real log
        done
     endscript
}

The hacky bit is that when you first set this up, you have to create an empty rotated.start.log file, otherwise logrotate will never pick it up and run the firstaction script.

I did see your logrotate bug ticket for which a fix was released in logrotate 3.9.0. Unfortunately, if I'm reading it correctly, the implemented fix only addresses part of the problem. It correctly copies the sparse log file to create another sparse file. But as you observed, that's not really what we want; we want the copy to exclude all the irrelevant null blocks and retain just the log entries. After the copytruncate, logrotate still has to gzip the file, and gzip does not handle sparse files efficiently (it reads and processes every null byte.)

Our solution is better than the copytruncate fix in logrotate 3.9.x because it results in clean logs that can be easily compressed.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .