Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have about 12 GB of image tiles made up of about 2 million files. I'd like to zip these up to make transferring them to a server simpler. I just plan on storing the files in the zip files for transferring, no compression. Helm is present on the web server and can handle unzipping files.

I'd like to point a program at all these files in one go and get it to zip them up into files of approx 1 GB each, but each zip file needs to be independent of the others.

I have 7-zip installed with supports splitting across volumes, but these volumes are dependent upon one another to be unzipped.

Anyone have any suggestions? Thanks in advance!

share|improve this question
2  
What operating systems are we working with here? –  Nitrodist Jul 27 '10 at 13:01
    
Sorry, forgot about that - Windows 7 is the OS. –  Gavin Jul 27 '10 at 22:30

6 Answers 6

up vote 3 down vote accepted

The freeware on Windows called "Spinzip" should do the work for your purpose ! ;) http://skwire.dcmembers.com/wb/pages/software/spinzip.php

It is based on IZARCC (automatically included in Spinzip). You have to check but the full original path may be kept in the zipped files !

See ya

share|improve this answer

I'm not aware of a program that can do that, since if you are making one zip in multi-volumes they will all be related. Your best bet may be to make 12 folders and put a GB in each one, then zip the folders individually.

share|improve this answer
    
Any nice methods of automatically spitting the files in to approx 1 GB chunks while preserving folder structure files live under? Bit tedious to do manually with 2 million files. –  Gavin Jul 27 '10 at 22:43
    
Try this program on cnet: download.cnet.com/File-Splitter-Deluxe/… I havent used it myself but it should fit the bill for a one time operation. –  JNK Jul 28 '10 at 1:05
    
Just read the File Splitter Deluxe blurb - seems it needs a small program present to rejoin files. I need to load independent zip files up to server and use Helm interface to extract them from zips, so won't work in this case by sounds of it. Cheers for the suggestion though. –  Gavin Jul 28 '10 at 9:40

OK here is a way out of it, but not all that good. You can try if you really need.

Assumptions: You need to divide in 12GB of data into 3 4GB DVDs.

Solution

  1. Take 3 empty pen drives of size 4GB.
  2. Insert pen drive #1.
  3. Now start copying your whole of the folder to pen drive #1 using a tool like TeraCopy which pauses on some error and doesn't terminate.
  4. When pen drive #1 is full, it will pause and by the time you can change to pen drive #2.
  5. Now resume after pen drive #2 is loaded.
  6. Repeat from step 4 when pen drive #2 is full. Just increase the pen drive #.

Now you have your data divided. Write them to DVDs or whatever you want to write on.

If you don't have 3 pen drives, you can write the first DVD there itself and then delete whole of the data on pen drive before you resume the copy process.

share|improve this answer
    
I like your creative thinking - I'll give it a go! –  Gavin Jul 27 '10 at 22:32
    
With one pen drive I guess I can probably copy files off to dir on computer and delete from pen drive, then click resume on TeraCopy? I'm thinking TeraCopy might get upset if detects a different physical device as the destination. I'll have a play tonight anyway. –  Gavin Jul 27 '10 at 22:45
    
There shouldn't be any problem to TeraCopy unless it starts the new Pen Drive on a different drive name. For eg. pen drive #1 was on G: and if pen drive # 2 is also on G:, it shouldn't be a problem for TeraCopy. –  iSid Jul 28 '10 at 9:12
    
If you have Windows 7 as OS, I think you should not need TeraCopy for it. Not sure though. May be someone can confirm this. –  iSid Jul 28 '10 at 9:15
    
Just been trying with TeraCopy - doesn't work... keeps re-copying the files from the start again - must notice they've gone missing :( –  Gavin Jul 28 '10 at 9:24

In the end I created a quick python script to split the files in to sub directories for me before zipping each individually.

In case it's useful to anyone else, here's my script:

import os
import csv
import shutil

def SplitFilesIntoGroups(dirsrc, dirdest, bytesperdir):
    dirno = 1
    isdircreated = False
    bytesprocessed = 0

    for file in os.listdir(dirsrc):
        filebytes = os.path.getsize(dirsrc+'\\'+file)

        #start new dir?
        if bytesprocessed+filebytes > bytesperdir:
            dirno += 1
            bytesprocessed = 0
            isdircreated = False

        #create dir?
        if isdircreated == False:
            os.makedirs(dirdest+'\\'+str(dirno))
            isdircreated = True

        #copy file
        shutil.copy2(dirsrc+'\\'+file, dirdest+'\\'+str(dirno)+'\\'+file)
        bytesprocessed += filebytes

def Main():
    dirsrc='C:\\Files'
    dirdest='C:\\Grouped Files'

    #1,024,000,000 = approx 1gb
    #512,000,000 = approx 500mb
    SplitFilesIntoGroups(dirsrc, dirdest, 512000000) 

if __name__ == "__main__":
    Main()
share|improve this answer
    
Other software options for handling the file grouping: Burn to the Brim, DVD Span, File Sift, FillCD, Ignition, and SizeMe. –  afrazier Aug 4 '10 at 14:31
    
+ 1 for sharing out things. –  iSid Oct 5 '10 at 4:58

We now use a free program called DirectorySlicer. It makes "copies" (uses "hardlinks" if destination is the same drive, so it doesn't use up more drive space) the files into folders of a specified size. This helps us create folders of files that will fit a 700MB CD.

NOTED DOWNSIDE: the files aren't necessarily in the same order; meaning the sequenced filenames (like photo images) might be spread across "chunks" to fit better

You can then create ZIP files of each folder.


Take a look at http://bkprograms.weebly.com/folder-axe.html it is something we've used in the past to split a batch of files into smaller chunks to fit onto 4.7GB DVDs.

share|improve this answer

SpinZip is the rigth tool for no compression. I wanted to use compression, so the result was unsatisfactory. Zipsplit does not work for files above 2GB, so I ended up to write my own quick and dirty perl script, which does its work. Its adds files to the archive as long as the file + archive is lower than the max. specified size:

# Use strict Variable declaration
use strict;
use warnings;
use File::Find;

# use constant MAXSIZE    => 4700372992; # DVD File size
use constant MAXSIZE    => 1566790997; # File size for DVD to keep below 2GB limit
# use constant MAXSIZE    => 100000000; # Test

use constant ROOTDIR    => 'x:/dir_to_be_zipped'; # to be zipped directory

my $zipfilename    = "backup"; # Zip file name
my $zipfileext    = "zip"; # extension

my $counter = 0;
my $zipsize = undef;
my $flushed = 1;

my $arr = [];

find({wanted =>\&wanted, no_chdir => 1}, ROOTDIR);

flush(@{$arr});

# Callback function of FIND
sub wanted {
    my $filesize = (-s $File::Find::name);

    LABEL: {
        if ($flushed) {
            $zipsize = (-s "$zipfilename$counter.$zipfileext");
            $zipsize = 0 unless defined $zipsize;

            printf("Filesize Zip-File %s: %d\n", 
                "$zipfilename$counter.$zipfileext", $zipsize);

            $flushed = 0;

            if (($zipsize + $filesize) >= MAXSIZE) {
                $counter++;
                $flushed = 1;
                printf("Use next Zip File %d, Filesize old File: %d\n",
                    $counter, ($zipsize + $filesize));
                goto LABEL;
            }
        }
    }

    if ( $zipsize + $filesize  < MAXSIZE ) {
        printf("Adding %s (%d) to Buffer %d (%d)\n",
            $File::Find::name, $filesize, $counter, $zipsize);

        push @{$arr}, $File::Find::name;
        $zipsize += $filesize;
    }
    else {
        printf("Flushing File Buffer\n");
        flush(@{$arr});

        $flushed = 1;
        $arr = [];
        goto LABEL;

    }
}

# Flush File array to zip file
sub flush {

    # open handle to write to STDIN of zip call
    open(my $fh, "|zip -9 $zipfilename$counter.$zipfileext -@")
        or die "cannot open < $zipfilename$counter.$zipfileext: $!";

    printf("Adding %d files\n", scalar(@_));
    print $fh map {$_, "\n"} @_;
    close $fh;
}
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.