9

I have about 12 GB of image tiles made up of about 2 million files. I'd like to zip these up to make transferring them to a server simpler. I just plan on storing the files in the zip files for transferring, no compression. Helm is present on the web server and can handle unzipping files.

I'd like to point a program at all these files in one go and get it to zip them up into files of approx 1 GB each, but each zip file needs to be independent of the others.

I have 7-zip installed with supports splitting across volumes, but these volumes are dependent upon one another to be unzipped.

Anyone have any suggestions? Thanks in advance!

3
  • 2
    What operating systems are we working with here?
    – Nitrodist
    Jul 27, 2010 at 13:01
  • Sorry, forgot about that - Windows 7 is the OS.
    – Gavin
    Jul 27, 2010 at 22:30
  • If the other end runs Linux or similar, you may want to look at rsync with compression. Oct 3, 2020 at 13:20

7 Answers 7

4

The freeware on Windows called "Spinzip" should do the work for your purpose ! ;) http://skwire.dcmembers.com/wb/pages/software/spinzip.php

It is based on IZARCC (automatically included in Spinzip). You have to check but the full original path may be kept in the zipped files !

See ya

3

I'm not aware of a program that can do that, since if you are making one zip in multi-volumes they will all be related. Your best bet may be to make 12 folders and put a GB in each one, then zip the folders individually.

3
  • Any nice methods of automatically spitting the files in to approx 1 GB chunks while preserving folder structure files live under? Bit tedious to do manually with 2 million files.
    – Gavin
    Jul 27, 2010 at 22:43
  • Try this program on cnet: download.cnet.com/File-Splitter-Deluxe/… I havent used it myself but it should fit the bill for a one time operation.
    – JNK
    Jul 28, 2010 at 1:05
  • Just read the File Splitter Deluxe blurb - seems it needs a small program present to rejoin files. I need to load independent zip files up to server and use Helm interface to extract them from zips, so won't work in this case by sounds of it. Cheers for the suggestion though.
    – Gavin
    Jul 28, 2010 at 9:40
3

In the end I created a quick python script to split the files in to sub directories for me before zipping each individually.

In case it's useful to anyone else, here's my script:

import os
import csv
import shutil

def SplitFilesIntoGroups(dirsrc, dirdest, bytesperdir):
    dirno = 1
    isdircreated = False
    bytesprocessed = 0

    for file in os.listdir(dirsrc):
        filebytes = os.path.getsize(dirsrc+'\\'+file)

        #start new dir?
        if bytesprocessed+filebytes > bytesperdir:
            dirno += 1
            bytesprocessed = 0
            isdircreated = False

        #create dir?
        if isdircreated == False:
            os.makedirs(dirdest+'\\'+str(dirno))
            isdircreated = True

        #copy file
        shutil.copy2(dirsrc+'\\'+file, dirdest+'\\'+str(dirno)+'\\'+file)
        bytesprocessed += filebytes

def Main():
    dirsrc='C:\\Files'
    dirdest='C:\\Grouped Files'

    #1,024,000,000 = approx 1gb
    #512,000,000 = approx 500mb
    SplitFilesIntoGroups(dirsrc, dirdest, 512000000) 

if __name__ == "__main__":
    Main()
2
1

OK here is a way out of it, but not all that good. You can try if you really need.

Assumptions: You need to divide in 12GB of data into 3 4GB DVDs.

Solution

  1. Take 3 empty pen drives of size 4GB.
  2. Insert pen drive #1.
  3. Now start copying your whole of the folder to pen drive #1 using a tool like TeraCopy which pauses on some error and doesn't terminate.
  4. When pen drive #1 is full, it will pause and by the time you can change to pen drive #2.
  5. Now resume after pen drive #2 is loaded.
  6. Repeat from step 4 when pen drive #2 is full. Just increase the pen drive #.

Now you have your data divided. Write them to DVDs or whatever you want to write on.

If you don't have 3 pen drives, you can write the first DVD there itself and then delete whole of the data on pen drive before you resume the copy process.

7
  • I like your creative thinking - I'll give it a go!
    – Gavin
    Jul 27, 2010 at 22:32
  • With one pen drive I guess I can probably copy files off to dir on computer and delete from pen drive, then click resume on TeraCopy? I'm thinking TeraCopy might get upset if detects a different physical device as the destination. I'll have a play tonight anyway.
    – Gavin
    Jul 27, 2010 at 22:45
  • There shouldn't be any problem to TeraCopy unless it starts the new Pen Drive on a different drive name. For eg. pen drive #1 was on G: and if pen drive # 2 is also on G:, it shouldn't be a problem for TeraCopy.
    – IsmailS
    Jul 28, 2010 at 9:12
  • If you have Windows 7 as OS, I think you should not need TeraCopy for it. Not sure though. May be someone can confirm this.
    – IsmailS
    Jul 28, 2010 at 9:15
  • Just been trying with TeraCopy - doesn't work... keeps re-copying the files from the start again - must notice they've gone missing :(
    – Gavin
    Jul 28, 2010 at 9:24
1

Replying after 10 years and 8 months later 💐

Just wrote my own tool for exact necessity.

https://github.com/hamza02x/zip-split-independent

$ zip-split-independent -h

  -d string
        the directory which need to be zipped
  -o string
        output zip directory (not zip file) (default "zip-splits")
  -s float
        split size (in MB) (default 1)
0

We now use a free program called DirectorySlicer. It makes "copies" (uses "hardlinks" if destination is the same drive, so it doesn't use up more drive space) the files into folders of a specified size. This helps us create folders of files that will fit a 700MB CD.

NOTED DOWNSIDE: the files aren't necessarily in the same order; meaning the sequenced filenames (like photo images) might be spread across "chunks" to fit better

You can then create ZIP files of each folder.


Take a look at http://bkprograms.weebly.com/folder-axe.html it is something we've used in the past to split a batch of files into smaller chunks to fit onto 4.7GB DVDs.

0

SpinZip is the rigth tool for no compression. I wanted to use compression, so the result was unsatisfactory. Zipsplit does not work for files above 2GB, so I ended up to write my own quick and dirty perl script, which does its work. Its adds files to the archive as long as the file + archive is lower than the max. specified size:

# Use strict Variable declaration
use strict;
use warnings;
use File::Find;

# use constant MAXSIZE    => 4700372992; # DVD File size
use constant MAXSIZE    => 1566790997; # File size for DVD to keep below 2GB limit
# use constant MAXSIZE    => 100000000; # Test

use constant ROOTDIR    => 'x:/dir_to_be_zipped'; # to be zipped directory

my $zipfilename    = "backup"; # Zip file name
my $zipfileext    = "zip"; # extension

my $counter = 0;
my $zipsize = undef;
my $flushed = 1;

my $arr = [];

find({wanted =>\&wanted, no_chdir => 1}, ROOTDIR);

flush(@{$arr});

# Callback function of FIND
sub wanted {
    my $filesize = (-s $File::Find::name);

    LABEL: {
        if ($flushed) {
            $zipsize = (-s "$zipfilename$counter.$zipfileext");
            $zipsize = 0 unless defined $zipsize;

            printf("Filesize Zip-File %s: %d\n", 
                "$zipfilename$counter.$zipfileext", $zipsize);

            $flushed = 0;

            if (($zipsize + $filesize) >= MAXSIZE) {
                $counter++;
                $flushed = 1;
                printf("Use next Zip File %d, Filesize old File: %d\n",
                    $counter, ($zipsize + $filesize));
                goto LABEL;
            }
        }
    }

    if ( $zipsize + $filesize  < MAXSIZE ) {
        printf("Adding %s (%d) to Buffer %d (%d)\n",
            $File::Find::name, $filesize, $counter, $zipsize);

        push @{$arr}, $File::Find::name;
        $zipsize += $filesize;
    }
    else {
        printf("Flushing File Buffer\n");
        flush(@{$arr});

        $flushed = 1;
        $arr = [];
        goto LABEL;

    }
}

# Flush File array to zip file
sub flush {

    # open handle to write to STDIN of zip call
    open(my $fh, "|zip -9 $zipfilename$counter.$zipfileext -@")
        or die "cannot open < $zipfilename$counter.$zipfileext: $!";

    printf("Adding %d files\n", scalar(@_));
    print $fh map {$_, "\n"} @_;
    close $fh;
}

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .