Questions related to the processing of large files, such as opening, moving or copying, searching for text, replacing text or extracting parts, or improving the efficiency of such operations, are to be placed under this tag.

learn more… | top users | synonyms (1)

1
vote
1answer
17 views

AWS S3 cp does not download 40 GB file

I just uploaded a 40GB backup file from my linux EC2 machine to an S3 bucket. However when I want to download it the aws command ends with no message and nothing was copied. Here is my command: ...
1
vote
0answers
11 views

Backup a big file with binary diff

Edited here is my problem: I have a file A (10GB) and a file B (10GB). File A and B are slightly different (diff is around 500MB). I want to the file A to be exactly as B as fast as possible. One ...
0
votes
2answers
134 views

Fastest way to copy 1TB safely over the wire [duplicate]

I want to transmit a 1TB file over the internet. I have control over both machines (sender unix, receiver linux). I can open FTP, OpenVPN, NFS, ... So far resuming for some reason is not stable on ...
0
votes
0answers
31 views

sending very large amounts data (500GB-1.5TB) over the internet [duplicate]

I am a dutch student and performed research in Brazil. I would like to get my data which adds up to about 500GB (filtered) or 1.5 TB (total) to Holand. I have already send my hard drive but as customs ...
0
votes
1answer
65 views

How large can a VMware datastore be? Name of VMWare software that monitors single server/multiple servers?

I had an interview and was asked the following questions about VMWare. I'm eager to know the answers, please help me : How large can a VMware datastore be? What is the name of the VMware software ...
0
votes
0answers
17 views

How can I print a Photoshop .psb file?

I'm creating a large image with Photoshop (> 30,000) pixels, and it makes me save it in .psb format. Once I do that, however, Photoshop (CS3) is no longer willing to print the image for me. Is this a ...
0
votes
2answers
106 views

rsync on Windows not transferring large files

I'm noticing a strange issue where rsync isn't transferring large (approx >3GB) files. I'm running MSYS rsync and I'm transferring on my LAN from my Windows box to my Linux based NAS over ssh. Most of ...
5
votes
2answers
141 views

How can I open a massive PDF?

I created a massive (7GB) PDF in R, and now every computer/program combination I try and open it with craps out. I'm not even sure why as the entire file fits easily in RAM on more than one of the ...
0
votes
1answer
67 views

MTP move within same partition takes a long time

I have an Android 4.3 device (Samsung Galaxy SIII LTE running stock ROM) that connects to the computer using MTP (Media Transfer Protocol). I wanted to organize some files and I was creating ...
0
votes
0answers
14 views

How to back up large amounts of data? [duplicate]

I have a lot of high res pictures, a lot meaning i have about at least 100gb/month that needs backing up. I wrote blu-rays until now when some of them started failing. I also need to have quick acces ...
1
vote
0answers
54 views

how to ensure large encrypted files are not corrupted over time?

I created a 20GB truecrypt container file. I use it to store videos. I store it on an external drive that is always connected to my Win7 daily PC, and only occasionally mount this container file with ...
0
votes
0answers
369 views

Why would large files not sync with skydrive on Windows 8.1?

I'm having a problem on Windows 8.1 (new Surface 2 RT if that matters). My SkyDrive has a mix of small (10 lines of text) and large (1.5 meg) files, and all these files happily came down to the ...
1
vote
1answer
116 views

How do I quickly parse large ASCII files for matching records?

I have lots of files containing data records, stored as space-delimited ASCII files. Each record is a row containing numeric data, with some columns integer and others floating-point eg. 1 1 5711 4 ...
0
votes
3answers
948 views

extract single file from huge tgz file

I have a huge tar file (about 500G) and I wan't to extract just a single file from it. However, when I run tar -xvf file.tgz path/to/file it seems like it is still loading the whole contents into ...
4
votes
1answer
1k views

How to easily break up a text file into pieces smaller than a threshold?

I have some text files that are 100 to 300 MB in length that I want to view in Emacs, but my Emacs gets into some performance issues opening and traversing such large files. Therefore, I'm looking ...
1
vote
1answer
66 views

Copying a large directory tree crashes my PC

Windows 7. I have been happily using PT replicator, which is a backup program with some good features, but since I got an external 2TB and a 4TB drive, I can't copy a whole directory tree. The PC ...
9
votes
2answers
254 views

How to interrupt emacs when control-g does not work in very large files?

I've run into some kind of O(n^2) or worse behavior when viewing very large (e.g. 128MB) files in Emacs and jumping to the end of the file. I think it may have to do with my coloring algorithm to ...
1
vote
1answer
118 views

Problems with large torrents in uTorrent Server @ Ubuntu Server and NTFS partition

I'm using uTorrent Server build 27079 with Ubuntu 10.10 Server. When I'm trying to download large file (>20 Gb) to NTFS volume, ntfs-3g driver starts throttling (top shows mount.ntfs-3g taking 100% ...
3
votes
3answers
91 views

Large file editing and versioning for non-programmers

As part of a collaborative Creative Commons project, I would like to allow editors to contribute by editing a relatively large text file: ~8 Mb, ~100k lines. Ideally, contributions would be git pull ...
1
vote
1answer
87 views

Excel “Go To Special” Crashes on Large File

I'm trying to find all cells that have constants. I'm use excel 2010 and the file has 206963 rows and 9 columns. They are all filled with constant values. When I search for constants excel crashes. ...
0
votes
0answers
88 views

VirtualBox: files missing after after attempting to move them from the shared folder to trash

I'm running an Ubuntu 12.04 VM on VirtualBox with a Mac host. I had some very large files(140GB+) in the shared folder and I needed to free up some disk space, so I moved the files to the VM trash ...
6
votes
3answers
418 views

How does `wc -l` work?

I have to read a large file and before I start reading it, I need to know the total number of lines in the file (which are in millions). I have implemented a lot of solutions and have found one. But ...
2
votes
0answers
133 views

Distributing Large Dataset (80GB+) Over The Internet [duplicate]

I've seen some similar questions but these are dealing with files under 1GB and the answers generally recommend services such as Dropbox, S3 and Skydrive. This does not appear to be suitable for my ...
1
vote
1answer
138 views

Is there replacement for split in Windows? [duplicate]

I want to split a large file ~9 GB by lines, I know split command in UNIX but is there a similar command for Windows?
2
votes
2answers
83 views

Performing a regex replace on a 900 MB logfile

My beloved TextPad says "out of memory" after a while when trying to perform this operation. Next best (i.e. simplest/fastest) free tool to do this?
2
votes
1answer
735 views

Opening 1GB+ XML file for editing [duplicate]

Running Windows 7 64-Bit w/6GB RAM. Attempting to open a 1GB XML file in an attempt to repair an Excel Spreadsheet. I have tried other tools but all caused memory errors. I even tried sed (Windows ...
0
votes
1answer
434 views

Mount SMB share without 2GB filesize limit in OSX Lion

I want to mount a network attached drive onto OSX Lion with Samba. Usually I do this in Finder with "Go>Connect to Server" and type "smb://192.xxx.xx.xxx" in the Server Adress field, however doing ...
0
votes
2answers
672 views

How can I see the last lines of a large file in Windows? [duplicate]

My servers hung, and after a restart it showed me that my hard drive was full (Windows Server 2008). I looked into the problem and it was a 189 GB Tomcat log file! To find out the cause for the ...
1
vote
2answers
2k views

Large zip files content is corrupted when unziped using windows explore

We have a large zip file (25GB) containing mosty pdfs and png files. (about 150000 files) When we unzip the file using windows explore some of the content is corrupted. Some png files shows content ...
-1
votes
1answer
281 views

Text editor for BIG files? [duplicate]

Possible Duplicate: Is there a text editor for very big files? I have text file (XML) of 40G. How to open it with text editor to see cursor and it's relative position to the beginning of ...
1
vote
4answers
2k views

How to download large files on problematic wifi connections?

I'm working out of a hotel with somewhat poor internet. I need to download a large 5 gig file. It looks like the estimate is about a day. I'm sure this will probably have 2 issues, the connection ...
0
votes
1answer
290 views

gVim crashing while opening 8GB text file

I have an 8GB text file which contains output from a structural analysis program. The output is in the form of lines corresponding to time steps - with each line representing state of a system at a ...
1
vote
1answer
48 views

What's the best way to view lines X through Y of a large file?

I have a very large text file and I want to view, say, lines 2000 through 2010 (with the line numbers included) I know one sort of roundabout way of getting there: sc -l [file] cat -n [file] | tail ...
6
votes
2answers
2k views

How to view large spreadsheet in Windows

What would you recommend to a non-technical Windows user for viewing (no need to edit) large spreadsheets (200MB - 5GB). Format is CSV and I am told Excel chokes.
3
votes
1answer
4k views

Cannot copy files from external hard drive to desktop hard drive in Window 7

I'm trying to copy some old files from one of my external hard-drives to the hard drive of my desktop PC. Some files can not be copied but giving the error like 'Cannot read from source file or disk'. ...
0
votes
2answers
199 views

External HHD won't copy large files

I have Buffalo HD_PXU2 (with password protection) and I'm trying to move a very important file of 90 GB (it's a virtual PC Hard disk and should be taken all) but once I try to copy I get the message ...
1
vote
0answers
403 views

How to convert very large HTML files to Word and avoid document mega-bloat

I need to convert some very large HTML files (about 800KB, PDF equivalent about 30 pages) into MS Word. It's easy up to a point: just open in Word and resave as docx. Problem is, the resulting Word ...
32
votes
9answers
5k views

How to uncompress a 9GB file in Windows FAT32

I have a 2GB RAR file that contains a 9GB video file. I'm using a FAT32 file system. Now I want to unzip that file but after 4GB I get an error due to the FAT32 file size limit. Now I want to know ...
9
votes
1answer
978 views

Why can Vim open large files faster than some other text editors?

When opening large log files (well, not that large, but 80mb is still a lot of text!), I've always used Vim since it loads them almost instantly. Other text editors like Notepad, Notepad++, etc will ...
2
votes
2answers
4k views

Is there a major speed difference between using N WiFi or Cat5e ethernet for large file transfers?

I have a network containing 802.11 N Wi-Fi Devices and gigabit ethernet devices but only using Cat5e wiring. I know that if I used Cat6, the ethernet would be much faster, however if I need to ...
1
vote
1answer
698 views

Seeking web-based FTP client for very large file upload

I have looked around for these for some time... the limits imposed by the web server and/or the dynamic programming environment (e.g. PHP) are far too restrictive for the application I'm working on. ...
8
votes
2answers
2k views

Using 'head' or 'tail' on HUGE text file - 19 GB

I have a problem with viewing chunks of a very large text file. This file, approximately 19 GB, is obviously too big to view by any traditional means. I have tried head 1 and tail 1 (head -n 1 ...
0
votes
1answer
325 views

Working with large files - preventing Out of Memory errors?

I'm on Windows 7 Professional x86. I'm working with some proprietary files that are relatively large in size (~900 MB compressed, 1.5 GB uncompressed). This program is also proprietary and the only ...
0
votes
4answers
443 views

Opening a big file [duplicate]

Possible Duplicate: Is there a text editor for very big files? I have a rather large log file on a Windows Server 2008 R2 computer. It's a couple gigabytes, but notepad, wordpad, ...
1
vote
1answer
1k views

Copy remote files on nfs without round-trip

I have a large amout of data (several GB) on a remote drive, that is transparently mounted via nfs. I'd like to copy these files into a subdirectory of where they are already residing, so everything ...
1
vote
1answer
4k views

Windows tool to convert large text file from UTF-8 to “Windows Unicode” (UTF-16)

I need to view large Unicode text files (current version is 2,379,415,348 bytes) on Windows 7. Normally I prefer UTF-8 but, but after looking on SuperUser it seems the best Windows large file viewer ...
0
votes
2answers
565 views

View at least parts of a huge text file on Windows [duplicate]

Possible Duplicate: Text Editor for very big file - Windows I have a huge text file. Text editors refuse to open it due to it's size. How do I view it, without using a text editor? ...
0
votes
2answers
2k views

How can I split a large video files?

I have a TV set that allows me to plug in a USB hard drive and will play movies in it. It supports a surprisingly wide range of codecs, but as I learned yesterday (the ugly way), while it'll play ...
3
votes
2answers
699 views

How do I randomly sort groups of multiple lines in a multi-GB file?

I've got a multi-GB file that has elements in 4 lines, and I would like to have every 4 lines randomized in another file, this is, keeping each four lines grouped and randomize those sets. Is there an ...
2
votes
2answers
236 views

Storing large amounts of small files into bigger files on Windows

Let's say I have 50 GiB of files that weights around 500 KiB each. My guess is that having, for example, 5 large files of 10 GiB each with the same content archived in them would be better for hard ...