Questions related to the processing of large files, such as opening, moving or copying, searching for text, replacing text or extracting parts, or improving the efficiency of such operations, are to be placed under this tag.

learn more… | top users | synonyms (1)

1
vote
1answer
22 views

Is there a method of pre-pending one text file to another much larger text file without cat?

I have a 330 Gb file and 10 lines of text I'd like to pre-pend to the file. I don't particularly want to write out all 330 Gb again, just for 10 lines of text. Is there some low level command that ...
0
votes
2answers
49 views

rsync on Windows not transferring large files

I'm noticing a strange issue where rsync isn't transferring large (approx >3GB) files. I'm running MSYS rsync and I'm transferring on my LAN from my Windows box to my Linux based NAS over ssh. Most of ...
5
votes
2answers
139 views

How can I open a massive PDF?

I created a massive (7GB) PDF in R, and now every computer/program combination I try and open it with craps out. I'm not even sure why as the entire file fits easily in RAM on more than one of the ...
0
votes
1answer
37 views

MTP move within same partition takes a long time

I have an Android 4.3 device (Samsung Galaxy SIII LTE running stock ROM) that connects to the computer using MTP (Media Transfer Protocol). I wanted to organize some files and I was creating ...
0
votes
0answers
14 views

How to back up large amounts of data? [duplicate]

I have a lot of high res pictures, a lot meaning i have about at least 100gb/month that needs backing up. I wrote blu-rays until now when some of them started failing. I also need to have quick acces ...
1
vote
0answers
43 views

how to ensure large encrypted files are not corrupted over time?

I created a 20GB truecrypt container file. I use it to store videos. I store it on an external drive that is always connected to my Win7 daily PC, and only occasionally mount this container file with ...
0
votes
0answers
299 views

Why would large files not sync with skydrive on Windows 8.1?

I'm having a problem on Windows 8.1 (new Surface 2 RT if that matters). My SkyDrive has a mix of small (10 lines of text) and large (1.5 meg) files, and all these files happily came down to the ...
1
vote
1answer
83 views

How do I quickly parse large ASCII files for matching records?

I have lots of files containing data records, stored as space-delimited ASCII files. Each record is a row containing numeric data, with some columns integer and others floating-point eg. 1 1 5711 4 ...
0
votes
3answers
405 views

extract single file from huge tgz file

I have a huge tar file (about 500G) and I wan't to extract just a single file from it. However, when I run tar -xvf file.tgz path/to/file it seems like it is still loading the whole contents into ...
4
votes
1answer
628 views

How to easily break up a text file into pieces smaller than a threshold?

I have some text files that are 100 to 300 MB in length that I want to view in Emacs, but my Emacs gets into some performance issues opening and traversing such large files. Therefore, I'm looking ...
1
vote
1answer
60 views

Copying a large directory tree crashes my PC

Windows 7. I have been happily using PT replicator, which is a backup program with some good features, but since I got an external 2TB and a 4TB drive, I can't copy a whole directory tree. The PC ...
9
votes
2answers
169 views

How to interrupt emacs when control-g does not work in very large files?

I've run into some kind of O(n^2) or worse behavior when viewing very large (e.g. 128MB) files in Emacs and jumping to the end of the file. I think it may have to do with my coloring algorithm to ...
1
vote
1answer
94 views

Problems with large torrents in uTorrent Server @ Ubuntu Server and NTFS partition

I'm using uTorrent Server build 27079 with Ubuntu 10.10 Server. When I'm trying to download large file (>20 Gb) to NTFS volume, ntfs-3g driver starts throttling (top shows mount.ntfs-3g taking 100% ...
3
votes
2answers
80 views

Large file editing and versioning for non-programmers

As part of a collaborative Creative Commons project, I would like to allow editors to contribute by editing a relatively large text file: ~8 Mb, ~100k lines. Ideally, contributions would be git pull ...
1
vote
1answer
72 views

Excel “Go To Special” Crashes on Large File

I'm trying to find all cells that have constants. I'm use excel 2010 and the file has 206963 rows and 9 columns. They are all filled with constant values. When I search for constants excel crashes. ...
0
votes
0answers
81 views

VirtualBox: files missing after after attempting to move them from the shared folder to trash

I'm running an Ubuntu 12.04 VM on VirtualBox with a Mac host. I had some very large files(140GB+) in the shared folder and I needed to free up some disk space, so I moved the files to the VM trash ...
6
votes
3answers
402 views

How does `wc -l` work?

I have to read a large file and before I start reading it, I need to know the total number of lines in the file (which are in millions). I have implemented a lot of solutions and have found one. But ...
2
votes
0answers
101 views

Distributing Large Dataset (80GB+) Over The Internet [duplicate]

I've seen some similar questions but these are dealing with files under 1GB and the answers generally recommend services such as Dropbox, S3 and Skydrive. This does not appear to be suitable for my ...
1
vote
1answer
123 views

Is there replacement for split in Windows? [duplicate]

I want to split a large file ~9 GB by lines, I know split command in UNIX but is there a similar command for Windows?
2
votes
3answers
79 views

Performing a regex replace on a 900 MB logfile

My beloved TextPad says "out of memory" after a while when trying to perform this operation. Next best (i.e. simplest/fastest) free tool to do this?
2
votes
1answer
560 views

Opening 1GB+ XML file for editing [duplicate]

Running Windows 7 64-Bit w/6GB RAM. Attempting to open a 1GB XML file in an attempt to repair an Excel Spreadsheet. I have tried other tools but all caused memory errors. I even tried sed (Windows ...
0
votes
1answer
357 views

Mount SMB share without 2GB filesize limit in OSX Lion

I want to mount a network attached drive onto OSX Lion with Samba. Usually I do this in Finder with "Go>Connect to Server" and type "smb://192.xxx.xx.xxx" in the Server Adress field, however doing ...
0
votes
2answers
477 views

How can I see the last lines of a large file in Windows? [duplicate]

My servers hung, and after a restart it showed me that my hard drive was full (Windows Server 2008). I looked into the problem and it was a 189 GB Tomcat log file! To find out the cause for the ...
1
vote
2answers
1k views

Large zip files content is corrupted when unziped using windows explore

We have a large zip file (25GB) containing mosty pdfs and png files. (about 150000 files) When we unzip the file using windows explore some of the content is corrupted. Some png files shows content ...
-1
votes
1answer
234 views

Text editor for BIG files? [duplicate]

Possible Duplicate: Is there a text editor for very big files? I have text file (XML) of 40G. How to open it with text editor to see cursor and it's relative position to the beginning of ...
1
vote
4answers
2k views

How to download large files on problematic wifi connections?

I'm working out of a hotel with somewhat poor internet. I need to download a large 5 gig file. It looks like the estimate is about a day. I'm sure this will probably have 2 issues, the connection ...
0
votes
1answer
214 views

gVim crashing while opening 8GB text file

I have an 8GB text file which contains output from a structural analysis program. The output is in the form of lines corresponding to time steps - with each line representing state of a system at a ...
0
votes
1answer
51 views

Gnome: How to open a very very large log file?

I've got a log file about 157Gb (Gnome). It's a very large file and nor gedit nor vim can open it. Alternatively I want only to see the last 20 lines and I can't back it up because of space storage ...
1
vote
1answer
45 views

What's the best way to view lines X through Y of a large file?

I have a very large text file and I want to view, say, lines 2000 through 2010 (with the line numbers included) I know one sort of roundabout way of getting there: sc -l [file] cat -n [file] | tail ...
6
votes
2answers
2k views

How to view large spreadsheet in Windows

What would you recommend to a non-technical Windows user for viewing (no need to edit) large spreadsheets (200MB - 5GB). Format is CSV and I am told Excel chokes.
3
votes
1answer
3k views

Cannot copy files from external hard drive to desktop hard drive in Window 7

I'm trying to copy some old files from one of my external hard-drives to the hard drive of my desktop PC. Some files can not be copied but giving the error like 'Cannot read from source file or disk'. ...
0
votes
2answers
188 views

External HHD won't copy large files

I have Buffalo HD_PXU2 (with password protection) and I'm trying to move a very important file of 90 GB (it's a virtual PC Hard disk and should be taken all) but once I try to copy I get the message ...
1
vote
0answers
347 views

How to convert very large HTML files to Word and avoid document mega-bloat

I need to convert some very large HTML files (about 800KB, PDF equivalent about 30 pages) into MS Word. It's easy up to a point: just open in Word and resave as docx. Problem is, the resulting Word ...
32
votes
9answers
4k views

How to uncompress a 9GB file in Windows FAT32

I have a 2GB RAR file that contains a 9GB video file. I'm using a FAT32 file system. Now I want to unzip that file but after 4GB I get an error due to the FAT32 file size limit. Now I want to know ...
9
votes
1answer
893 views

Why can Vim open large files faster than some other text editors?

When opening large log files (well, not that large, but 80mb is still a lot of text!), I've always used Vim since it loads them almost instantly. Other text editors like Notepad, Notepad++, etc will ...
2
votes
2answers
3k views

Is there a major speed difference between using N WiFi or Cat5e ethernet for large file transfers?

I have a network containing 802.11 N Wi-Fi Devices and gigabit ethernet devices but only using Cat5e wiring. I know that if I used Cat6, the ethernet would be much faster, however if I need to ...
1
vote
1answer
614 views

Seeking web-based FTP client for very large file upload

I have looked around for these for some time... the limits imposed by the web server and/or the dynamic programming environment (e.g. PHP) are far too restrictive for the application I'm working on. ...
8
votes
2answers
2k views

Using 'head' or 'tail' on HUGE text file - 19 GB

I have a problem with viewing chunks of a very large text file. This file, approximately 19 GB, is obviously too big to view by any traditional means. I have tried head 1 and tail 1 (head -n 1 ...
0
votes
1answer
309 views

Working with large files - preventing Out of Memory errors?

I'm on Windows 7 Professional x86. I'm working with some proprietary files that are relatively large in size (~900 MB compressed, 1.5 GB uncompressed). This program is also proprietary and the only ...
0
votes
4answers
365 views

Opening a big file [duplicate]

Possible Duplicate: Is there a text editor for very big files? I have a rather large log file on a Windows Server 2008 R2 computer. It's a couple gigabytes, but notepad, wordpad, ...
1
vote
1answer
1k views

Copy remote files on nfs without round-trip

I have a large amout of data (several GB) on a remote drive, that is transparently mounted via nfs. I'd like to copy these files into a subdirectory of where they are already residing, so everything ...
1
vote
1answer
3k views

Windows tool to convert large text file from UTF-8 to “Windows Unicode” (UTF-16)

I need to view large Unicode text files (current version is 2,379,415,348 bytes) on Windows 7. Normally I prefer UTF-8 but, but after looking on SuperUser it seems the best Windows large file viewer ...
0
votes
2answers
465 views

View at least parts of a huge text file on Windows [duplicate]

Possible Duplicate: Text Editor for very big file - Windows I have a huge text file. Text editors refuse to open it due to it's size. How do I view it, without using a text editor? ...
0
votes
2answers
2k views

How can I split a large video files?

I have a TV set that allows me to plug in a USB hard drive and will play movies in it. It supports a surprisingly wide range of codecs, but as I learned yesterday (the ugly way), while it'll play ...
3
votes
2answers
639 views

How do I randomly sort groups of multiple lines in a multi-GB file?

I've got a multi-GB file that has elements in 4 lines, and I would like to have every 4 lines randomized in another file, this is, keeping each four lines grouped and randomize those sets. Is there an ...
2
votes
2answers
226 views

Storing large amounts of small files into bigger files on Windows

Let's say I have 50 GiB of files that weights around 500 KiB each. My guess is that having, for example, 5 large files of 10 GiB each with the same content archived in them would be better for hard ...
4
votes
4answers
391 views

Transferring large files (1GB+) over internet, securely, one time only [duplicate]

Possible Duplicates: Best method of transferring files over internet? Free way to share large files over the Internet? I have been asked to design a system for transfering large files ...
2
votes
2answers
327 views

converting a 12MB text file with notes to orgmode compatible file?

I have a text file that contains about 200000 lines with basically notes, with the most recent notes on top of the file, and the oldest notes at the end of the file. I want to convert it to a ...
1
vote
2answers
315 views

Read error in large file

I have a MPG file, 2.1 GB. I tried copying it from my windows machine to linux one. Only about 1GB of data got copied (the resulting file plays fine), and then I got a read error. So, on the source ...
0
votes
0answers
353 views

Win 7 - computer locks up when dragging large video files in explorer or closing them in WMP?

I have a library of 720 and 1080p videos of my son on my computer (Q6600, 3gb ram, ATI 2600 xt, up-to-date win 7). The problem lately, is that if I view a video, it plays fine. However, once I close ...