62
votes
How can I parse extremely large (70+ GB) .txt files?
Choose your tools
It seems Excel is not a suitable tool for what you want to do.
One approach would to be to use a different tool to consolidate or summarise the data. awk, sed, grep or perl might be ...
- 83.1k
33
votes
How can I parse extremely large (70+ GB) .txt files?
OS agnostic answer:
Learn just a tiny bit of Python and you will have a tool to do similar conversions in any way you wish.
Type this into a file, save it as e.g. cvt.py (originally based on code ...
- 9,337
15
votes
How can I parse extremely large (70+ GB) .txt files?
Disclaimer: I have not actually tried this with a 70 GB file, but I have done a couple of GB and above 4 million rows.
The intended workflow for huge files is not to load whole file into a sheet, but ...
- 543
9
votes
How can I parse extremely large (70+ GB) .txt files?
If your data format is well-known (CSV or other character delimited file, JSON, etc.) you can usually find a general-purpose command-line tool to help query it.
xsv is a popular one for comma-...
- 91
7
votes
Is the ".sha256" file format formally defined somewhere? How should it be parsed?
A .sha256 file is a text file generated by the sha256sum program. The purpose of a .sha256 file is to enable one to check the integrity of files using the sha256sum program. Its content is not ...
- 3,247
4
votes
How can I parse extremely large (70+ GB) .txt files?
Lots of good advice from elsewhere about the mechanics of data extraction, however you are going to need some dirty coding skills to do anything useful with it.
Large data sets often contain corrupt ...
- 241
4
votes
Accepted
Excel Does Not Parse Quoted String containing comma in CSV File Correctly
Remove the extraneous spaces and it'll work correctly:
T7,7,"331-8463 C - Ind. HT collar, laser HT insert, crown TR",85,0.37,N,0.00,N
T7,8,"331-8463 C - Ind. HT collar, laser HT insert, crown TR",90,...
- 5,759
4
votes
Automatic parsing of citation text in academic references
At the moment (2017) the most active Open-Source project implementing this seem to be Anystyle Parser (last version 07-2016). It can be used through a web-interface, API, or downloaded as a RubyGem.
...
- 1,651
3
votes
How do I delete BOTH lines of a duplicate pair in a text file in bash?
You can use the -u command-line argument to the uniq utility, which does precisely what you want:
−u Suppress the writing of lines
that are repeated in the
input.
You still need to sort ...
- 3,933
3
votes
Accepted
Machine readable list of files in rar archive
From rar -?: l[t[a],b] List archive contents [technical[all], bare]
rar lb *.rar
- 9,195
2
votes
Accepted
Get first two strings of each filename in a directory as a txt file
Try this from command line:
for /F "tokens=*" %g in (textfile.txt) do @for /F "tokens=1,2" %G in ("%~ng") do @if not "%H"=="" echo(%G %H
or start from scratch
for /F "tokens=*" %g in ('dir /A/B/S') ...
- 13.2k
2
votes
Excel Does Not Parse Quoted String containing comma in CSV File Correctly
Sadly, Steve Rindsberg is correct. Excel will accept double-quote encapsulation only if the field separator is:
,rather than:,space
We can't always control the format of our import files.
If you ...
- 19.4k
2
votes
How to properly unescape strace output
tl;dr – see "conclusion" at the very end.
What happens here is quite interesting.
First of all echo -e acts like this (from man 1 echo):
\0NNN
byte with octal value NNN (1 to 3 digits)
This ...
- 74.3k
2
votes
Parsing the result of a command in a Linux bash script
The concept:
Invoke pactl … with LC_ALL=C set to avoid localized output (you apparently don't need this but in general case people do).
Use egrep to discard irrelevant lines. You want to get pairs of ...
- 74.3k
2
votes
Parsing the result of a command in a Linux bash script
The problem is that variables set in the do loop are not maintained from one pass to the next.
The simple solution is to save the list output to a temporary file, then scan it twice:-
pactl list ...
- 17.5k
2
votes
Accepted
Parsing and reformatting date string using PowerShell
This was essentially the problem addressed by the Idera PowerTip of the Day for October 12, 2017. The solution presented was to use the ParseExact method of the DateTime .NET class. Assuming that you'...
- 4,646
2
votes
How can I parse extremely large (70+ GB) .txt files?
I don't use Macs, but assuming the standard UNIX tools are available, then
Open a terminal
type in cd and drag in the folder containing the files
type in awk -v FS=, -v OFS=, '{ print $3, $2, $5 }' ...
- 1,164
1
vote
Accepted
Is it theoretically possible to create a log parser/analyzer with the quality of manual parsing without use of an AI?
Is it theriotically possible to create a log parser/analyzer with the quality of manual parsing without use of an AI?
Absolutely. All you have to do is grab a copy of the latest edition Cambridge ...
- 74k
1
vote
Docx file from an XML file of docx
A DOCX file is much more complex than one XML files.
It actually contains a series of XML files, organized in several folders.
All the files inside the DOCX are XML files, even those that don't have ...
- 481k
1
vote
How can I parse extremely large (70+ GB) .txt files?
As always you have two ways to choose.
A. Make a fairly complex program (multiple choices of separators, multiple choices of data handling etc.)
B. Use a "Unix toolbox" approach. Learn some ...
- 176
1
vote
Parsing Command Prompt Output
I believe its in part because of your regex you have
.map(value => value.trim().split(/\s+/))
the /\s+/ is grabbing every all white space in the output from what I tested
I wrote up a quick ...
1
vote
Accepted
Deleting list of files with non-standard file names
You could try
Get-Content d:\dupelist3.txt | % {Remove-Item -LiteralPath $_ -WhatIf}
Remove the -WhatIf to actually execute
From Remove-Item
-LiteralPath Specifies a path to one or more ...
- 1,573
1
vote
Accepted
Retrieving Formulas within Excel concatenate function
Try the following User Defined Function:
Public Function raghav(rng As Range) As Variant
Dim s As String
s = Mid(rng(1).Formula, 2)
arr = Split(s, Chr(34))
For Each a In arr
...
- 19.4k
1
vote
Accepted
Write URL's to a text file that match a pattern
If you use Chrome, you can press F12 to bring up the developer tools and then run the following small snippet in the console to write the list of links:
$("a.route").each(function (i, a) {if (a.href....
- 1,638
1
vote
Parsing errors due to special characters in dns txt records
CSV files are simply text files formatted with a delimiter that you might choose. So basically the delimiter is not something that you force in the document you're writing but it's the person that ...
- 5,667
1
vote
Accepted
How to find all websites matching the pattern website.edu/~<some_string>?
If you're using Google, this is actually pretty easy:
site:http://www.cameron.edu/~
... for instance lists various professor pages. (In response to the edited question>>>) Note: there is no ...
- 574
1
vote
Accepted
Return specific matched (sub) value with grep in linux
I would use sed instead of grep, so your filter becomes:
... | sed -n 's%^.* = .* = .*/\(.*\)/.*%\1%pg'
Notes:-
Because we are matching the / character, I have replaced the normal string delimiter ...
- 17.5k
1
vote
Parse a complex string into into separate Cells
I would use the Text to Columns method, but since you asked for a formula, put this in B2 copy over and down:
=TRIM(MID(SUBSTITUTE($A2," ",REPT(" ",999)),(COLUMN(A:A)-1)*999+1,999))
- 23.2k
1
vote
Parse a complex string into into separate Cells
In Excel go to Text to Columns
select Delimited:
select Space:
Clcik Next and you'll have it.
- 17.5k
1
vote
Get first two strings of each filename in a directory as a txt file
Download the free text editor called Notepad++
Open your textfile.
Press CTRL-H to open the find and replace dialog.
At the bottom left, check Regular Expression
In Find: enter ^(.+?[ ].+?)[ ].+$
...
- 61.5k
Only top scored, non community-wiki answers of a minimum length are eligible