I always use wc-l command to count number of lines. But when my files(900 mill) are big, i have to wait at least 5 minutes to see the results. Any better ideas ?
1 Answer
In theory you could take the first N lines (where N is a number you determine by experiment), average their length, then divide the filesize by the average length. This will give you a very crude approximation (which will be more accurate but slower the higher N is) of the actual number of lines.
-
Nice, guess we were thinking along the same lines :)– RichthofenDec 25, 2012 at 17:04
-
Take ~1500 lines starting from a truly random line, assuming the line length fits a normal distribution (which may not be a good assumption), then you have a 95% chance that the mean length of those lines represents the actual mean length. ~1500 would constitute statistically valid sample. So dividing (filesize /mean record length) would render pretty good estimate. This is way more trouble than wc -l. Your real problem is that your wc -l is I/O bound, and even with a 15000 rpm SATA drive or a really good SAN ~99% of elapsed time will be I/O wait. Dec 26, 2012 at 1:56
-
I would imagine an SSD would fare better? What kind of performance would be you able to expect from one? Dec 26, 2012 at 2:08
-
"would be you"... right words, not necessarily the right order! Dec 26, 2012 at 2:08
-
SSD's are really expensive per GB of storage and are more effective in a SAN environment where software tiering is active. A hypothetical 900 million line file with 128 byte records (avg) would use 11.5GB, and a 128GB OCZ Vertex 4 costs $US140 at newegg. Storing one file would use $12.57 of storage, filesystem overhead excluded. That is crazy. IMO - creating gigantic files is frequently ill-advised, a poor use of a resource, and is always expensive. An SSD would provide at least a factor of 10 speed up on a full file read. Dec 26, 2012 at 3:20
wc -l
does.