25

Is there on Linux a program that can calculate how much data a program would produce?

For example, if I would like to take backup of my MySQL database I would usually do

mysqldump > dumpfile.sql

Instead I would like to redirect to /dev/null but calculate how much disk-space would have been used, like

mysqldump | fancy_space_calc_program

Output:

123456789 Bytes would have been used

Note, the MySQL backup is just an example. I'm very well aware of how I could estimate the size before-hand, so please no comments about that.

  • 1
    I don't even think you can really make one; for specific cases yes, but not general usage, because how you can estimate if some app calls some server and downloads data from there - no chance you can estimate such things in foreign apps. So this would be per app - as you write that you already know for MYSQL - no explanation there, but other apps - per app, no general tool could do such prediction correctly. – Drako May 29 '17 at 13:43
  • 1
    I hope you realize that any attempt to make the estimate would require to actually run the program and observe the output while it is send somewhere safe. This is going to be impossible if the program has some sort of irreversible effect on something else so you can ONLY run it once without unintended side-effects. The other problem is that if the program derives its output from a changing input the next run is going to create another (different size) output-file. Last but not least: diskspace <> (bytes of output). And various filesystems have different overheads for bookkeeping. – Tonny May 29 '17 at 18:52
  • 1
    Yes, I'm well aware of that. It's still good enough for me. – fancyPants May 29 '17 at 19:55
  • @Drako You can have a general way of measuring the text output of a program. That does not need to be per app (see e.g. the accepted answer). Whether or not the text output will be reliably identical on subsequent runs is app-specific, but that does not prevent you from measuring the output in a general way. Presumably the OP and anyone else trying to measure output would only do so if the data was meaningful for any given application. – Jon Bentley May 30 '17 at 12:59
  • @JonBentley I never told you can not have it, read more carefully: "as I wrote general prediction is not going to be precise or even close :)" and now imagine that my app after running will check for updates of itself, of plugins, etc and will download x amount of data from i-net and store that on your hdd; how you are going to precisely measure in advance with general tool not knowing anything about my app, how much storage will be needed after running it? Still you can do your best guess with accepted answer and in many cases even be pretty precise. – Drako May 30 '17 at 14:53
37

Taken from https://stackoverflow.com/questions/13418688/use-pipe-with-du-to-compute-size-of-stdin

You can pipe it to wc -c to count the number of bytes that goes through the pipeline.

Of course, this is just the raw bytes, and have nothing to do with sector size etc, so take it with a grain of salt...

  • as I wrote general prediction is not going to be precise or even close :) – Drako May 29 '17 at 13:44
  • 6
    @cat a good implementation of wc will discard data it no longer needs as soon as practical. – Ruslan May 29 '17 at 15:54
  • 2
    @cat I think it's unlikely to be buffered, since you don't need buffering to count lines or characters. GNU coreutils wc on my computer easily handles 40 GB stdin data, with only 8 GB memory. – Frxstrem May 29 '17 at 15:55
  • 8
    @Magnus. I think you missed the wordplay. WC is a British term for what Americans call a bathroom. You're piping the unused data into the WC. – Nic Hartley May 30 '17 at 9:26
  • 3
    @Frxstrem You certainly do need buffering to count lines or characters - as soon as you're no longer working with an isomorphic encoding. Since POSIX.2, wc -c doesn't count characters - it counts bytes. wc -m counts characters. The most obvious difference is in multi-byte characters like in UTF-16 or the Windows \r\n (two bytes in ASCII, but one character). It doesn't necessarily need a lot of buffering most of the time, but Unicode can have an arbitrary amount of bytes to represent a single character; not something you'd see in trusted data, but a possible buffer overflow vector. – Luaan May 30 '17 at 11:37
28

The command pv is perfect for this.

mysqldump | pv -b > /dev/null

I think the above will give you the right command you want, it may need some adjusting such as pv -b | > /dev/null as I can't test right now

-b gives you a value in bytes.

  • 1
    Holy, I forgot about pv as well as wc. Shame on me. I'd like to accept both answers. So, sorry, but Magnus was a little faster and he can use the reputation. – fancyPants May 29 '17 at 13:49
  • Yeah no worries, the wc trick is real nice, not sure why that didn't immediately occur to me tbh. I first went 'bar!' then realised what I meant was pv! :) – djsmiley2k May 29 '17 at 13:57
  • And now you've got me wondering about grabbing the file handle, and checking for a size in /proc somewhere.... – djsmiley2k May 29 '17 at 13:58
  • 2
    I've never heard of pv before.. You learn something new every day :) – Magnus May 29 '17 at 13:58
  • 2
    @Magnus : I think wc is older (part of some older Unix systems), not in as much documentation, and (quite possibly as a result) pv is pre-installed in fewer distributions. Still, nice to know about. See this conceptually beautiful picture which comes from the "pv" ("pipe viewer") program's home page – TOOGAM May 29 '17 at 18:41
0

You can use dd for it, like this cat /dev/zero | dd status=progress of=/dev/null bs=4M.

This provides you with some data during and after the execution about the amount of data passed to it, like:

$ cat /dev/zero | dd status=progress of=/dev/null                                                                                                                              
5371334656 bytes (5.4 GB, 5.0 GiB) copied, 4 s, 1.3 GB/s^C # this is progress data
12271136+0 records in #summary
12271135+0 records out #summary
6282821120 bytes (6.3 GB, 5.9 GiB) copied, 4.66683 s, 1.3 GB/s #summary

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.