51

I wanted to produce a 1 GB random file, so I used following command.

dd if=/dev/urandom of=output bs=1G count=1

But instead every time I launch this command I get a 32 MB file:

<11:58:40>$ dd if=/dev/urandom of=output bs=1G count=1
0+1 records in
0+1 records out
33554431 bytes (34 MB, 32 MiB) copied, 0,288321 s, 116 MB/s

What is wrong?

EDIT:

Thanks to great answers in this topic I came with solution that reads 32 chunks 32 MB large which makes 1GB:

dd if=/dev/urandom of=output bs=32M count=32

Other solution was given that reads 1 GB straight to the memory and then writes to disk. This solution takes a lot of memory so it is not preffered:

dd if=/dev/urandom of=output bs=1G count=1 iflag=fullblock
11
  • 3
    IMHO I don't think there are many valid use cases for dd at all. I'd use head, cat or rsync in its place almost always. And your question if one of the reasons why the alternatives are usually safer. – Bakuriu Dec 28 '18 at 17:13
  • @Bakuriu - also, if you just want to produce a file full of zeroes (or rather you do not care about what is inside it) use truncate. It is much faster. – Konrad Gajewski Dec 29 '18 at 9:09
  • @KonradGajewski FYI truncate tries to make a sparse file (if that matters) – Xen2050 Dec 29 '18 at 20:33
  • 5
    @Bakuriu head cannot do this task without the -c option that isn't in POSIX. I don't know any version of cat which can solve this. rsync is a completely nonstandard utility. That is neither here nr there; skimming through its man page, I don't see how it can solve this problem, either. – Kaz Dec 31 '18 at 1:09
  • Technically, /dev/urandom isn't in POSIX either... – user1686 Dec 31 '18 at 9:47
94

bs, the buffer size, means the size of a single read() call done by dd.

(For example, both bs=1M count=1 and bs=1k count=1k will result in a 1 MiB file, but the first version will do it in a single step, while the second will do it in 1024 small chunks.)

Regular files can be read at nearly any buffer size (as long as that buffer fits in RAM), but devices and "virtual" files often work very close to the individual calls and have some arbitrary restriction of how much data they'll produce per read() call.

For /dev/urandom, this limit is defined in urandom_read() in drivers/char/random.c:

#define ENTROPY_SHIFT 3

static ssize_t
urandom_read(struct file *file, char __user *buf, size_t nbytes, loff_t *ppos)
{
    nbytes = min_t(size_t, nbytes, INT_MAX >> (ENTROPY_SHIFT + 3));
    ...
}

This means that every time the function is called, it will clamp the requested size to 33554431 bytes.

By default, unlike most other tools, dd will not retry after receiving less data than requested – you get the 32 MiB and that's it. (To make it retry automatically, as in Kamil's answer, you'll need to specify iflag=fullblock.)


Note also that "the size of a single read()" means that the whole buffer must fit in memory at once, so massive block sizes also correspond to massive memory usage by dd.

And it's all pointless because you usually won't gain any performance when going above ~16–32 MiB blocks – syscalls aren't the slow part here, the random number generator is.

So for simplicity, just use head -c 1G /dev/urandom > output.

11
  • 7
    "... you usually won't gain any performance when going above ~16–32 MiB blocks" - In my experience, you tend not to gain much, or even lose performance above 64-128 kilobyte. At that point, you're well in the diminishing returns wrt syscall cost, and cache contention starts to play a role. – marcelm Dec 27 '18 at 20:43
  • 3
    @marcelm I've helped architect high performance systems where IO performance would improve as block size increased to 1-2 MB blocks, and in some cases up to 8 MB or so. Per LUN. And as filesystems were constructed using multiple parallel LUNs, to get get best performance meant using multiple threads for IO, each doing 1 MB+ blocks. Sustained IO rates were over 1 GB/sec. And those were all spinning disks, so I can see high-performance arrays of SSDs swallowing or generating data faster and faster as the block size grows to 16 or even 32 MB blocks. Easily. Maybe even larger. – Andrew Henle Dec 28 '18 at 11:01
  • 4
    I'll explicitly note that iflag=fullblock is a GNU extension to the POSIX dd utility. As the question doesn't specify Linux, I think the use of Linux-specific extensions should probably be explicitly noted lest some future reader trying to solve a similar issue on a non-Linux system be confused. – Andrew Henle Dec 28 '18 at 12:37
  • 6
    @AndrewHenle Ah, interesting! I did a quick test with dd on my machine, with block sizes from 1k to 512M. Reading from an Intel 750 SSD, optimal performance (about 1300MiB/s) was achieved at 2MiB blocks, roughly matching your results. Larger block sizes neither helped nor hindered. Reading from /dev/zero, optimal performance (almost 20GiB/s) was at 64KiB and 128KiB blocks; both smaller and larger blocks decreased performance, roughly matching my previous comment. Bottom line: benchmark for your actual situation. And of course, neither of us benchmarked /dev/random :P – marcelm Dec 28 '18 at 23:48
  • 3
    @Xen2050 I did some more quick tests, and it appears dd is faster. A quick strace showed that head uses 8KiB reads, and two 4KiB writes, which is interesting (GNU coreutils 8.26 on Debian 9.6 / Linux 4.8). head speeds are indeed somewhere between dd bs=4k and dd bs=8k. head speeds are down ~40% compared to dd if=/dev/zero bs=64k and down ~25% compared to dd if=/dev/nvme0n1 bs=2M. The reads from /dev/zero are of course more CPU-limited, but for the SSD I/O queing also plays a role. It's a bigger difference than I expected. – marcelm Dec 29 '18 at 15:41
21

dd may read less than ibs (note: bs specifies both ibs and obs), unless iflag=fullblock is specified. 0+1 records in indicates that 0 full blocks and 1 partial block was read. However any full or partial block increases the counter.

I don't know the exact mechanism that makes dd read a block that is less than 1G in this particular case. I guess any block is read to the memory before it's written, so memory management may interfere (but this is only a guess). Edit: this concurrent answer explains the mechanism that makes dd read a block that is less than 1G in this particular case.

Anyway, I don't recommend such large bs. I would use bs=1M count=1024. The most important thing is: without iflag=fullblock any read attempt may read less than ibs (unless ibs=1, I think, this is quite inefficient though).

So if you need to read some exact amount of data, use iflag=fullblock. Note iflag is not required by POSIX, your dd may not support it. According to this answer ibs=1 is probably the only POSIX way to read an exact number of bytes. Of course if you change ibs then you will need to recalculate the count. In your case lowering ibs to 32M or less will probably fix the issue, even without iflag=fullblock.

In my Kubuntu I would fix your command like this:

dd if=/dev/urandom of=output bs=1M count=1024 iflag=fullblock

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.