up vote 0 down vote favorite
share [g+] share [fb]

How to do thing like dd if=somefile bs=1 skip=1337 count=31337000, but efficiently, not using not 1-byte reads and writes?

The solution is expected:

  1. To be simple (for non-simple I can write some Perl oneliner that will do this)
  2. To support large offsets and lengths (so hacks with block size in dd won't help)

Partial solution (not simple enough, trying the same with length will make it even more complex):

dd if=somefile bs=1000 skip=1 count=31337 | { dd bs=337 count=1 of=/dev/null; rest_of_pipeline; }
# 1337 div 1000 and 1337 mod 1000
link|improve this question
64% accept rate
Are you trying to change the blocksize that dd is using? – cmorse Jan 20 at 21:24
Changed blocksize => changed units for skip and count – Vi. Jan 20 at 23:09
feedback

2 Answers

up vote 0 down vote

Instead of bs=1 use bs=4096 or more.

link|improve this answer
Then it will read from offset 1337*4096 instead of 1337 – Vi. Jan 21 at 6:02
Aha, I see, then it will probably be easier to write a simple Python script, e.g. like in this example stackoverflow.com/questions/1035340/… with f.seek(1337) before using read(MY_CHUNK_SIZE) – ccpizza Jan 21 at 19:03
feedback
up vote 0 down vote

Use one process to ditch all the initial bytes, then a second to read the actual bytes, e.g.:

echo Hello, World\! | ( dd of=/dev/null bs=7 count=1 ; dd bs=5 count=1 )

The second dd can read the input with whatever blocksize you find efficient. Note that this requires an extra process to be spawned; depending on your OS that will incur a cost, but it is probably smaller than having to read the files one-by-one byte (unless you have a very small file, in which case there wouldn't be a problem).

link|improve this answer
Will it work well (i.e. don't hog too much memory) for large offsets and counts? dd if=/dev/sda bs=10000000001 | dd bs=255 count=1 | hd -> "dd: invalid number `10000000001'" – Vi. Jan 21 at 6:03
@Vi. If you want to skip a huge offset then you should do the initial read as a series of "ideally" (depending on your source) sized blocks (16M), then drop a series of lesser size blocks (512) which will be in memory, to "zoom" in on your data, before you drop off an odd partion which doesn't fit the block size (bs=1 below) and then read the block you want. E.g. you want to read 255 bytes from offset 10000000001: dd if=/dev/sda bs=16M skip=596 count=1 | dd bs=512 skip=1522 count=1 | (dd bs=1 count=1 of=/dev/null ; dd bs=255 count=1) – RolKau Feb 4 at 23:30
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.