43

How to do thing like dd if=somefile bs=1 skip=1337 count=31337000, but efficiently, not using not 1-byte reads and writes?

The solution is expected:

  1. To be simple (for non-simple I can write some Perl oneliner that will do this)
  2. To support large offsets and lengths (so hacks with block size in dd won't help)

Partial solution (not simple enough, trying the same with length will make it even more complex):

dd if=somefile bs=1000 skip=1 count=31337 | { dd bs=337 count=1 of=/dev/null; rest_of_pipeline; }
# 1337 div 1000 and 1337 mod 1000
Improve this question
2
  • Are you trying to change the blocksize that dd is using?
    – cmorse
    Jan 20, 2012 at 21:24
  • Changed blocksize => changed units for skip and count
    – Vi.
    Jan 20, 2012 at 23:09

4 Answers 4

Reset to default
61

This should do it (on gnu dd):

dd if=somefile bs=4096 skip=1337 count=31337000 iflag=skip_bytes,count_bytes

In case you are using seek= as well, you may also consider oflag=seek_bytes.

From info dd:

`count_bytes'
      Interpret the `count=' operand as a byte count, rather than a
      block count, which allows specifying a length that is not a
      multiple of the I/O block size.  This flag can be used only
      with `iflag'.

`skip_bytes'
      Interpret the `skip=' operand as a byte count, rather than a
      block count, which allows specifying an offset that is not a
      multiple of the I/O block size.  This flag can be used only
      with `iflag'.

`seek_bytes'
      Interpret the `seek=' operand as a byte count, rather than a
      block count, which allows specifying an offset that is not a
      multiple of the I/O block size.  This flag can be used only
      with `oflag'.

Ps: I understand this question is old and it seems these flags were implemented after the question was originally asked, but since it's one of the first google results for a related dd search I did, I though it would be nice to update with the new feature.

Note: this answer applies only to GNU dd, used by most Linux distributions, it's part of GNU coreutils package, this feature was introduced on coreutils release 8.16 (2012-03-26, a couple months after the original question was answered).

Note for Mac users: MacOS uses a bsd based variant of unix utils (for licensing reasons mostly), but GNU versions of unix utilities in general have much more active development and usually have much more features. You can install GNU coreutils on Mac with Homebrew: brew install coreutils.

Improve this answer
2
  • 1
    Beware, these options do not exist in the dd on macOS.
    – mxmlnkn
    Sep 5, 2021 at 17:09
  • 1
    @mxmlnkn I mentioned on the top that this answer applies only to GNU dd, it's part of GNU coreutils package, this feature was introduced on coreutils release 8.16 (2012-03-26). MacOS uses a bsd based variant of unix utils. GNU versions of unix utilities in general have much much more features, you can install it on mac with Homebrew: brew install coreutils.
    – Fabiano
    Sep 22, 2021 at 18:48
3

Use one process to ditch all the initial bytes, then a second to read the actual bytes, e.g.:

echo Hello, World\! | ( dd of=/dev/null bs=7 count=1 ; dd bs=5 count=1 )

The second dd can read the input with whatever blocksize you find efficient. Note that this requires an extra process to be spawned; depending on your OS that will incur a cost, but it is probably smaller than having to read the files one-by-one byte (unless you have a very small file, in which case there wouldn't be a problem).

Improve this answer
4
  • Will it work well (i.e. don't hog too much memory) for large offsets and counts? dd if=/dev/sda bs=10000000001 | dd bs=255 count=1 | hd -> "dd: invalid number `10000000001'"
    – Vi.
    Jan 21, 2012 at 6:03
  • 1
    @Vi. If you want to skip a huge offset then you should do the initial read as a series of "ideally" (depending on your source) sized blocks (16M), then drop a series of lesser size blocks (512) which will be in memory, to "zoom" in on your data, before you drop off an odd partion which doesn't fit the block size (bs=1 below) and then read the block you want. E.g. you want to read 255 bytes from offset 10000000001: dd if=/dev/sda bs=16M skip=596 count=1 | dd bs=512 skip=1522 count=1 | (dd bs=1 count=1 of=/dev/null ; dd bs=255 count=1)
    – RolKau
    Feb 4, 2012 at 23:30
  • 1
    Surely it would be easier to use read -n to skip? And then head -c to count? E.g. cat somefile | (read -n 1337; head -c 31337000) Or you could do it without spawning an extra process: exec 3<somefile; read -n 1337 -u 3; head -c 31337000 <&3
    – Gannet
    Apr 11, 2015 at 3:49
  • @Gannet I kinda like your solution but I really think you should use read -N as the lowercase -n might read less then the requested number of bytes if a newline is encountered! Also I'm not sure whether the read called without a buffer will simply seek to that position or not. If not, it might be introduce a performance problem. Actually: "If no NAMEs are supplied, the line read is stored in the REPLY variable.". You don't wanna store gigabytes of data there, I think.
    – mxmlnkn
    Sep 5, 2021 at 17:19
1

Instead of bs=1 use bs=4096 or more.

Improve this answer
3
  • 3
    Then it will read from offset 1337*4096 instead of 1337
    – Vi.
    Jan 21, 2012 at 6:02
  • 2
    Aha, I see, then it will probably be easier to write a simple Python script, e.g. like in this example stackoverflow.com/questions/1035340/… with f.seek(1337) before using read(MY_CHUNK_SIZE)
    – ccpizza
    Jan 21, 2012 at 19:03
  • It feels like the most reliable way is probably to write a custom executable. Some systems don't have Python, or Ruby, or even Perl. :|
    – Hakanai
    Oct 24, 2018 at 1:27
1

You can try the hexdump command:

hexdump  -v <File Path> -c -n <No of bytes to read> -s <Start Offset>

If you simply want to see the contents :

#/usr/bin/hexdump -v -C mycorefile -n 100 -s 100
00000064 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 
00000074 00 00 00 00 01 00 00 00 05 00 00 00 00 10 03 00 |................| 
00000084 00 00 00 00 00 00 40 00 00 00 00 00 00 00 00 00 |......@.........| 
00000094 00 00 00 00 00 00 00 00 00 00 00 00 00 a0 03 00 |................| 
000000a4 00 00 00 00 00 10 00 00 00 00 00 00 01 00 00 00 |................| 
000000b4 06 00 00 00 00 10 03 00 00 00 00 00 00 90 63 00 |..............c.| 
000000c4 00 00 00 00 |....| 
000000c8 #
Improve this answer
1
  • 1
    It's not about viewing the file as hex. It's about extracting the content of a file (to copy it somewhere, for example) from the specified offset in bytes.
    – Vi.
    Jul 13, 2014 at 23:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .