1

I want to download a huge file (dataset) which is about 200GB but do not have enough disk space in my internet-connected PC. I want to download a piece of the file (e.g. 30GB chunks) in each run and move the chunks to my server (which has large storage capacity). Then I will eventually merge the chunks to build the whole file. (FYI, the server is network-isolated, I transfer the files using a flash memory.)

How can I download a part of a file (for example, from 10,000,000'th byte to 20,000,000'th byte)?

Any tool/command in linux (preferred) or windows is acknowledged.

Thanks

2
  • How are you going to move the 30 GB chunks to your server? Are these two computers air-gapped? If not, can't you just set up a (temporary) connection so that you can download directly to the server?
    – Jan Doggen
    Apr 15, 2014 at 20:45
  • Yes, the server is isolated by the admin, but I can transfer the files to the server using a flash memory.
    – Ali
    Apr 15, 2014 at 20:48

2 Answers 2

2

You can use cURL for this. Positions are specified in bytes:

curl -r 10000-20000 http://server/file >chunk2

to get a second 10kB chunk.

1
  • While this is probably the best answer to your question, the best solution to your problem would probably be an USB hard drive connected to the internet-connected PC.
    – Squeezy
    Apr 15, 2014 at 21:56
1

In order for you to download a file in such a manner it would be easier if the person providing the file broke it into chunks for you, then you use the tool they broke it up with to put it back together.

In order for you to break it into chunks reliably, you'd need the entire file anyhow, which you cannot get.

1
  • I Know, but they have shared the file in this way! this is MY problem anyway!!!
    – Ali
    Apr 15, 2014 at 20:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .