0

How can I hash/checksum any sequential download, without having to store the whole file locally? It needs to work with any kind of sequential download, so "basic HTTP download" specific methods like httpdirfs, piping curl to hash untilities and python scripts using http + checksum libraries, aren't any good. It needs to work with file hosters that require an interactive browser session or propriety download software, for instance.

Maybe there is a FUSE filesystem (or similar) that will hash/checksum files put into it, but not actually store the files, but makes it appear to the program that is writing into the filesystem that the files are stored (assuming the programs don't try and read the file back)? Or maybe people can think of some other option(s)?

Simply writing to tmpfs wouldn't work as some files will be bigger than my available RAM.

Improve this question
2
  • "Any kind of download" is too broad. A process may download file parts in any order (see: torrents). If the first chunk happens to be downloaded last then with most hashing algorithms there is no way to even start until you have the whole file.
    – Kamil Maciorowski
    Jan 21, 2022 at 22:08
  • Good point. I will change my question to say "any kind of sequential download".
    – Hiccup
    Jan 22, 2022 at 17:14

2 Answers 2

Reset to default
1

This is not possible. In order to run an a file level operation, such as a hash or checksum, on a remote file, you need access to the remote file. You would either need remote access to the hosting filesystem or download the file locally.

Improve this answer
4
  • 1
    is this strictly true? for example, cksum can operate on streams, and one can use curl with -s and a pipe cksum or sha etc. never tired it, but see for example: stackoverflow.com/questions/6801704/checksum-remote-file
    – Yorik
    Jan 21, 2022 at 19:00
  • Yes, Yorik has the right idea. I just want to find a way to do this that isn't dependant on the download tool being able to "pipe" that output, like curl can. Things like megacmd, file hosters that only work with user interaction etc - you cannot pipe them like you can with curl.
    – Hiccup
    Jan 21, 2022 at 20:51
  • 1
    this still downloads the file, so it makes no sense.
    – Keltari
    Jan 21, 2022 at 22:24
  • 1
    The OP did not exclude downloading the file, only storing it.
    – Jörg W Mittag
    Jan 22, 2022 at 10:16
0

The fact that you need it to work with any kind of download means that it is impossible: some download protocols download fragments out-of-order, which means in the worst case, they might download the first fragment last, and in that case, you need to store the whole file, which you, however, disallow in your question.

Improve this answer
1
  • Good point. I will change my question to say "any kind of sequential download".
    – Hiccup
    Jan 22, 2022 at 17:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .