2

If I'm downloading a bunch of torrents, is there a way or a client that detects files already downloaded and stops downloading them?

Edit: Clarifying from the comments. Yes this means finding duplicate files between different torrents and avoiding downloading them.

I know some clients allow you to select which files inside a torrent to download. Is it possible to get a checksum of individual files inside a torrent, before you download them? If so I could make a script or something.

10
  • 1
    If it is the same torrent it will notify when you start downloading if the other one is running, other than that it does not check each torrent that is different for duplicate files
    – Moab
    May 16, 2018 at 21:24
  • 1
    Duplicate files within the torrent would contain the same pieces. If you're talking about deduplication between torrents thats a different matter.
    – jdwolf
    May 16, 2018 at 22:43
  • @jdwolf No, the pieces would have different offsets, so they will not be identical.
    – Encombe
    May 17, 2018 at 12:19
  • @Encombe If they had different "offsets" they wouldn't be the same file now would they?
    – jdwolf
    May 18, 2018 at 1:27
  • Is it possible to get a checksum of individual files inside a torrent before you download them? If so I could make a script or something. May 18, 2018 at 12:03

1 Answer 1

3

It's not possible to determine if a file in a torrent also exist in another torrent from the data available in the .torrent files. There simply is no deduplication support in the standard BitTorrent protocol.

When a torrent is created, all files in it is concatenated together and then chunked up in pieces.

example:
Files  |---#1|#2|---#3|-----------------------#4|#5|------#6|-----#7|-------#8|
Pieces |--0|--1|--2|--3|--4|--5|--6|--7|--8|--9|-10|-11|-12|-13|-14|-15|-16|17|
  • All pieces has the same length except the last one.
  • A file has one or more pieces.
  • A piece may contain (parts from) more than one file.
  • It's very rare that internal file and piece boundaries align (except if padding files is used).
  • A file in a multi-file torrent almost always has a piece shared with another file.

The problem is that it's not possible to create a file hash from the piece hashes.

(@jdwolf Even if file #1 and file #3 is identical, all their piece hashes will be different.)

There is a few torrent creator programs that optionaly can add file hashes, but afaik it's not used anywhere.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.