2

If I'm downloading a bunch of torrents, is there a way or a client that detects files already downloaded and stops downloading them?

Edit: Clarifying from the comments. Yes this means finding duplicate files between different torrents and avoiding downloading them.

I know some clients allow you to select which files inside a torrent to download. Is it possible to get a checksum of individual files inside a torrent, before you download them? If so I could make a script or something.

10
  • 2
    If it is the same torrent it will notify when you start downloading if the other one is running, other than that it does not check each torrent that is different for duplicate files
    – Moab
    May 16, 2018 at 21:24
  • 2
    Duplicate files within the torrent would contain the same pieces. If you're talking about deduplication between torrents thats a different matter.
    – jdwolf
    May 16, 2018 at 22:43
  • 1
    @jdwolf No, the pieces would have different offsets, so they will not be identical.
    – Encombe
    May 17, 2018 at 12:19
  • 1
    @Encombe If they had different "offsets" they wouldn't be the same file now would they?
    – jdwolf
    May 18, 2018 at 1:27
  • 1
    BT checksums (hashes) chunks, not files. May 18, 2018 at 12:07

2 Answers 2

4

It's not possible to determine if a file in a torrent also exist in another torrent from the data available in the .torrent files. There simply is no deduplication support in the standard BitTorrent protocol.

When a torrent is created, all files in it is concatenated together and then chunked up in pieces.

example:
Files  |---#1|#2|---#3|-----------------------#4|#5|------#6|-----#7|-------#8|
Pieces |--0|--1|--2|--3|--4|--5|--6|--7|--8|--9|-10|-11|-12|-13|-14|-15|-16|17|
  • All pieces has the same length except the last one.
  • A file has one or more pieces.
  • A piece may contain (parts from) more than one file.
  • It's very rare that internal file and piece boundaries align (except if padding files is used).
  • A file in a multi-file torrent almost always has a piece shared with another file.

The problem is that it's not possible to create a file hash from the piece hashes.

(@jdwolf Even if file #1 and file #3 is identical, all their piece hashes will be different.)

There is a few torrent creator programs that optionaly can add file hashes, but afaik it's not used anywhere.

1

The first BitTorrent protocol version doesn't offer deduplication as @encombe mentioned.

Version 2 on the other hand comes with embedded file hashes, which are not sha256 flow, but more like it's modified (f.e. with additional padding data).

They called BTMR (BitTorrent Merkle Root) hashes.

Those keys are located in info dictionary inside file tree property, and accessed by pieces root binary encoded field.

There are also tools to extract them.

Example BTMR hash (not decoded):

Image

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .