To properly understand this you first have to understand fragmentation.
Your hard drive is a magnetic disk that can store binary on/off (magnetic/not magnetic) data in every "track" around the disk (imagine a record, it reads out from the center)
When you save a file to disk, the OS finds a spot for it, and saves it there, so for example if you had 3 text files, 1.txt, 2.txt, and 3.txt, they could be on disk as:
[1.txt]==[2.txt]=========[3.txt]
Now, if you wanted to save 4.txt, you'd have to do it between 2 and 3 - there is no room between 1 and 2 (perhaps a file that only took up 2 =s there was deleted, leaving a gap), so your new disk looks like
[1.txt]==[2.txt]===[4.txt]==[3.txt]
Neat, right? Not really, how are you going to save 5.txt?!
[1.txt][5[2.txt].tx[4.txt]t][3.txt]
Eugh! That takes a while to read, doesn't it? It's the same for your hard drive, the read head has to skip over 2.txt AND 4.txt while it's reading it! That's much slower!
Now, most hard drives are big enough to store more than 5 text files, so you have a lot more space to play with, but you're also saving much larger files, so it sort of balances out.
In your context, a backup file is MORE likely to get damaged by disk failure, as it's spread out across the disk, so a failed bit is much more likely to contain a bit of your backup. Defragmenting doesn't change the file as the OS sees it, it just sticks it together on the disk, meaning reading and writing it is much faster.