1

I have a Hitachi 5K3000 drive (HDS5C3020ALA) 2TB drive in an external enclosure.

Today I installed an application and copied some data to it, and it is now almost completely full: Windows Explorer reports 34GB free out of 1.81TB total space.

For about 10 minutes it continually made a noise which was rather alarming to me. It sounded like the disk was being accessed, it would thrash for about 1 second or two, then pause for about a second, then start up again. It continued even when the drive was no longer being accessed according to Resource Monitor, and indeed even after I removed the device via "Safely Remove Hardware" this continued until I powered it off. I carefully picked the drive up while it was doing this and I could feel a slight vibration as it made the noise. Surely it was the read head moving across a large portion of its range. I also noticed that during the actual thrashing the HDD activity light was not active but it came on during the second-or-so pauses.

And so it went like this for a good 10 minutes, me not being sure if my drive was slowly dying. I installed HD Tune and was slightly relieved to see no SMART warnings. I have no means of backing up this much data: It would never be worth the cost. So if it was going to die, the data would die with it.

And then all of a sudden it just stopped. Blissful silence.

I run Windows 7's disk defragmenter utility, and analysis reports 0% fragmentation on this 98.2% full drive.

So is this the explanation? Modern hard disk firmwares now defragment automatically? Does this mean that disk defrag software has now become completely obsolete?

3
  • Note that a USB drive is different from a directly-attached drive. The USB drive has an on-board processor, and these vary in sophistication. To one degree or another they manage the disk and present a high-level view of the disk to the OS. (However, I've never seen a good description of how this "high-level view" is structured.) Mar 7, 2012 at 17:16
  • In this particular instance I had the enclosure connected to my computer via eSATA.
    – Steven Lu
    Mar 8, 2012 at 10:13

5 Answers 5

6

No, they don't do that, though Windows runs defrag by default from time to time automatically.

Your HDD activity can come from various sources, OS indexing your files for search on the new HDD, anti-virus software doing some random scanning etc...

1
  • AV program access would show up in resource manager. I'm sure anything else the OS is doing with it would stop once I "Remove Device" and make it ready to unplug. But this activity continued even in that state.
    – Steven Lu
    Mar 7, 2012 at 16:06
3

They do not defragment automatically. Hard drives are ignorant of any arrangement of the data other than a given LBA = 512 or 4096 bytes.

Hard drives do have the capabilitiy to conduct self tests for SMART, and I believe they do this periodically - would guess some do this by default.

Other possible explanations:

  • Hard drive firmware bug - you might try seeing if a firmware update is available
  • Possibly some miscommunication or subtle incompatibility between the enclosure chipset and the drive - maybe the drive received some mistakenly received some weird command to do a test or something similar. Entirely possible for some drives to simply "not like" some enclosures.
  • Read about hard drive error recovery - maybe the hard drive encountered some kind of error condition and was unresponsive to the enclosure chipset for too long, and the enclosure chipset is trying to reset the drive.
  • Possibly the heat from the drive has degraded the enclosure electronics.
  • Your drive really is dying.

If you do find a firmware update for your drive, DO NOT try to apply the update over the USB enclosure. Directly connect the drive.

1

Windows 7 has the Disk Defragmenter software on a schedule. By default it occurs once a week. Some drives still need to be defragged. Solid State Drives do not need this and would actually hurt the life of the drive. There is no software in the platter drives that auto defragment.

1
  • I turned defragmentation scheduling off. Note this activity started immediately after I made changes on the drive.
    – Steven Lu
    Mar 7, 2012 at 16:08
0

NTFS nowadays uses space reservations, thus copying big files they are highly likley to avoid fragmentation Also any small (512) allocation will caouse 512k continuous allocation.

Windows defragmenter will not defragment drives with less than 20% free space (10% after reserving MFT)

You can get a better insight with defraggler or jkdefrag.

3
  • I'd like to point out that each answer seems to add more contradictions to the evidence I have provided. On one hand I'm glad my drive isn't dead yet, but something definitely isn't adding up.
    – Steven Lu
    Mar 7, 2012 at 16:43
  • 1
    Defraggler does show me that the files I added on most recently are in fact somewhat fragmented. I guess what the drive was doing may remain a mystery after all.
    – Steven Lu
    Mar 7, 2012 at 16:48
  • Drive does not do anything, it just stores data blocks and then gives back. Most notably it lacks intelligence of what filesystem or even partitions are on it. NTFS MFT gets badly fragmented when use exceeds 12,5% free space (or gets close to that).
    – ZaB
    Mar 8, 2012 at 0:27
0

Hard drives do not defrag themselves.

The answer to "does defragging still matter" is complicated.

Here's the problem. NTFS generally does a better job of avoiding fragmentation than FAT did, AS LONG AS you never use more than half the volume's total space at any one time. Between 25% and 50% free, you can start to see performance drop, especially if files tend to be large and volatile.

Think of NTFS as being kind of like a metaphorical walk-in closet full of boxes.

When the closet is half empty, you can just walk into the closet, set the box on the floor or on top of another box, and leave.

As the closet fills up, you might have to re-stack boxes into higher piles to make room for a bigger box (this is akin to defragging).

Eventually, you'll get to the point where you just can't put any more BIG boxes in the closet, but you can still split the contents of a big box into two or more smaller boxes that fit nicely. This is approximately the point where a NTFS volume has 10-25% available.

Finally, you'll get to the point where there's literally no room for even a single new box. HOWEVER, as good luck would have it, most of the boxes that are already stacked in the closet aren't COMPLETELY full, and some of them are mostly empty. Here's where things get ugly.

Microsoft's defragger will (sort of) go into the metaphorical closet, pull out a couple of mostly-empty boxes, and neatly unpack them onto floor, one item at a time. If it finds any blatantly-related items, like "Hurricane Supplies", it puts them into the same box. Whatever's left gets piled into the remaining boxes, which then go back into the closet.

Here's where things get really ugly. Christmas arrives. You buy lots of decorations, and get lots of stuff you can't throw away because it would hurt your mom's feelings, but you'll never use. Some of it is kind of bulky. But you have no empty boxes, and no space for empty boxes. Worse, your parents are coming for a visit in 2 hours and staying in the room where the closet is, so all you have time to do is scoop stuff up and indiscriminately throw it into boxes that aren't completely full, before piling them all back into the closet. You now have Christmas stuff scattered across 17 boxes, and a mountain of more than a hundred boxes piled 2 or 3 deep from floor to ceiling.

Your metaphorical closet (hard drive) is now very, very full. Worse, it takes forever to get anything out, because you have to unstack boxes and go digging to find anything. Accessing the closet's contents has become very slow and tedious. You KNOW you had an Arduino in there, somewhere! Ah, there it is... now, you just have to find the LCD board...

You now have what could be called a "catastrophically fragmented NTFS volume". Not only is there almost no free space, but you have unrelated items randomly sharing boxes, and multiple parts of one thing spread randomly among multiple other boxes. Worse, every single thing you go to store in the closet multiplies the problem. You're now emptying half the closet and digging through an average of 27 boxes to get anything out. In desperation, you decide to try and "defrag".

If you're running Microsoft's defragger, or a thirdparty defragger that uses the official Windows defragging API, it's going to be like having someone walk into the closet, say "Tsk, what a mess!", and walk away without doing much of anything useful. That's because Microsoft's defragger is kind of stupid, very anal-retentive, and extremely paranoid. It has precise rules governing the way it will sift through boxes and attempt to consolidate them. It will spread things out on the floor exactly ONE way, a single layer deep, and ONLY if it can do it in alphabetical order while maintaining a 3' path to both the door and a window, just in case there's a fire. Even worse, it refuses to dig through boxes and start with the ones that have the most scattered files to consolidate. It starts with box one, neatly arranges its contents on the floor, and continues until the floor (minus the egress paths) are covered. At that point, it stops, consolidates what it can, puts the rest back, and calls it a day.

If you're using a more aggressive thirdparty defragger that runs when Windows ISN'T running, it might do a better job... it won't care about maintaining the egress path to the window and door, and might even willingly stack things up a bit, or dig through boxes one by one and pull out only the things that are ripe for consolidation. But at the end of the day, if your drive/closet is 90% full, it doesn't have much space to work in, and won't make a lot of progress. You might manage to get most of your stuff consolidated into other boxes with mostly related items, but you're still going to have unrelated items sharing boxes, and you're still going to have things that go together split into several boxes.

OK, you give up. Feeling inspired by "Hoarders Weekend" on A&E, you rent a dumpster, sift through the boxes one by one, and throw out half of your stuff. But wait! The closet still looks full! Half the boxes only have one or two items in them, but none of them are truly empty, so the closet is STILL packed two levels deep, floor to ceiling. It's faster to get stuff, because you don't have to dig as hard, but related items are still scattered randomly among multiple boxes. So, you defrag again.

If you're Microsoft's defragger (or official Windows defragging API), you follow the same rules as before... with the same uninspiring results. Maybe one or two boxes out of 13 was rearranged, and the remainder weren't even touched. If you send it back for another round, it might make it through one or two more, but each time you get less and less improvement. So you bring out the heavier artillery -- a thirdparty defragger.

A thirdparty defragger that works offline (when Windows isn't running) might be a little more aggressive than Windows was... but when you're done, you notice that the closet is STILL packed with boxes 2-3 levels deep, from floor to ceiling. WHY?!? Therein lies the rub with NTFS.

NTFS really likes to put things in their own boxes, and it hates to throw perfectly good empty boxes away. When the closet is mostly empty, and it has to spend time folding and taping new boxes, it might pack a few items stored at the same time into one box. But later, when defragging, it'll go to almost any extreme to preserve those boxes, even if it means putting one string of Christmas lights into each box.

Thus, the solution... wait until Windows is asleep, carry the boxes into the living room so the closet is now empty, dump them all out into a big pile in the middle of the floor, and burn the now-empty boxes in the back yard before Windows wakes up (by copying the files to another hard drive, then reformatting the catastrophically-fragmented NTFS volume). NOW, when you tell Windows to re-pack the stuff into the closet, it will follow the original strategy... assemble a new box for each write, but willingly write files saved at the same time into the same box to save assembly time. When everything is done, you're back to having a half-empty closet with room for new boxes.

Anyway, that's the general metaphor for NTFS. Windows likes to spread files out when saving them to NTFS, and it would rather move things into existing empty boxes than consolidate the empty boxes into contiguous free space.

So, what about the holy grail -- SSDs? Can they become noticeably slowed down by catastrophic fragmentation? The answer is, "of course". Will defragging help? Um... er... well... maybe, but you should really look at other options instead.

Remember, the memory blocks on a SSD have finite lifetimes that are determined mostly by how many times you erase them. The traditional defragging algorithm is absolutely BRUTAL on a SSD, triggering tens or hundreds of thousands of erase cycles. You can literally burn through your drive's entire lifetime erasure budget within a few days if you defrag a couple of times. And ultimately, Windows is going to be as useless and anal-retentive about defragging the SSD as it would have been for a regular drive.

So... what can you do to fix a catastrophically-fragmented SSD? The same thing you'd do for a catastrophically-fragmented hard drive... copy the files to another volume, do a secure erase & repartition the SSD, then copy the files back, so you get the performance benefit of a deep, satisfying defrag, without destroying your SSD and burning through its lifetime erase budget within a matter of days or weeks.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .