RAID Level 5 data redundancy: theory and reality
protects your data. In reality,
is often a painful failure. Why? Mean time-to-data-loss (MTTDL) is a fraud: actual rates of double-disk failures are 2 to 1500 times higher than MTTDL predicts.
What’s behind MTTDL’s failure?
In A Highly Accurate Method for Assessing Reliability of
researchers from the NetApp and the University of Maryland, compared RAID theory against actual field data. They found that MTTDL calculations inaccurate for 3 reasons:
- Errors in statistical theory of repairable systems.
- Incomplete consideration of failure modes.
- Inaccurate time-to-failure distributions.
By repairing MTTDL’s theoretical basis, adding real-world failure
RAID data recovery
and using Monte Carlo simulations they found that today’s MTTDL estimates are wildly optimistic. Which means your data is a lot less safe with
than you know.
The typical MTTDL assumption is that once repaired - i.e. a disk is replaced with a new one and
automatic RAID data recovery
completed - the RAID system is as good as new. But this isn’t true: at best, the system is only slightly better than it was right before
hard drive recovery
One component is new - but the rest are as old and worn as they were before the failure - so the system is not “like new". The system is no less likely to fail after the repair than it was before.
The problem is that in RAID arrays repairs take time: the disk fails; a hot spare or a new disk is added; and the
starts - a process that can take hours or days - while the other components continue to age.
MTTDL calculations use the wrong failure distributions and incorrectly correlate component and system failures.
MTTDL typically considers only catastrophic disk failures and disks have latent errors as well. A catastrophic failure + latent error is a dual-disk failure, something
Anatomy of a RAID failure
There are 4 transition events in a RAID 5 failure:
- Time to operational failure. Drive failure distributions are not constant. Sub-populations of drives may have specific failure modes, like infant mortality, that MTTDL models do not account for.
- Time to restore. Minimum restore times are functions of several variables, including HDD capacity, HDD data rate, data bus bandwidth, number of HDDs on the bus and the on-going I/O load on the array. A 2 TB
might take 40 hours or more to restore.
- Time to latent defect. Latent defect rates vary with usage, age and drive technology.
- Time to scrub. Scrubbing is a background process meant to find and repair latent errors. Busy systems have less time to scrub which increases the chance of a latent error hosing a
RAID 5 rebuild. Scrub strategy has a major impact on latent error rates.
Using field-validated distributions for these 4 transition events and Monte Carlo simulations, the researchers concluded:
The model results show that including time-dependent failure rates and restoration rates along with latent defects yields estimates of dual-disk failures that are as much as 4,000 times greater than the MTTDL-based estimates.
This is why
has caused so much trouble to so many people over the last 20 years.