Large Healthcare Provider Overwrites Patient Data

Large Healthcare Provider Accidentally Overwrites Patient Data

It is standard practice in the industry to run full system backups on a weekly basis and do daily incrementals. Many companies still rely on LTO or similar tapes to do these backups due to their low cost and high capacities. It is best to store the different backups on separate tapes to prevent accidental overwriting. However, if the system administrator is mounting the tapes manually every day, it dramatically increases the chances of loading the wrong tape in the drive and writing over previously valid backup data. This is exactly what happened to a major healthcare provider in Dallas when they overwrote an 83.4GB full system backup with a 181MB incremental. That in itself would not have been disastrous had they caught their mistake immediately. They could have simply created another full back tape.

Unfortunately the mistake was not discovered until they had a failure of one of their business critical RAID 5 systems. After attempting to rectify the RAID 5 issue it was determined they needed to do a full system restore. However, they soon realized that their "full backup" tape did not contain the 83.4GB full backup, but a 181MB incremental. Now they had an unusable RAID member drive and no valid full backup to restore from. This particular production system contained current patient data including MRI and X-Ray scans. Until this patient data could be restored NONE of those patients could receive proper treatment.

The healthcare provider immediately called their IT service partner and requested their help in rectifying this critical situation. Their IT service provider knew exactly who to call: ACE Data Recovery Services. They gave ACE Data Recovery the LTO-3 tape that had previously contained the 85GB full backup and asked them to recovery ANY data they could and to get it done as quickly as possible. Even though it was during the Christmas holiday season, the dedicated engineers at ACE Data Recovery worked around the clock to get this recovery completed successfully.

The challenge was that backup software writes a marker to the tape known as an End of File (EOF), End of Data (EOD), or End of Media (EOM) marker. When the tape drive reads this marker it will not go past it, so there is no way to access any data beyond this marker via normal conventions. Knowing that the drive's firmware would not allow crossing over the EOF marker under normal circumstances, ACE Data Recovery Services used proprietary software to take control over the drive's data block positioning on the tape. Since the EOF is written across all tracks of the tape, and the tape is read in a serpentine fashion, positioning override had to happen reading both forwards and backwards. Since there are 704 tracks on an LTO-3 tape, there are 704 instances where the EOF appears and the data block information must be calculated for each.

ACE Data Recovery was able to extract a raw data stream from beyond each EOF marker and successfully wrote this raw data to another LTO-3 tape. However, the recovery was not complete. This raw data stream had no header information and could not be cataloged or restored using the backup software that wrote it. The stream had to be further processed in order to acquire the actual folders and files originally written to the tape. The tape engineering experts at ACE Data Recovery wrote custom software to find and use each stream to obtain parent folder name, file name, etc., and extracted the raw data putting it back into its correct folder and file structure. Of the original 83.4GB full backup, DRS recovered 81.6GB of usable data! The healthcare network was able to accomplish a successful restore of the RAID 5 system and patient data was once again available to doctors.

ACE Data Recovery is the only firm with the capability to consistently recover data maintaining the correct directory structure and file names from beyond tape EOF markers. So, when you have recovery issues with any media including tapes call the authority: ACE Data Recovery Services.