Just two days before the closure of their financial year, a private bank contacted us frantically regarding their SAN storage failure. The subject unit was an IBM DS3512 SAN with 12 drives configured in RAID5 using IBM System Storage DS Storage Manager. We took the case on an Emergency Service basis, working on it 24/7 until we arrived at a solution.

The Storage Manager reported the failure of four drives. After our preliminary diagnosis, we confirmed that two had faulty read/write heads, while the other two had SMART errors and lost detection intermittently. The catastrophe halted the bank's day-to-day operations, including ATMs. With financial year closure approaching, they also faced regulatory compliance requirements.

The unit was operating in RAID5 with a hot-spare drive. However, the hot-spare had already been used for data reconstruction when one drive failed sometime earlier — a failure that went unnoticed by the client's IT team. When the second drive failed permanently, the unit lost data access entirely.

Despite all the challenges, we managed to image all the drives overnight using our clean room recovery expertise. We identified the RAID parameters and de-striped the array, excluding the previously failed drive. The SAN had one logical drive using Linux file system housing Oracle Database files, and three VMware LUNs (VMFS) with the remaining data.

The mix of Linux Ext3/4 and VMFS LUNs posed a real challenge during logical recovery. The VMware ESXi also used Thin Provisioning, meaning virtual disks started small and expanded as more space was required — adding further complexity to the reconstruction.

We succeeded in 100% recovery of nearly 5TB of data across multiple VMware LUNs in 48 hours, despite the four-drive failure within the DS3512 SAN. The customer was delighted to have the entire data restored on time, in its original shape and form.