Jul 29 2014

Hard Drive Failure vs. UPS Failure

Published by at 8:37 am under Home,Technology,Virtualization,VMware

When is a hard drive failure NOT a hard drive failure?  When it’s a bad UPS battery that is dying.  For the past week and a half I have noticed that my VMware ESXi server which hosts three systems for me (2 Microsoft Windows Server 2008R2 systems and an Ubuntu Linux server) was complaining about a corrupt datastore (specifically the boot disk).  While the VMware support site didn’t provide much information on the specific error that I was seeing I felt that it pointed to a hard drive that had bad sectors on it and was on its last legs (mind you, this drive is NOT that old and certainly doesn’t get a lot of activity).  I thought, “oh great – this is going to be fun to fix!”  I had moved the VMs off the server and was about to order a new disk when I then noticed that my APC SmartUPS 1400 was indicating that the battery in the UPS had gone bad (the old “when it rains it pours” adage came to mind immediately).  I figured the battery was not an issue – I’ll just replace it…it’s under warranty (1 year warranty and I bought the battery in September of 2013).  I called up AtBatt.com and spoke with the customer service representative, told them the problem and they authorized the return.  Given that my VMs were crashing (which I thought was due to the ESXi server having a kernel oops and then restarting) I setup a DHCPd server off of my Cisco PIX 501E firewall, enabled it, got the VMs restarted and then disabled the PIX’s DHCPd process (but did not do a “write mem” on the PIX – so in the saved config the PIX DHCPd was set to enabled).

Yesterday, I suddenly notice that I’m getting an IP address from the range configured in the PIX DHCPd server.  I go in and poke around and discover that the PIX had rebooted at 6:11AM yesterday morning.  On top of that my Cisco AP1200 wireless had also rebooted at 6:11AM, and so did my ESXi server (and the event logs were complaining about a corrupt datastore).  Suddenly it occurred to me that the problem was not in the ESXi server (or the PIX or any other network gear) but rather in the UPS.  The UPS was doing a self-test at 6:11AM, the battery failed and the UPS rebooted itself (thereby interrupting power to my entire network stack).  I quickly replaced the UPS with my other SmartUPS 1400 which is still good and everything has been humming along well since (no problems noticed).

This morning I open up the SmartUPS with the bad battery and to my shock I find that the battery is deformed in shape as can be seen from the pictures below.

photo 2 photo 1 photo 3 photo 4

In essence the battery failed horribly and I am quite lucky that it didn’t explode or start a fire!  It took me 15 minutes and the removal of the UPS cover and pulling the case apart a little bit just to get the battery out.  The battery is an Amstron battery and is manufactured in China.  Suffice it to say I am shipping it back today.  Now, I’m supposed to receive a replacement battery from AtBatt but I will also order one from APC.  I am not willing to risk a fire or a battery explosion to save $80.  It’s just not worth it.

No responses yet

Trackback URI | Comments RSS

Leave a Reply