Jul 29 2014

Hard Drive Failure vs. UPS Failure

When is a hard drive failure NOT a hard drive failure?  When it’s a bad UPS battery that is dying.  For the past week and a half I have noticed that my VMware ESXi server which hosts three systems for me (2 Microsoft Windows Server 2008R2 systems and an Ubuntu Linux server) was complaining about a corrupt datastore (specifically the boot disk).  While the VMware support site didn’t provide much information on the specific error that I was seeing I felt that it pointed to a hard drive that had bad sectors on it and was on its last legs (mind you, this drive is NOT that old and certainly doesn’t get a lot of activity).  I thought, “oh great – this is going to be fun to fix!”  I had moved the VMs off the server and was about to order a new disk when I then noticed that my APC SmartUPS 1400 was indicating that the battery in the UPS had gone bad (the old “when it rains it pours” adage came to mind immediately).  I figured the battery was not an issue – I’ll just replace it…it’s under warranty (1 year warranty and I bought the battery in September of 2013).  I called up AtBatt.com and spoke with the customer service representative, told them the problem and they authorized the return.  Given that my VMs were crashing (which I thought was due to the ESXi server having a kernel oops and then restarting) I setup a DHCPd server off of my Cisco PIX 501E firewall, enabled it, got the VMs restarted and then disabled the PIX’s DHCPd process (but did not do a “write mem” on the PIX – so in the saved config the PIX DHCPd was set to enabled).

Yesterday, I suddenly notice that I’m getting an IP address from the range configured in the PIX DHCPd server.  I go in and poke around and discover that the PIX had rebooted at 6:11AM yesterday morning.  On top of that my Cisco AP1200 wireless had also rebooted at 6:11AM, and so did my ESXi server (and the event logs were complaining about a corrupt datastore).  Suddenly it occurred to me that the problem was not in the ESXi server (or the PIX or any other network gear) but rather in the UPS.  The UPS was doing a self-test at 6:11AM, the battery failed and the UPS rebooted itself (thereby interrupting power to my entire network stack).  I quickly replaced the UPS with my other SmartUPS 1400 which is still good and everything has been humming along well since (no problems noticed).

This morning I open up the SmartUPS with the bad battery and to my shock I find that the battery is deformed in shape as can be seen from the pictures below.

photo 2 photo 1 photo 3 photo 4

In essence the battery failed horribly and I am quite lucky that it didn’t explode or start a fire!  It took me 15 minutes and the removal of the UPS cover and pulling the case apart a little bit just to get the battery out.  The battery is an Amstron battery and is manufactured in China.  Suffice it to say I am shipping it back today.  Now, I’m supposed to receive a replacement battery from AtBatt but I will also order one from APC.  I am not willing to risk a fire or a battery explosion to save $80.  It’s just not worth it.

No responses yet

Dec 17 2009

ESXi Struggles

I’ve finally built my new virtual server — the one in which I’m going to consolidate my current machines as virtual machines. The intention is to measure the amount of energy my current systems (consisting of a Sun Ultra 60 — dual 400MHz UltraSPARC II CPUs, 2GB memory, and 2x20GB SCSI drives; a Dell Workstation 610 — dual 700MHz Pentium III CPUs, 768MB memory, and a 20GB IDE drive as well as a 30GB IDE drive; a home-built server with an AMD Athlon 1.2GHz CPU, 512MB memory and a 30GB IDE drive as well as a 9.1GB SCSI drive) and then migrate everything to the virtual machine world and measure the energy used by the VM server.

The VM server consists of the following hardware:

Seasonic SS-500ES 500W power supply
Gigabyte GA-MA790GPT-UD3H
AMD Athlon x64 X4 CPU 630 Propus 2.8GHz (quad-core, 95W)
8GB memory
1 x 160GB 7200RPM SATA drive
1 x 500GB 7200RPM SATA drive

The first idea was to install Windows Server 2008R2 core with Hyper-V on the machine and use that to build the VM images. However, the Athlon x64 X4 CPU is not supported by Hyper-V. So far as I can tell it does use AMD-V technology and I did enable virtualization in the motherboard BIOS but Hyper-V won’t start. So, the fall back was to go with ESXi.

I wanted to use ESXi 4.0 Update 1 however the network interface on the motherboard (consisting of a Realtek 8111/8168 chip) is not supported by ESXi. The only supported network interfaces are gigabit interfaces (which the Realtek is — it just isn’t supported and I didn’t have a supported interface card on hand). So, I figured no problem…I’ll just use ESXi 3.5 Update 4. Well, the Realtek chip is also not supported in ESXi 3.5 — but the PCI 3Com 905TX and an Intel EE Pro 10/100 are. However, SATA drives are not supported — well, not completely. I managed to get the system installed by booting from the CD, switching to the tech support console (hitting ALT-F1) and then logging in using the unsupported login. I then loaded the AHCI driver and restarted the install and ESXi installed nicely. However, booting off the install on the hard drives was a no-go since the AHCI drive wouldn’t load (for reasons I’m not sure of) and the system crashed. Back to square one.

I then noticed that VMware released Update 5 to ESXi 3.5 earlier this month (about two weeks ago). I read the release notes and realized that they had resolved the AHCI/SATA drive issues. I downloaded it, burned it to a CD and tried it. Bingo! It installed without a hitch and booted without a problem. Awesome. Now I’m in the process of building out my VM images.

No responses yet

Nov 17 2009

Cloning Windows Server 2003R2 System Disk

I have a situation that requires to clone a system disk on a Windows SBS 2003 system since the C: drive is running out of space. The issue is that the SBS server is down to about 1.5MB of free disk space (don’t ask how that happened…it’s not pretty) and, while the physical disk is 80GB in overall size, the system partition is only 10GB overall. The only real solution is to clone the system partition to a new disk, then boot the machine with the new disk as a slave and use diskpart to extend the partition. Then, after that’s done, change the jumpers on the two drives so that the new clone is the master drive and the original becomes the slave and boot the machine off the new clone drive. Simple.

I ended up choosing to use Clonezilla since it appeared to be more mature than g4u (whether this is true or not I cannot say for sure…nevertheless, they both appear to do the job). I found it by searching for “open source disk clone” and stumbled across the OSALT site. After downloading the ISO I went into my VM lab (a HP DL380 G3 running VMWare ESXi 3.5) and created a new VM consisting of the same hardware as the source except for a 25GB disk versus a 10GB disk.

I then attached the new 25GB disk to the source system (my Windows Server 2003R2 domain controller for the lab), started the VM, attached the ISO via the VMWare Infrastructure client and booted off the ISO/CD. After about 5 minutes Clonezilla came up and I setup the cloning process — all in all about 10 minutes worth of work at most. I then fired up the cloning process and after an additional 45 minutes the clone was done. The clone was still 10GB large as I didn’t explore whether Clonezilla could resize the partition on the fly while it was cloning it. No big deal…it’s easy to extend the partition under Windows if you have the space on your drive. I rebooted into the Windows OS (with the clone disk still attached) and I resized the partition using diskpart. Once that was completed I shutdown the VM, detached the cloned disk from the source and booted the new VM. After a complaint from Windows Product Activation that the system hardware had substantially changed and having to go through the reactivation process again I had a domain controller with more than double the disk space of the original system.

The next step is to do this to the SBS system.

No responses yet

Dec 09 2008

VMware Server 2.0 — Finally!

Published by under Linux,Virtualization,VMware

Well, VMware put Server 2.0 (the final release) out at the end of October and I have to see…it looks really good. I haven’t had a chance to compare it to Server 2008 Hyper-V yet (as I need to finish building my Hyper-V box first) however I have to say that between Server 2.0 and VMware ESXi 3.5 update 2 VMware has some very nice virtualization products out on the market. Let’s cover Server 2.0 first before we get into ESXi 3.5. Here’s the specs for the Server 2.0 system:

  • HP Proliant DL380
  • Dual 3.06GHz Xeon CPUs
  • Dual 10/100MB interfaces
  • 6 x 36GB 10,000 RPM SCSI disks w/hardware RAID-5
  • 12GB memory
  • O/S: Ubuntu Server 8.04LTS

Previously I was running VMware Server 1.0.6 on this system but now that VMware has released Server 2.0 I wanted to take another look at it. While the download is big (570MB) it certainly is not too bad and the install went flawlessly. The Server 2.0 detected the 1.0.6 software on the machine, uninstalled it (but kept the settings) and then installed Server 2.0 and configured it using the settings from the 1.0.6 version. All in all, within about 10 minutes I was back up and running. I like the look of the new interface as well — reminds me very much of the clean look of ESXi…and the best part is that it runs in IE8 Beta 2 (well, almost). To login I simply point the browser to the host system port 8333 as shown.

Server 2.0 Login

Once you login the amount of information at your fingertips is great.

VMware 2.0 summary page

Selecting a specific machine also gives you an immense amount of detail

Machine information in VMware Server 2.0

You can start machines directly from the summary page or you can start them from the console tab by clicking on the big white arrow

Starting a new VM image

Here’s where IE8 was a bit problematic (and I’m not sure it was IE8 or if it’s also an issue with IE7…I’ll have to test this). I had to do two things to get the console to work under Vista/IE8 Beta 2. First, I had to add the VMware server system into IE’s trusted site list. Second, I also had to be running in compatibility mode. Finally, once those two items were done I was able to install the VMRC plugin and get it to work. I still haven’t tried this under Firefox 3.0 yet and will try that later. Once the machine starts you click in the console window (or at one of the two squares in the console tab in the VMware shown below)

Console options in the Console tab

The option on the left opens up a new VMRC window as a separate window. The box on the right opens the VMRC in full screen mode (which you can then reduce to a regular sized window). Not sure what the benefit of the two options are but it’s there if you need it. Once the machine starts up you can have multiple VMRC windows running just like you have with ESXi.

VM console - Server 2008

VMware Server 2.0 Multiple VMRC windows

All in all, I’m quite impressed with VMware Server 2.0. I plan on playing with it for quite a while and will consider whether I will migrate this host server over to VMware ESXi 3.5. Given the small footprint of ESXi (both in terms of disk as well as memory) it may well be the way to go in the long run. I’ll review my ESXi experience in an upcoming post as well as my Hyper-V experience as well.

No responses yet