Archive for the 'Virtualization' Category

Jul 29 2014

Hard Drive Failure vs. UPS Failure

When is a hard drive failure NOT a hard drive failure?  When it’s a bad UPS battery that is dying.  For the past week and a half I have noticed that my VMware ESXi server which hosts three systems for me (2 Microsoft Windows Server 2008R2 systems and an Ubuntu Linux server) was complaining about a corrupt datastore (specifically the boot disk).  While the VMware support site didn’t provide much information on the specific error that I was seeing I felt that it pointed to a hard drive that had bad sectors on it and was on its last legs (mind you, this drive is NOT that old and certainly doesn’t get a lot of activity).  I thought, “oh great – this is going to be fun to fix!”  I had moved the VMs off the server and was about to order a new disk when I then noticed that my APC SmartUPS 1400 was indicating that the battery in the UPS had gone bad (the old “when it rains it pours” adage came to mind immediately).  I figured the battery was not an issue – I’ll just replace it…it’s under warranty (1 year warranty and I bought the battery in September of 2013).  I called up and spoke with the customer service representative, told them the problem and they authorized the return.  Given that my VMs were crashing (which I thought was due to the ESXi server having a kernel oops and then restarting) I setup a DHCPd server off of my Cisco PIX 501E firewall, enabled it, got the VMs restarted and then disabled the PIX’s DHCPd process (but did not do a “write mem” on the PIX – so in the saved config the PIX DHCPd was set to enabled).

Yesterday, I suddenly notice that I’m getting an IP address from the range configured in the PIX DHCPd server.  I go in and poke around and discover that the PIX had rebooted at 6:11AM yesterday morning.  On top of that my Cisco AP1200 wireless had also rebooted at 6:11AM, and so did my ESXi server (and the event logs were complaining about a corrupt datastore).  Suddenly it occurred to me that the problem was not in the ESXi server (or the PIX or any other network gear) but rather in the UPS.  The UPS was doing a self-test at 6:11AM, the battery failed and the UPS rebooted itself (thereby interrupting power to my entire network stack).  I quickly replaced the UPS with my other SmartUPS 1400 which is still good and everything has been humming along well since (no problems noticed).

This morning I open up the SmartUPS with the bad battery and to my shock I find that the battery is deformed in shape as can be seen from the pictures below.

photo 2 photo 1 photo 3 photo 4

In essence the battery failed horribly and I am quite lucky that it didn’t explode or start a fire!  It took me 15 minutes and the removal of the UPS cover and pulling the case apart a little bit just to get the battery out.  The battery is an Amstron battery and is manufactured in China.  Suffice it to say I am shipping it back today.  Now, I’m supposed to receive a replacement battery from AtBatt but I will also order one from APC.  I am not willing to risk a fire or a battery explosion to save $80.  It’s just not worth it.

No responses yet

Aug 16 2013

Building an ARM-based VM for SDL Training

Before I get into the details of this post I want to provide a bit of a background to this project.  I run the Security Engineering Team for a manufacturer.  We build hardware and software products and my responsibilities include integrating a Security Development Lifecycle (SDL) into our product development process.  I’ve been working on this for about two years, slowly integrating a Microsoft-style SDL into each product’s development lifecycle.  Part of this effort involves developer and architect training.  Initially we chose to use a third-party to provide this training to our developers and kicked off that effort in April of last year.  Unfortunately, the initial training did not go well.  We had significant problems which forced us to re-evaluate our approach to the training.  After the problems we had with the initial round of the Class-Based Training (CBT) we decided to move forward with our own internally developed training.  I mentioned this in a previous blog post that I’ve been focused on putting together training on topics like:

  • Threat Modeling
  • Secure Coding in C and C++
  • Secure Coding in C#/.NET

I went about putting the relevant material for these classes and building out both class presentations as well as hands-on lab materials.  The curve ball I was thrown which resulted in my putting together this VM was sent my way when I gave this class at one of our facilities in France.  I always ask for feedback from the attendees and I got very positive feedback from this class.

About a week after the class was over I got the “unofficial” feedback through one of the guys who works in the Security Engineering Team in Europe.  The gist of this feedback was that while they really liked the labs they felt that it would be more realistic if the labs were done on an ARM-based processor VM rather than the Intel x86 based VMs that I was using.   All of our  hardware products utilize an ARM based processor due to a wide-variety of reasons – not least of which is that they’re embedded devices.

On top of that request the developers were more familiar with Windows (they do all of their development either in Eclipse or, in the case of the guys who develop our .NET based applications, Visual Studio) and they wanted more of a windowing environment like Windows (I don’t know of any version of Windows that runs on the ARM processor).  So, between these two requests I had to start looking at building my own ARM based VM – with the caveat is that it has to run under Windows…yeah…fun!

First thing I had to do was find a processor emulator.  That part was pretty easy – QEMU!  QEMU is a fantastic VM tool – while everyone talks about VMware or VirtualBox or Xen – people tend to overlook QEMU.  QEMU not only provides for building out your own VM that is x86 based but it also allows you to build out VMs that use other processor architectures like ARM, PowerPC, Alpha, SPARC, S390, Motorola 68K, and others.  I mean, this is a really cool tool.

I downloaded the latest version of QEMU (at least it was the latest when I downloaded it): 1.5.1 and installed it on my Windows 7 laptop and then did a search for pointers on how to build an ARM based VM using QEMU.  And boy did I find the links – unfortunately they were all expecting that you would build the VM under a Linux system – not a Windows system.  Some of the links I found that were really helpful were:

Between these sites and some tinkering I was finally able to get the ARM based Debian image built.  However, I had such difficulties with building it under Windows 7 that I finally punted and built it on a spare machine I had in the basement (an HP DL380 2U server with 12GB of memory and 216 GB of hard drive space – you’d think that was overkill but believe it or not the VM took almost half a day to finish building – and then it was a matter of getting the development packages installed!).  I’ll post about the whole effort (and the effort of building the VM under Windows in the next few weeks).

One response so far

Dec 17 2009

ESXi Struggles

I’ve finally built my new virtual server — the one in which I’m going to consolidate my current machines as virtual machines. The intention is to measure the amount of energy my current systems (consisting of a Sun Ultra 60 — dual 400MHz UltraSPARC II CPUs, 2GB memory, and 2x20GB SCSI drives; a Dell Workstation 610 — dual 700MHz Pentium III CPUs, 768MB memory, and a 20GB IDE drive as well as a 30GB IDE drive; a home-built server with an AMD Athlon 1.2GHz CPU, 512MB memory and a 30GB IDE drive as well as a 9.1GB SCSI drive) and then migrate everything to the virtual machine world and measure the energy used by the VM server.

The VM server consists of the following hardware:

Seasonic SS-500ES 500W power supply
Gigabyte GA-MA790GPT-UD3H
AMD Athlon x64 X4 CPU 630 Propus 2.8GHz (quad-core, 95W)
8GB memory
1 x 160GB 7200RPM SATA drive
1 x 500GB 7200RPM SATA drive

The first idea was to install Windows Server 2008R2 core with Hyper-V on the machine and use that to build the VM images. However, the Athlon x64 X4 CPU is not supported by Hyper-V. So far as I can tell it does use AMD-V technology and I did enable virtualization in the motherboard BIOS but Hyper-V won’t start. So, the fall back was to go with ESXi.

I wanted to use ESXi 4.0 Update 1 however the network interface on the motherboard (consisting of a Realtek 8111/8168 chip) is not supported by ESXi. The only supported network interfaces are gigabit interfaces (which the Realtek is — it just isn’t supported and I didn’t have a supported interface card on hand). So, I figured no problem…I’ll just use ESXi 3.5 Update 4. Well, the Realtek chip is also not supported in ESXi 3.5 — but the PCI 3Com 905TX and an Intel EE Pro 10/100 are. However, SATA drives are not supported — well, not completely. I managed to get the system installed by booting from the CD, switching to the tech support console (hitting ALT-F1) and then logging in using the unsupported login. I then loaded the AHCI driver and restarted the install and ESXi installed nicely. However, booting off the install on the hard drives was a no-go since the AHCI drive wouldn’t load (for reasons I’m not sure of) and the system crashed. Back to square one.

I then noticed that VMware released Update 5 to ESXi 3.5 earlier this month (about two weeks ago). I read the release notes and realized that they had resolved the AHCI/SATA drive issues. I downloaded it, burned it to a CD and tried it. Bingo! It installed without a hitch and booted without a problem. Awesome. Now I’m in the process of building out my VM images.

No responses yet

Nov 17 2009

Cloning Windows Server 2003R2 System Disk

I have a situation that requires to clone a system disk on a Windows SBS 2003 system since the C: drive is running out of space. The issue is that the SBS server is down to about 1.5MB of free disk space (don’t ask how that happened…it’s not pretty) and, while the physical disk is 80GB in overall size, the system partition is only 10GB overall. The only real solution is to clone the system partition to a new disk, then boot the machine with the new disk as a slave and use diskpart to extend the partition. Then, after that’s done, change the jumpers on the two drives so that the new clone is the master drive and the original becomes the slave and boot the machine off the new clone drive. Simple.

I ended up choosing to use Clonezilla since it appeared to be more mature than g4u (whether this is true or not I cannot say for sure…nevertheless, they both appear to do the job). I found it by searching for “open source disk clone” and stumbled across the OSALT site. After downloading the ISO I went into my VM lab (a HP DL380 G3 running VMWare ESXi 3.5) and created a new VM consisting of the same hardware as the source except for a 25GB disk versus a 10GB disk.

I then attached the new 25GB disk to the source system (my Windows Server 2003R2 domain controller for the lab), started the VM, attached the ISO via the VMWare Infrastructure client and booted off the ISO/CD. After about 5 minutes Clonezilla came up and I setup the cloning process — all in all about 10 minutes worth of work at most. I then fired up the cloning process and after an additional 45 minutes the clone was done. The clone was still 10GB large as I didn’t explore whether Clonezilla could resize the partition on the fly while it was cloning it. No big deal…it’s easy to extend the partition under Windows if you have the space on your drive. I rebooted into the Windows OS (with the clone disk still attached) and I resized the partition using diskpart. Once that was completed I shutdown the VM, detached the cloned disk from the source and booted the new VM. After a complaint from Windows Product Activation that the system hardware had substantially changed and having to go through the reactivation process again I had a domain controller with more than double the disk space of the original system.

The next step is to do this to the SBS system.

No responses yet

Dec 09 2008

VMware Server 2.0 — Finally!

Published by under Linux,Virtualization,VMware

Well, VMware put Server 2.0 (the final release) out at the end of October and I have to see…it looks really good. I haven’t had a chance to compare it to Server 2008 Hyper-V yet (as I need to finish building my Hyper-V box first) however I have to say that between Server 2.0 and VMware ESXi 3.5 update 2 VMware has some very nice virtualization products out on the market. Let’s cover Server 2.0 first before we get into ESXi 3.5. Here’s the specs for the Server 2.0 system:

  • HP Proliant DL380
  • Dual 3.06GHz Xeon CPUs
  • Dual 10/100MB interfaces
  • 6 x 36GB 10,000 RPM SCSI disks w/hardware RAID-5
  • 12GB memory
  • O/S: Ubuntu Server 8.04LTS

Previously I was running VMware Server 1.0.6 on this system but now that VMware has released Server 2.0 I wanted to take another look at it. While the download is big (570MB) it certainly is not too bad and the install went flawlessly. The Server 2.0 detected the 1.0.6 software on the machine, uninstalled it (but kept the settings) and then installed Server 2.0 and configured it using the settings from the 1.0.6 version. All in all, within about 10 minutes I was back up and running. I like the look of the new interface as well — reminds me very much of the clean look of ESXi…and the best part is that it runs in IE8 Beta 2 (well, almost). To login I simply point the browser to the host system port 8333 as shown.

Server 2.0 Login

Once you login the amount of information at your fingertips is great.

VMware 2.0 summary page

Selecting a specific machine also gives you an immense amount of detail

Machine information in VMware Server 2.0

You can start machines directly from the summary page or you can start them from the console tab by clicking on the big white arrow

Starting a new VM image

Here’s where IE8 was a bit problematic (and I’m not sure it was IE8 or if it’s also an issue with IE7…I’ll have to test this). I had to do two things to get the console to work under Vista/IE8 Beta 2. First, I had to add the VMware server system into IE’s trusted site list. Second, I also had to be running in compatibility mode. Finally, once those two items were done I was able to install the VMRC plugin and get it to work. I still haven’t tried this under Firefox 3.0 yet and will try that later. Once the machine starts you click in the console window (or at one of the two squares in the console tab in the VMware shown below)

Console options in the Console tab

The option on the left opens up a new VMRC window as a separate window. The box on the right opens the VMRC in full screen mode (which you can then reduce to a regular sized window). Not sure what the benefit of the two options are but it’s there if you need it. Once the machine starts up you can have multiple VMRC windows running just like you have with ESXi.

VM console - Server 2008

VMware Server 2.0 Multiple VMRC windows

All in all, I’m quite impressed with VMware Server 2.0. I plan on playing with it for quite a while and will consider whether I will migrate this host server over to VMware ESXi 3.5. Given the small footprint of ESXi (both in terms of disk as well as memory) it may well be the way to go in the long run. I’ll review my ESXi experience in an upcoming post as well as my Hyper-V experience as well.

No responses yet

Nov 16 2008

New VM System

Published by under Linux,Virtualization

Well…I managed to get a hold of an HP DL380 G3 with dual 3.06GHz Xeon processors, 2GB of RAM, and 6 36GB hard drives with a RAID 5 controller for very little cost. This has accelerated my move towards virtualizing all of my systems at home. In the process of doing so I have also decided that I’m going to upgrade my domain controllers to Server 2008 Standard.

First off I’m using as a base OS on the DL380 Ubuntu 8.04 with VMware Server 1.0.6. The reason being is that I can’t install Server 2008 with Hyper-V on the DL380 G3 — it’s only supported on the DL380 G5 (which has the Intel procs with VT). So, given that I decided to go with Ubuntu and VMware. The install of Ubuntu went easy enough and VMware was just slightly more complicated. I used the instructions here to get the VMware MUI working properly (or course, now it’s not working properly and I’ve got to figure out why).

Anyway, now that I’ve gotten VMware installed and configured I have installed two VM guests — Server 2008 Standard and Server 2008 Core Standard. I’m using both network interfaces on the machine — one for connecting to the VM host system and the other is a bridged interface for the VM guests. I configured the Server 2008 Core system’s network settings according to Dan Petri’s instructions here. In addition I enabled NetBIOS over TCP/IP using the wmic command utility. The instructions for doing that are here.

Finally, once the VMware Tools were installed on the Server 2008 Core system I had to change the display resolution. Turns out, when you install the VMware Tools on Server 2008 Core the display resolution goes from 800×600 to 640×480 (which is really annoying – see the following picture).

Server 2008 Core After VMware Tools Install

Server 2008 Core After VMware Tools Install

Changing that is not so simple though as it requires that you modify the registry. You need to find the right registry setting. The video settings can be found in HKLM\SYSTEM\CurrentControlSet\Control\Video\ where is the ID for the video card and driver currently being used. Look for the Device Description registry key (it should be VMware SVGA II). Right above it are the DefaultSettings.XResolution and the DefaultSettings.YResolution registry key. They are both DWORDs and can be input either in hexadecimal or decimal (see below)

Server 2008 Core Registry Settings for Display Resolution

Server 2008 Core Registry Settings for Display Resolution

I changed the values in both registry keys to 1024 and 768 respectively and then restarted the VM (using the command shutdown /r /t 0). The final result is shown below.

Server 2008 Core with 1024x768 Display Resolution

Server 2008 Core with 1024x768 Display Resolution

Now I need to finish migrating the rest of my VM images off the old VALinux FullOn 2240 and onto the new system. The only things I need to do to the DL380 is increase the RAM from 2GB to 12GB and buy a cabinet for it…man is it loud…I forgot just how loud the DL380s are.

No responses yet

Aug 27 2008

Rebuilding the VMware Server

My trusty VALinux Full-On 2240 system finally reached the point where I couldn’t use it. The system disk is fine with Ubuntu 08.04 on it. The problem was with the RAID array that I created using 3 36GB disks. That’s where I stored my VM images and one disk back in June began to have problems. Well, a few weeks ago a second disk began to have problems — wonderful. The issue is that I had it configured as a RAID as a RAID 0 array…just concatenating the three partitions together. Doing that I wasn’t just able to replace a single drive and reconstruct it. Now I had to replace two out of the three drives. Oh joy.

The first thing I needed to do was to “rescue” the VM images I had installed on the RAID array. I copied the images to my desktop system (it pays to have 500GB of available space for this) and saved them that way. Then I ordered 3 73.4GB Seagate ST173404LCV drives from PC Progress in Elk Grove Village, IL. The disks arrived on Monday and I installed them on Tuesday. This time, I built the RAID array using mdadm as a RAID 5 array so that if one of the disks goes south I can always pull it, put another 73.4GB disk in and rebuild the array easily. Now the only thing left to do is to restore the images to the server. Again…oh joy!

No responses yet