Jul 29 2014

Hard Drive Failure vs. UPS Failure

When is a hard drive failure NOT a hard drive failure?  When it’s a bad UPS battery that is dying.  For the past week and a half I have noticed that my VMware ESXi server which hosts three systems for me (2 Microsoft Windows Server 2008R2 systems and an Ubuntu Linux server) was complaining about a corrupt datastore (specifically the boot disk).  While the VMware support site didn’t provide much information on the specific error that I was seeing I felt that it pointed to a hard drive that had bad sectors on it and was on its last legs (mind you, this drive is NOT that old and certainly doesn’t get a lot of activity).  I thought, “oh great – this is going to be fun to fix!”  I had moved the VMs off the server and was about to order a new disk when I then noticed that my APC SmartUPS 1400 was indicating that the battery in the UPS had gone bad (the old “when it rains it pours” adage came to mind immediately).  I figured the battery was not an issue – I’ll just replace it…it’s under warranty (1 year warranty and I bought the battery in September of 2013).  I called up AtBatt.com and spoke with the customer service representative, told them the problem and they authorized the return.  Given that my VMs were crashing (which I thought was due to the ESXi server having a kernel oops and then restarting) I setup a DHCPd server off of my Cisco PIX 501E firewall, enabled it, got the VMs restarted and then disabled the PIX’s DHCPd process (but did not do a “write mem” on the PIX – so in the saved config the PIX DHCPd was set to enabled).

Yesterday, I suddenly notice that I’m getting an IP address from the range configured in the PIX DHCPd server.  I go in and poke around and discover that the PIX had rebooted at 6:11AM yesterday morning.  On top of that my Cisco AP1200 wireless had also rebooted at 6:11AM, and so did my ESXi server (and the event logs were complaining about a corrupt datastore).  Suddenly it occurred to me that the problem was not in the ESXi server (or the PIX or any other network gear) but rather in the UPS.  The UPS was doing a self-test at 6:11AM, the battery failed and the UPS rebooted itself (thereby interrupting power to my entire network stack).  I quickly replaced the UPS with my other SmartUPS 1400 which is still good and everything has been humming along well since (no problems noticed).

This morning I open up the SmartUPS with the bad battery and to my shock I find that the battery is deformed in shape as can be seen from the pictures below.

photo 2 photo 1 photo 3 photo 4

In essence the battery failed horribly and I am quite lucky that it didn’t explode or start a fire!  It took me 15 minutes and the removal of the UPS cover and pulling the case apart a little bit just to get the battery out.  The battery is an Amstron battery and is manufactured in China.  Suffice it to say I am shipping it back today.  Now, I’m supposed to receive a replacement battery from AtBatt but I will also order one from APC.  I am not willing to risk a fire or a battery explosion to save $80.  It’s just not worth it.

No responses yet

Dec 17 2009

ESXi Struggles

I’ve finally built my new virtual server — the one in which I’m going to consolidate my current machines as virtual machines. The intention is to measure the amount of energy my current systems (consisting of a Sun Ultra 60 — dual 400MHz UltraSPARC II CPUs, 2GB memory, and 2x20GB SCSI drives; a Dell Workstation 610 — dual 700MHz Pentium III CPUs, 768MB memory, and a 20GB IDE drive as well as a 30GB IDE drive; a home-built server with an AMD Athlon 1.2GHz CPU, 512MB memory and a 30GB IDE drive as well as a 9.1GB SCSI drive) and then migrate everything to the virtual machine world and measure the energy used by the VM server.

The VM server consists of the following hardware:

Seasonic SS-500ES 500W power supply
Gigabyte GA-MA790GPT-UD3H
AMD Athlon x64 X4 CPU 630 Propus 2.8GHz (quad-core, 95W)
8GB memory
1 x 160GB 7200RPM SATA drive
1 x 500GB 7200RPM SATA drive

The first idea was to install Windows Server 2008R2 core with Hyper-V on the machine and use that to build the VM images. However, the Athlon x64 X4 CPU is not supported by Hyper-V. So far as I can tell it does use AMD-V technology and I did enable virtualization in the motherboard BIOS but Hyper-V won’t start. So, the fall back was to go with ESXi.

I wanted to use ESXi 4.0 Update 1 however the network interface on the motherboard (consisting of a Realtek 8111/8168 chip) is not supported by ESXi. The only supported network interfaces are gigabit interfaces (which the Realtek is — it just isn’t supported and I didn’t have a supported interface card on hand). So, I figured no problem…I’ll just use ESXi 3.5 Update 4. Well, the Realtek chip is also not supported in ESXi 3.5 — but the PCI 3Com 905TX and an Intel EE Pro 10/100 are. However, SATA drives are not supported — well, not completely. I managed to get the system installed by booting from the CD, switching to the tech support console (hitting ALT-F1) and then logging in using the unsupported login. I then loaded the AHCI driver and restarted the install and ESXi installed nicely. However, booting off the install on the hard drives was a no-go since the AHCI drive wouldn’t load (for reasons I’m not sure of) and the system crashed. Back to square one.

I then noticed that VMware released Update 5 to ESXi 3.5 earlier this month (about two weeks ago). I read the release notes and realized that they had resolved the AHCI/SATA drive issues. I downloaded it, burned it to a CD and tried it. Bingo! It installed without a hitch and booted without a problem. Awesome. Now I’m in the process of building out my VM images.

No responses yet

Nov 17 2009

Cloning Windows Server 2003R2 System Disk

I have a situation that requires to clone a system disk on a Windows SBS 2003 system since the C: drive is running out of space. The issue is that the SBS server is down to about 1.5MB of free disk space (don’t ask how that happened…it’s not pretty) and, while the physical disk is 80GB in overall size, the system partition is only 10GB overall. The only real solution is to clone the system partition to a new disk, then boot the machine with the new disk as a slave and use diskpart to extend the partition. Then, after that’s done, change the jumpers on the two drives so that the new clone is the master drive and the original becomes the slave and boot the machine off the new clone drive. Simple.

I ended up choosing to use Clonezilla since it appeared to be more mature than g4u (whether this is true or not I cannot say for sure…nevertheless, they both appear to do the job). I found it by searching for “open source disk clone” and stumbled across the OSALT site. After downloading the ISO I went into my VM lab (a HP DL380 G3 running VMWare ESXi 3.5) and created a new VM consisting of the same hardware as the source except for a 25GB disk versus a 10GB disk.

I then attached the new 25GB disk to the source system (my Windows Server 2003R2 domain controller for the lab), started the VM, attached the ISO via the VMWare Infrastructure client and booted off the ISO/CD. After about 5 minutes Clonezilla came up and I setup the cloning process — all in all about 10 minutes worth of work at most. I then fired up the cloning process and after an additional 45 minutes the clone was done. The clone was still 10GB large as I didn’t explore whether Clonezilla could resize the partition on the fly while it was cloning it. No big deal…it’s easy to extend the partition under Windows if you have the space on your drive. I rebooted into the Windows OS (with the clone disk still attached) and I resized the partition using diskpart. Once that was completed I shutdown the VM, detached the cloned disk from the source and booted the new VM. After a complaint from Windows Product Activation that the system hardware had substantially changed and having to go through the reactivation process again I had a domain controller with more than double the disk space of the original system.

The next step is to do this to the SBS system.

No responses yet

Jun 09 2009

SSH Server on Windows Server 2008 Core

Published by under Security,Windows

I’ve been playing around (in my copious free time 😉 ) with other methods of connecting to and managing Server 2008 Core. One of the things I’ve wanted to do was to be able to SSH directly to Server 2008 Core and have the same command line capability as I do on the console. To that end I did a quick search for similar work and found the following article at TechRepublic about installing an SSH server in Windows 2008. The difference that I wanted to do was to install it in Server Core rather than the full-blown version of 2008.

Like David Davis over at TechRepublic I decided to start with FreeSSHd as my SSH server. The first thing I needed to do was to get it onto the Server Core VM. Rather than downloading it to my desktop and then transferring it to the Server Core VM I decided that would rather download it directly to the Server Core machine. In order to do that I needed wget that would run on Windows. I used the wget binary I downloaded (to my desktop) from Bart Puype in Belgium. Once I copied wget to C:\Windows\System32 I used it download the FreeSSHd.exe binary from FreeSSHd.com.

To install freesshd, just run the freesshd.exe program and it will start up the install wizard. A couple of items to note — on Server Core do not bother with creating a Start Menu item for FreeSSHd and don’t bother with creating a desktop icon either. One of the problems that I encountered when I installed FreeSSHd on Server Core was that I could not configure the SSH server since the task bar icon did not appear on the right (as should be the case since there is no task bar in Server 2008 Core). To configure FreeSSHd I had to edit the freesshdservice.ini file in the C:\Program Files\freesshd directory (the default location for the installation).

A small point to note. Server 2008 Core’s firewall is on by default (even if it’s a domain joined machine) and the policy is to block all inbound connection attempts but to allow outbound connections. After installing FreeSSHd I needed to modify the firewall and decided to use netsh to do so. The command I used was

netsh advfirewall firewall add rule name="SSHd" dir=in action=allow protocol=TCP localport=22

Very simple…I love netsh 🙂
Another problem I ran into was getting the NT authentication to work. I did manage to get the password authentication working but I wanted to tie the FreeSSHd server into the Windows authentication. I’m still not 100% sure as to where the problem lies with the NT authentication integration and will investigate it further.

One of the biggest drawbacks to FreeSSHd is that there is very little (re: almost none) documentation that covers the freesshdservice.ini file. You need to read the forums over at freesshd.com in order to get a sense of what the settings are for the file and what specific changes to the file cause in the overall operation of the server. I hope to get that put together and posted here this summer as I think others will find it useful.

To get the password authentication working I installed FreeSSHd on a Windows Server 2003 system and then created the users I wanted there and copied over the relevant portions of the freesshdservice.ini file to the one on the Server 2008 Core VM. Then, to restart the service I would just issue the commands: net stop freesshdservice and net start freesshdservice and I was good to go. As you can see from the last capture in the gallery below I was able to connect to the server and log in using the account I had created on the Server 2003 system and copied over to the freesshdservice.ini file on the Server 2008 Core VM.

In the future I’m going to try some of the other freely available SSH servers and see if they provide an easier integration into Server 2008 Core.

One response so far

Jan 19 2009

Server 2008 DNS Global Query Block List

Published by under Windows

I recently promoted one of the Server 2008 VMs in my lab to a domain controller for the lab domain and installed the DNS role as well (so now I have three DNS servers in the VM lab — all three are domain controllers). The “firewall” to the VM lab is an ISA 2006 server with the Web Proxy Auto-Discovery (WPAD) configured and I have a CNAME entry in DNS for wpad so that the ISA firewall clients can dynamically detect the ISA server and configure the settings in IE.

According to the document, “Windows Server 2008 – DNS Server Global Query Block List” the initial query block list contains the entries ‘wpad’ and ‘isatap’ by default. However, when you install or upgrade a server to Windows Server 2008 and you install the DNS role the installation is supposed to detect whether entries already exist in DNS for the names ‘wpad’ and ‘isatap’ and to remove those entries from the block list upon detection (remember, this only occurs upon installation or upgrade — not later on during normal operation). It only detects, however, when the record is either an A (IPv4 address) or AAAA (IPv6 address). In my case the record is a CNAME and therefore ‘wpad’ was automatically added to the global query block list. This generates an EventID of 7600 with the following text in my case:

The global query block list is a feature that prevents attacks on your network by blocking DNS queries for specific host names. This feature has caused the DNS server to fail a query with error code NAME ERROR for WPAD.DOMAIN.TLD even though data for this DNS name exists in the DNS database. Other queries in all locally authoritative zones for other names that begin with labels in the block list will also fail, but no event will be logged when further queries are blocked until the DNS server service on this computer is restarted. See product documentation for information about this feature and instructions on how to configure it.

Below is the current global query block list (this list may be truncated in this event if it is too long):

EventID 7600 - DNS Query Block List

EventID 7600 - DNS Query Block List

The solution can be found at the Forefront TMG (ISA Server) Product Team blog in their entry titled Windows Server 2008 DNS Block Feature. Additional information can be found on TechNet in the document “DNS Server Global Query Block List” under the Windows Server 2008 resource section covering the Domain Name System.

To make a long story short, the simple solution is to reconfigure the global query block list using the dnscmd command as shown below

Reconfiguring DNS Global Query Block List on Server 2008

Reconfiguring DNS Global Query Block List on Server 2008

No responses yet

Jan 15 2009

Migrating WSUS 3.0 SP1 susdb from SQL Server 2005 – Part 2

Published by under System Center,Windows

It took a little effort to figure it out along with some research but I’ve finally managed to move the SUSDB from the SQL Server instance running on my System Center Configuration Manager VM back over to the WSUS VM with SQL Server 2005 Express Edition. Essentially I followed a combination of procedures that can be found at the following links:

In essence I did the following procedure:

  1. Install SQL Server 2005 Express Edition SP2 on the WSUS VM
  2. Stop “Update Services” on WSUS VM (to avoid updating and locking the SUSDB on the SQL Server 2005)
  3. Stop “IISAdmin” service on WSUS VM (this also stops the World Wide Web Publishing Service, the Windows Remote Management (WS-Management) service, and the HTTP SSL service)
  4. On the SCCM VM, stop the “IISAdmin” service (as above that also stops three other dependent services)
  5. Detach the SUSDB on SQL Server 2005 – this can be done using either the SQLCMD command line interface with the ‘sp_detach_db’ command or the SQL Server Management Studio
  6. With the SUSDB database detached, copy it from it’s current location (under SQL Server 2005 SP2 it’s found in C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Data) to the appropriate directory on the WSUS VM (the default location for the data files with SQL Server 2005 Express is the same as the full SQL Server)
  7. Once the SUSDB (and the attendant log file) has been copied over, attach it to the SQL Server 2005 Express instance. To do this I used the SQL Server Express Management Studio as shown in the picture below
  8. Adding SUSDB to SQL Server 2005 Express

    Adding SUSDB to SQL Server 2005 Express

  9. After attaching the database to the SQL Server 2005 EE I then followed through with the steps outlined in the >WSUS Support Team’s blog: How to migrate your WSUS Windows Internal Database to SQL Server 2005 Express Edition. Except that in this case I don’t think I really needed to add the NT AUTHORITY\NETWORK SERVICE account as a login on the SQL Server 2005 EE instance since the WSUS software and the SQL Server were on the same VM.
  10. Nevertheless, I added NT AUTHORITY\NETWORK SERVICE to the SQL Server 2005 EE logins as shown below
  11. Adding NT AUTHORITY\NETWORK SERVICE to SQL Server 2005 EE Logins

    Adding NT AUTHORITY\NETWORK SERVICE to SQL Server 2005 EE Logins

  12. I discovered that you also needed to add the NT AUTHORITY\NETWORK SERVICE as a user in the actual SUSDB. This is subtle point that is not made clear by the WSUS Team Blog on this issue in their post. To do that you need to go to the actual database (in this case SUSDB) and under -> Security -> Users you create an account for the NETWORK SERVICE account.
  13. Adding NETWORK SERVICE as a user on SUSDB

    Adding NETWORK SERVICE as a user on SUSDB

  14. You also need to add NETWORK SERVICE to the webService role in SUSDB as shown below
  15. Adding NETWORK SERVICE to webService role in SUSDB

    Adding NETWORK SERVICE to webService role in SUSDB

  16. Finally, I changed the SqlServerName registry key in HKLM\Software\Microsoft\Update Services\Server\Setup from the original SQL Server 2005 system (winsrv-ca) to the WSUS server (SCCM). However, because I went from a SQL Server 2005 system to a SQL Server 2005 Express Edition database I had to append the instance name (in this case SQLExpress) in order to get the MMC snap-in to work.

A bit of a sidenote — the name of my WSUS server virtual machine is SCCM and the name of the machine running the System Center Configuration Manager software is actually winsrv-ca. It’s kind of confusing but I had already installed WSUS on the virtual machine SCCM before I decided that adding System Center Configuration Manager would be pretty much an overload on that one machine. That’s why I ended up installing Configuration Manager on winsrv-ca and not SCCM…but I had already modified AD to point to SCCM for WSUS services so I found it less troublesome to just leave the names as they are)

Finally, the job is done. Yes, I realize that it’s probably not a big thing but I wanted to see if it could be done. Most of the sites out there talk about migrating the WSUS database either from the Windows Internal Database (SQL Server Embedded Edition) to a SQL Server 2005 system or from an older SQL Server 2000 database to a SQL Server 2005 database. I haven’t seen anyone who tried to go from a full blown version of SQL Server 2005 to a SQL Server 2005 Express Edition (which, in many cases is more limited that the Windows Internal Database that WSUS can use). Anyway, it was fun…with a few snags along the way…but fun nontheless.

One response so far