Problems Clustering Virtual Machines on Windows Server 2012 Hyper-V

I was re-building our lab environment at work the other week in preparation for our big Summit13 event, that and the lab had been trashed over the last year...

As part of the re-build I had decided to implement a couple of virtual machine clusters, one for a scale-out file server and one as a SQL cluster.

I'd deployed the virtual machines for the cluster nodes using Service Templates in SCVMM and as part of that template chosen to use an availability set to ensure the VM's were separated across hosts (a cluster doesn't provide much High Availability if they all reside on the same host that's failed!).

When I started to create the cluster I ran straight into a problem with the Failover Cluster Manager reporting problems due to timeouts when creating the cluster.

Creating a single node cluster worked fine, but then would again fail when trying to add in another node.

I happened to put one of the Hyper-V hosts into maintenance mode for something and it migrated all the VM's onto the same host, at which point creating the cluster worked flawlessly, yay!

However, when the Hyper-V host came back out of maintenance mode and the availability sets kicked in during optimisation forcing a VM node back onto a separate physical host, the clusters broke again, not yay :(

So after some Googling Binging about and a shout on Twitter (Thanks @hvredevoort and @WorkingHardInIT) an issue with Broadcom NICs was brought to my attention and I came across this TechNet Forum post talking about the same issue.

Sophia_whx suggested trying to use Disable-NetAdapterChecksumOffload on the NICs to help with the issue.

So, first things first.  Use the Get-NetAdapterChecksumOffload to see just what the configuration was and sure enough Checksum Offload is enabled for just about all services across the majority of the NICs.

I then used the Disable-NetAdapterChecksumOffload * -TcpIPv4 command which resulted in this:

 

A reboot later and then perform it on the second host and whoa....

For some reason, the virtual switch really didn't like having that done to it.

I wish I had some screenshots, but I went into "get it fixed fast" mode.

 

Basically the switch via powershell was showing as up, the NIC Teaming GUI was showing it down and all the bound adapters as failed. SCVMM had lost all configuration for the switch altogether.

Deleting the switch from SCVMM didn't delete it from the host, but brought it back to life on the host but was missing in SCVMM.  SCVMM then wouldn't redetect it or let you build it again as it was still there, apparently???

 

I had to manually remove the team from a remote NIC Teaming GUI (I could of PowerShell'd it I know!) and then recreated via SCVMM.
 

Anyway... at first this looked to have fixed the clustering within virtual machine issues, but it only delayed the symptoms i.e. it took longer to evict nodes and randomly brought them back online.

 

So next I tried disabling Checksum Offload for all services, being careful not to touch the Virtual Switch this time.

 

Rather than going adapter by adapter I used the following command:

 

Get-NetAdapter | Where-Object {$_.Name -notlike "Converged*"} | Disable-NetAdapterChecksumOffload

 

This resulted in the Checksum Offload being disabled for the various services as shown, except for my virtual switch.

After doing this on the other host and giving them a reboot, my clustered virtual machines appear to be nice and stable when split across physical hosts. Yay! Problem fixed.

Just as another side note about Broadcom adapters, there have also been reports of performance issues when the Virtual Machine Queue (VMQ) setting is enabled, despite it being a recommended setting.

A quick check of my hosts showed it was enabled:

Another quick PowerShell line later and it wasn't:

Get-NetAdapterVmq -InterfaceDescription Broad* | Disable-NetAdapterVmq

Extended Virtual Machine Discovery Management Pack

Infront released a new management pack, a while ago, for FREE, to the community via System Center Central.

http://www.systemcentercentral.com/tabid/143/indexid/94055/default.aspx?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+systemcentercentral%2Fblogs+%28Blogs+at+System+Center+Central%29

This management pack will give you a deeper insight into your virtual machines including recognising which platform (Hyper-V, VMware & Citrix), which hosts they're located on, if VM Tools are installed and the version.

(footnote. I actually wrote this post the day they released it, but forgot to post it /facepalm)

Building the TESG Private Cloud Customer Experience Centre - Part 1

Every year my employer holds an event for customers (and potential new customers) to show case what we do and give customers a chance to meet our partner vendors.

This year, nicely coinciding with just after the System Center 2012 release, I landed the brilliant job of setting up something to demonstrate our System Center and Desktop expertise.

And so the concept of the Private Cloud and Optimised Desktop Customer Experience Centre was born.

The goal?

1. To showcase the full System Center 2012 suite
2. To showcase the interactions of each component and how they drive efficiencies
3. To showcase an elastic and easily scalable datacentre that can flex into the Public Cloud
4. To showcase the dynamic desktop with OS, Data, User and Application layers abstracted
5. To showcase BYOD and specifically desktop/application access on tablet devices

Over a couple of blog posts I'll aim to share some of the planning, thoughts and tips & tricks that went into building it.
What I'll not be doing is guides on how to install the different components as there are plenty of them out there, but I will post links to some relevant good guides.

My original test lab was made up of a couple of HP Proliant DL380 G7's with some shared space pinched off the corporate SAN, but as this was going to need to host a lot more and it would need to be "slightly" portable for attending events like the T360 it was time to purchase some upgrades.

1. More memory.  Upgrade from 64Gb per host to 128Gb
2. Dedicated Storage.  iSCSI SAN that would also allow me to show some of the VMM storage management features (N.B. More details on this later, plus some pitfalls to watch out for!)
3. Dedicated Switches.  To show SCOM network management & keep the environment self contained.
4. More NIC's.  The original environment only had 4 onboard NICs, not good enough.
5. Flight case to rack it all in to make it portable (kind of!)

Now that might sound slightly overkill for a test/demo environment.  However, I have a laptop which is quite capable of showing 2-3 of the System Center products at the same time, but this Customer Experience Center had to host the following:

• Active Directory
• Virtual Machine Manager
• Operations Manager
• Service Manager
• Configuration Manager
• Data Protection Manager
• Orchestrator
• App Controller
• SQL 2008 R2 Server
• SharePoint Enterprise Server
• Exchange
• Lync
• ForeFront UAG
• ForeFront TMG
• File Servers
• XenDesktop Mgt Server
• XenDesktop VDI Desktops
• XenApp Mgt Server
• XenApp App Servers
• Remote Desktop Session Hosts
• Remote Desktop Broker/Gateway/Licensing
• RDS/Hyper-V VDI Desktops
• Dedicated Win 7 Admin Workstations
• Citrix NetScaler VM Appliance
• App-V Sequencer Workstations

When you consider that all of this needs to be up and running at the same time, my laptop just wasn't going to cope!

So far this has spread out across 34 VM's and there's still more to come...

This is a quick example diagram that I drew up to show the Hyper-V layout

Once all the hardware components were installed and racked then Hyper-V was the first thing to tackle and all I can say is thank god for Aidan Finn and his blog: http://www.aidanfinn.com/

Lots of useful posts, for example: http://www.aidanfinn.com/?p=10311

I'm going to leave the rest for the next post, but I just want to mention something that came to light when I installed the first System Center component, Virtual Machine Manager.

This is a logical first place to start if you've got the chance to build a private cloud from scratch like I have as you can implement Service Templates for deploying your VM's to help structure the environment and provide servicing and scale out options.

However, I hit a problem almost straight away, I struggled to get it to see my storage provider.

Originally I was ordering a Dell Equalogic iSCSI SAN for the environment, but due to certain disks not being available and increased costs for alternatives I was suggested to look at a DotHill AssuredSAN 2332.

The first thing I did was ask/check it supported SMI-S protocol, which it did as this is what VMM requires for the new features.
However when trying to set it up in VMM, it soon came to light that it only supported SMI-S 1.3 whereas VMM requires version 1.5.

So lesson learnt, make sure that when checking specifications, especially SAN's that you check in detail, right down to the version number!

There is a useful table (I found this afterwards!) that details the supported arrays:
http://technet.microsoft.com/en-us/library/gg610600.aspx

Part 1 - Building the TESG Private Cloud Customer Experience Centre

Part 2 - Service Accounts & Permissions

Part 3 - Installation Guide Links
Part 4 - Partner Solutions & Extensions

SCOM & SCVMM Dirty Views

Both myself and Rob have mentioned a few times about the dirty views in Operations Manager after importing the Virtual Machine Manager MP (the SQL view contains ALL virtual computers rather than just servers with SQL installed).

Well good news, it's fixed in the new SP1 release of SCVMM!!!!

Along with Kevin Holman giving an outline of the new SCVMM SP1 he also confirms the problem, that for myself was a show stopper on the last System Center product for us to tackle, is now fixed.

More information can be found here:
http://blogs.technet.com/b/kevinholman/archive/2011/03/29/system-center-virtual-machine-manager-2008-r2-sp1-ships.aspx

Hurahh for Microsoft, it's only taken just over two years to fix ;)

P2V Migration - Microsoft Deployment Toolkit

I just got this e-mail from the MDT Team via the Microsoft Connect site.
Rather than type waffle to describe it I thought i'd just repost it.
--------------------------------------------------------------------------

Help unblock OS deployments by redelivering blocking users' old Windows environments, applications and browsers seamlessly in Windows 7 using automated physical-to-virtual migration

P2V Migration for Software Assurance uses the Microsoft Deployment Toolkit and Sysinternals Disk2VHD to convert a user's existing Windows XP or newer client environment to a virtual hard disk then automates the delivery of an updated and personalized Windows 7 operating system containing a virtual machine with the user's previous Windows environment, applications and Web browser. The user's previous virtual desktop retains its existing management components, domain membership and policies. The process also publishes applications and the browser for the user to access them seamlessly within Windows 7's
start menu.

How it Works

Starting Windows environment with Windows XP SP3 or newer. Environment is personalized with applications not compatible with Windows 7.

Microsoft Deployment Toolkit 2010 initiates fully-automated migration to Windows 7. Process includes P2V conversion of the running OS using Sysinternals Disk2VHD.

Windows 7 migration complete. Windows 7 contains the previous operating system in its entirety within a virtual machine.

Standalone application and Internet Explorer links published from virtual machine to native Windows 7 start menu.

Incompatible application from previous operating system is launched seamlessly within Windows 7 using RemoteApp integration and Virtual PC.

Even the most skilled IT shops, with resources to work through application compatibility challenges, often find a few users blocking roll-outs to entire sites or larger numbers of users due to:

• Compatibility of specialized, region or user-specific applications, browser customizations or USB device drivers.
• Missing installation files or media to install applications or drivers for testing or redelivery.
• Critical users who cannot risk losing any functionality of their current desktop environments and require a working backup before moving to Windows 7.

These types of issues make it difficult to completely replace existing systems with Windows 7 without causing user disruption. Allowing users to retain their current desktop environments while moving to Windows 7 can smooth the transition while mitigating or delaying investments required for making applications and hardware natively compatible with Windows 7.

Download P2V Migration for Software Assurance now:
P2V Migration for Software Assurance beta (https://connect.microsoft.com/site14/Downloads/DownloadDetails.aspx?DownloadID=30989)

Note: Physical-to-virtual hard drive migration of a Windows installation is a valid operation for customers with Software Assurance and full retail copies of Windows XP, Windows Vista, and Windows 7. Software Assurance provides users valuable benefits. Windows XP, Windows Vista and Windows 7 installed by Original Equipment Manufacturers (OEM) using OEM versions of these products may not be transferred to a virtual hard drive in accordance with Microsoft licensing terms.

Customer Benefits
Help Reduce Windows 7 Deployment Times: The ability to perform P2V conversion of Windows XP or newer 32-bit systems as part of Windows 7 and/or 64-bit deployment means that IT organizations do not need to wait as long to get value from Windows 7. IT organizations will deliver the new Windows 7 operating system builds while preserving the old environments of a limited set of users that would otherwise delay production deployment.

Extend the Timeframe to Mitigate Application Compatibility: Using this solution, targeted users can have access to their previous set of
applications, just in case something was not provisioned as part of the Windows 7 deployment. Accessing previous applications is also easy for end users, as those applications are published to the Windows 7 start menu.

Users can Access Incompatible Legacy OS Applications: Using this solution, targeted users can have access to their previous set of applications, just in case something was not provisioned as part of the Windows 7 deployment. Accessing previous applications is also easy for end users, as those applications are published to the Windows 7 start menu.

How does P2V Migration compare to Windows XP Mode or MED-V?
Windows XP Mode: Windows XP Mode is not optimized for managed desktop environments. The operating system included in Windows XP Mode cannot be customized easily without manual effort. The time spent recreating a user's previous environment can easily take several hours per computer and it is not guaranteed that software installation media exists for every application on the previous system.

Microsoft Enterprise Desktop Virtualization (MED-V): Microsoft Enterprise Desktop Virtualization (MED-V) removes the barriers to Windows upgrades by resolving application incompatibility with Windows Vista or Windows 7. MED-V enables administrative control over the distribution and management of Virtual PC images, thereby ensuring that those images are up-to-date and compliant with regulations. P2V Migration is targeted at specific deployment-blocking users with specialized requirements or conditions where a standardized virtual environment delivered with MED-V will not
suffice.

Download P2V Migration (https://connect.microsoft.com/site14/Downloads/DownloadDetails.aspx?DownloadID=30989)

Tell us what you think! Send your feedback and questions to the SAT team.
For If you are not already a member of the MDT beta program, click link below to join:
http://connect.microsoft.com/site14/InvitationUse.aspx?ProgramID=1646&InvitationID=P2VM-C49K-PQHR

MDT Team