SCOM Console Install on VMM Server Causes Service Crash

Had a strange one today.

Customer had tried previously to setup the connection between VMM and SCOM but made some mistake somewhere along the line and then uninstalled the SCOM console without removing the connection as they said the console was causing the VMM service to constantly crash.

Having not seen that behaviour before and slightly doubting it somewhat I re-installed the console and sure enough was prevented from accessing the VMM console as the service was crashing.

Just as an added check, I tried running some PowerShell commands to check it wasn't a GUI issue only to be created by error messages complaining that the VMM service wasn't running or accessible.

So I uninstalled the console again which allowed me access back to VMM and running the Get-SCOpsMgrConnection showed me the broken connection.  However attempts to remove via the console or PowerShell were both met by errors telling me I needed the SCOM console installed first in order to manage the connection.  Arh.. slight problem...

After checking everything I could think of (SPN's, SCP's, Service Accounts etc etc) and not finding anything that stood out (Including nothing useful in the event logs) I thought I'd try a timing trick.

So I opened up a SCVMM PowerShell window ready, kicked off the SCOM console install again and then repeatedly spammed Remove-SCOpsMgrConnection -Force and wouldn't you know it after a few messages saying the SCOM console must be installed, just before the install completed the command completed successfully and removed the broken connection.  More to the point the SCOM console installation completed and the VMM service didn't crash.

After recreating the connection everything remained stable, but even though the create connection job ran successfully, the following error was present in the connector details:

“Operations Manager discovery failed with error: “Exception of type ‘Microsoft.VirtualManager.EnterpriseManagement.common.discoverydatainvalidrelationshipsourceexceptionOM10’ was thrown.

This is because the SCOM connection was created with the PRO-Tips enabled but without a SCOM monitoring agent deployed to the VMM Server.
Easily fixable, just untick the PRO and Maintenance Mode connection settings, deploy a SCOM agent to the VMM server and once the agent is installed and reporting, re-enable the options.

Offload Data Transfer (ODX) in Windows Server 2012

I've been working on a nice Dell R720 host based Hyper-V cluster this week with a Dell Compellent array providing the storage.

One of the things I was looking forward to with this job was getting hands on with the ODX feature of the Compellent.

ODX (Offload Data Transfer) is a feature found on some of the newer storage arrays that helps with large file operations by (in simplified terms) keeping the transfers within the array rather than passing the file to the source then destination servers then back to the array.

The first thing to do (assuming you know the hardware supports it) would be to check that the OS and it's software components supports ODX.

Now this is a Windows Server 2012 and 2012 R2 only feature so if you're on 2008 R2, time to upgrade.

From a PowerShell prompt, run the following command:
Fltmc instances

Take a note of the volume name of either the drive, or in my case the CSV volume you want to check.  Then run:
Fltmc instances -v <Volume Name>

e.g. Fltmc instances -v C:\ClusterStorage\Volume1

This will give you the filter names that you will need to check.
Run this command, replacing the <Filter Name> with those shown by the previous command.

Get-ItemProperty hklm:\system\currentcontrolset\services\<FilterName> -Name "SupportedFeatures"

So for my two filters of FsDepends and MpFilter I get the following output:

The property that needs checking is "SupportedFeatures".  If this has a value of 3 then ODX is supported and you're good to go.  Anything else and you'll need to look into it further.

Lastly, check if ODX is enabled on your server using this command:
Get-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode"

If it returns a "FilterSupportedFeaturesMode" other than 0 as shown below then ODX isn't enabled.

Run this to enable ODX:
Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode" -Value 0 -Type DWord

Or this to disable ODX if needed:
Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode" -Value 1 -Type DWord

In order to demonstrate to the client that ODX was indeed enabled and more to the point worth having, I modified the script on Hans Vredevoort shows on his blog post discussing ODX testing between 3Par and Compellents here: http://www.hyper-v.nu/archives/hvredevoort/2013/07/notes-from-the-field-using-odx-with-hp-3par-storage-arrays/

I ran the script which loops through creating 10 x 50Gb and 10 x 475Gb fixed disks with ODX enabled and then does the same but with ODX disabled.

This was the timings from the test:

With ODX
12.6 seconds for 10 x 50Gb vhdx files
84.2 seconds for 10 x 475Gb vhdx files
96.8 seconds total for all vhdx files

Without ODX
1015.5 seconds (nearly 17 mins) for 10 x 50Gb vhdx files
8615.8 seconds (just over 2 hours) for 9 x 475Gb vhdx files (N.B. I ran out of disk space for the 10th)
9631 seconds or 2.7 hours total for all vhdx files

 

There is a MASSIVE difference in creation times!

ODX is a feature well worth having in my opinion.  What I really can't wait for is ODX support with SCVMM libraries in the SCVMM 2012 R2 release!!


I've uploaded the ODX Test script to SkyDrive here: http://sdrv.ms/16QhZZE

Problems Clustering Virtual Machines on Windows Server 2012 Hyper-V

I was re-building our lab environment at work the other week in preparation for our big Summit13 event, that and the lab had been trashed over the last year...

As part of the re-build I had decided to implement a couple of virtual machine clusters, one for a scale-out file server and one as a SQL cluster.

I'd deployed the virtual machines for the cluster nodes using Service Templates in SCVMM and as part of that template chosen to use an availability set to ensure the VM's were separated across hosts (a cluster doesn't provide much High Availability if they all reside on the same host that's failed!).

When I started to create the cluster I ran straight into a problem with the Failover Cluster Manager reporting problems due to timeouts when creating the cluster.

Creating a single node cluster worked fine, but then would again fail when trying to add in another node.

I happened to put one of the Hyper-V hosts into maintenance mode for something and it migrated all the VM's onto the same host, at which point creating the cluster worked flawlessly, yay!

However, when the Hyper-V host came back out of maintenance mode and the availability sets kicked in during optimisation forcing a VM node back onto a separate physical host, the clusters broke again, not yay :(

So after some Googling Binging about and a shout on Twitter (Thanks @hvredevoort and @WorkingHardInIT) an issue with Broadcom NICs was brought to my attention and I came across this TechNet Forum post talking about the same issue.

Sophia_whx suggested trying to use Disable-NetAdapterChecksumOffload on the NICs to help with the issue.

So, first things first.  Use the Get-NetAdapterChecksumOffload to see just what the configuration was and sure enough Checksum Offload is enabled for just about all services across the majority of the NICs.

I then used the Disable-NetAdapterChecksumOffload * -TcpIPv4 command which resulted in this:

 

A reboot later and then perform it on the second host and whoa....

For some reason, the virtual switch really didn't like having that done to it.

I wish I had some screenshots, but I went into "get it fixed fast" mode.

 

Basically the switch via powershell was showing as up, the NIC Teaming GUI was showing it down and all the bound adapters as failed. SCVMM had lost all configuration for the switch altogether.

Deleting the switch from SCVMM didn't delete it from the host, but brought it back to life on the host but was missing in SCVMM.  SCVMM then wouldn't redetect it or let you build it again as it was still there, apparently???

 

I had to manually remove the team from a remote NIC Teaming GUI (I could of PowerShell'd it I know!) and then recreated via SCVMM.
 

Anyway... at first this looked to have fixed the clustering within virtual machine issues, but it only delayed the symptoms i.e. it took longer to evict nodes and randomly brought them back online.

 

So next I tried disabling Checksum Offload for all services, being careful not to touch the Virtual Switch this time.

 

Rather than going adapter by adapter I used the following command:

 

Get-NetAdapter | Where-Object {$_.Name -notlike "Converged*"} | Disable-NetAdapterChecksumOffload

 

This resulted in the Checksum Offload being disabled for the various services as shown, except for my virtual switch.

After doing this on the other host and giving them a reboot, my clustered virtual machines appear to be nice and stable when split across physical hosts. Yay! Problem fixed.

Just as another side note about Broadcom adapters, there have also been reports of performance issues when the Virtual Machine Queue (VMQ) setting is enabled, despite it being a recommended setting.

A quick check of my hosts showed it was enabled:

Another quick PowerShell line later and it wasn't:

Get-NetAdapterVmq -InterfaceDescription Broad* | Disable-NetAdapterVmq

Extended Virtual Machine Discovery Management Pack

Infront released a new management pack, a while ago, for FREE, to the community via System Center Central.

http://www.systemcentercentral.com/tabid/143/indexid/94055/default.aspx?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+systemcentercentral%2Fblogs+%28Blogs+at+System+Center+Central%29

This management pack will give you a deeper insight into your virtual machines including recognising which platform (Hyper-V, VMware & Citrix), which hosts they're located on, if VM Tools are installed and the version.

(footnote. I actually wrote this post the day they released it, but forgot to post it /facepalm)

VMM 2012 SP1 (Beta) with Server 2012 Hyper-V Issue

After migrating my 2008 R2 cluster across to Server 2012 I was having problems adding in the cluster/hosts to SC2012 Virtual Machine Manager SP1.

The cluster could be discovered and imported fine, but a few minutes later would report back that it could no longer be managed.

It turns out that because on the rebuild I didn't make the hosts DC's (it's a testlab before someone comments on best practise!!) so the computer accounts got placed into an OU along with the normal servers that had a Group Policy applying some WinRM settings.
Unfortunately doing a winrm /qc or allowing VMM to check that WinRM is enabled doesn't set/check all the required settings are actually in place.

Having a look around I found the following TechNet forum post from Wes Kroesbergen who listed a registry key that's advisable to have set, plus the winrm commands to set all of the required settings.

http://social.technet.microsoft.com/Forums/en-US/virtualmachingmgrhyperv/thread/4bd9be4b-0ff9-46f3-bf32-1b7c1245c494


Wrap the below into a batch file (you can't use these commands via PowerShell before you try), run it on your Hyper-V hosts and you're good to go!

reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v LocalAccountTokenFilterPolicy /t REG_DWORD /d 1 /f

winrm set winrm/config/service/auth @{CredSSP='True'}
winrm set winrm/config/winrs @{AllowRemoteShellAccess='True'}
winrm set winrm/config/winrs @{MaxMemoryPerShellMB='2048'}
winrm set winrm/config/client @{TrustedHosts='*'}
winrm set winrm/config/client/auth @{CredSSP='True'}

SCVMM 2012 SP1 - Missing properties and unable to refresh VM

Giddy as a kid in a sweet shop I migrated my testlab cluster to Server 2012, migrated my VM's across and installed System Center 2012 Virtual Machine Manager SP1 to manage it.

Straight away I ran into some slightly strange issues.

1. Majority of the Auto Discovered VM's refused to see hardware configuration properties such as:
   Virtual Hard Disks and Controllers
   Virtual Network Cards
2. Majority of the Auto Discovered VM's left hardware configuration properties at defaults, such as:
   Processor Count and Memory Amount
   High Availability Status
3. Every time I tried to refresh a VM I would get the error:
   Error (2923)
   
   A malformed response was received while trying to contact the VMM agent on hyperv01.domain.local.
   Unspecified error (0x80004005)
   
   Recommended Action
   1) Ensure the VMM agent is installed and that the SCVMMAgent service is running.
   2) If the SCVMMAgent service is running, restart the service, and then verify that WS-Management is installed correctly and that the WinRM service is running.
   3) Check the manufacturers' Web sites to ensure that you have the latest drivers for your network adapters and other devices.

Looking on the Microsoft Connect site I found that someone called egilsk was having the same issues.

Thankfully, last Friday (05/10/12) Mark at Microsoft posted the following suggestion to try:

On the Hyper-V hosts run the following PowerShell commands

Import-Module Hyper-V

Get-WmiObject –Namespace “root\virtualization\v2” Msvm_PlannedComputerSystem | foreach{Get-VM –Id $_.Name} | Remove-VM –Force

Once this has been run on your hosts, refreshing the VM's should work successfully and all of the missing/default hardware config should be updated and displayed.

It looks like this is to do with the hosts having the VM stuck in a temporary state, even if brought across as a planned import or migration.

The problem is now at least known about and should be fixed/handled in the next release of the Service Pack.

Building the TESG Private Cloud Customer Experience Centre - Part 1

Every year my employer holds an event for customers (and potential new customers) to show case what we do and give customers a chance to meet our partner vendors.

This year, nicely coinciding with just after the System Center 2012 release, I landed the brilliant job of setting up something to demonstrate our System Center and Desktop expertise.

And so the concept of the Private Cloud and Optimised Desktop Customer Experience Centre was born.

The goal?

1. To showcase the full System Center 2012 suite
2. To showcase the interactions of each component and how they drive efficiencies
3. To showcase an elastic and easily scalable datacentre that can flex into the Public Cloud
4. To showcase the dynamic desktop with OS, Data, User and Application layers abstracted
5. To showcase BYOD and specifically desktop/application access on tablet devices

Over a couple of blog posts I'll aim to share some of the planning, thoughts and tips & tricks that went into building it.
What I'll not be doing is guides on how to install the different components as there are plenty of them out there, but I will post links to some relevant good guides.

My original test lab was made up of a couple of HP Proliant DL380 G7's with some shared space pinched off the corporate SAN, but as this was going to need to host a lot more and it would need to be "slightly" portable for attending events like the T360 it was time to purchase some upgrades.

1. More memory.  Upgrade from 64Gb per host to 128Gb
2. Dedicated Storage.  iSCSI SAN that would also allow me to show some of the VMM storage management features (N.B. More details on this later, plus some pitfalls to watch out for!)
3. Dedicated Switches.  To show SCOM network management & keep the environment self contained.
4. More NIC's.  The original environment only had 4 onboard NICs, not good enough.
5. Flight case to rack it all in to make it portable (kind of!)

Now that might sound slightly overkill for a test/demo environment.  However, I have a laptop which is quite capable of showing 2-3 of the System Center products at the same time, but this Customer Experience Center had to host the following:

• Active Directory
• Virtual Machine Manager
• Operations Manager
• Service Manager
• Configuration Manager
• Data Protection Manager
• Orchestrator
• App Controller
• SQL 2008 R2 Server
• SharePoint Enterprise Server
• Exchange
• Lync
• ForeFront UAG
• ForeFront TMG
• File Servers
• XenDesktop Mgt Server
• XenDesktop VDI Desktops
• XenApp Mgt Server
• XenApp App Servers
• Remote Desktop Session Hosts
• Remote Desktop Broker/Gateway/Licensing
• RDS/Hyper-V VDI Desktops
• Dedicated Win 7 Admin Workstations
• Citrix NetScaler VM Appliance
• App-V Sequencer Workstations

When you consider that all of this needs to be up and running at the same time, my laptop just wasn't going to cope!

So far this has spread out across 34 VM's and there's still more to come...

This is a quick example diagram that I drew up to show the Hyper-V layout

Once all the hardware components were installed and racked then Hyper-V was the first thing to tackle and all I can say is thank god for Aidan Finn and his blog: http://www.aidanfinn.com/

Lots of useful posts, for example: http://www.aidanfinn.com/?p=10311

I'm going to leave the rest for the next post, but I just want to mention something that came to light when I installed the first System Center component, Virtual Machine Manager.

This is a logical first place to start if you've got the chance to build a private cloud from scratch like I have as you can implement Service Templates for deploying your VM's to help structure the environment and provide servicing and scale out options.

However, I hit a problem almost straight away, I struggled to get it to see my storage provider.

Originally I was ordering a Dell Equalogic iSCSI SAN for the environment, but due to certain disks not being available and increased costs for alternatives I was suggested to look at a DotHill AssuredSAN 2332.

The first thing I did was ask/check it supported SMI-S protocol, which it did as this is what VMM requires for the new features.
However when trying to set it up in VMM, it soon came to light that it only supported SMI-S 1.3 whereas VMM requires version 1.5.

So lesson learnt, make sure that when checking specifications, especially SAN's that you check in detail, right down to the version number!

There is a useful table (I found this afterwards!) that details the supported arrays:
http://technet.microsoft.com/en-us/library/gg610600.aspx

Part 1 - Building the TESG Private Cloud Customer Experience Centre

Part 2 - Service Accounts & Permissions

Part 3 - Installation Guide Links
Part 4 - Partner Solutions & Extensions

Windows Server 2012 - Quick Change Overview List

There is a wealth of information coming out at the moment regarding Windows Server 2012 and rather than doing an indepth post of all the changes, I thought I'd just call out some of the most interesting changes and new features that have caught my attention.
I've also linked some useful sites that contain some more indepth discussions.

General Windows Changes

• Native NIC Teaming
• SMB 3.0
• Ability to switch between Server Core and Full UI
• Multi Server Management from Server Manager
• Metro interface (present, but not in your face at boot)
• Storage Spaces
• Everything doable in the UI can be done with PowerShell 

Infrastructure Changes

• Domain Controller Replication (Clone & Deploy a DC)
• Virtual Domain Controller Support
• Storage across Remote SMB 3.0 File Shares (SQL & VM's run from file shares)
• IP Address Management
• Dynamic Access Control (File & Folder permissions on steriods!)
• RemoteFX without a hardware GPU
• RemoteFX in RDS Sessions, not just VDI

Hyper-V Changes

• Shared Nothing Live Migration
• Live Storage Migration
• Multiple Concurrent Live Migrations
• Hyper-V Replica
• Extensible Virtual Switch
• SR-IOV
• Network Virtualisation (servers running on the same IP without conflict)

Hyper-V Scalability Changes

• 160 Logical Processors per Host
• 2Tb RAM per Host
• 32 vCPUs per VM
• 512Gb per VM
• 16Tb VHDx
• 63 Nodes per Cluster
• 4,000 VMs per Cluster

Hyper-V 3.0 versus vSphere 5
While comparing an unreleased product to a released product isn't exactly fair, I thought I'd include it for completeness sake.  Also it's the most common question that customers ask me.
This article has a fairly clear comparison table:
http://up2v.nl/2012/01/29/microsoft-hyper-v-3-0-compared-to-vmware-vsphere-5/


Information Links
http://www.aidanfinn.com/ - Massive resource for Hyper-V
http://workinghardinit.wordpress.com/ - Another huge Hyper-V resource

http://blogs.technet.com/b/server-cloud/archive/2011/10/11/windows-server-8-hyper-v-overview.aspx

http://www.microsoft.com/en-us/server-cloud/