SCOM Console Install on VMM Server Causes Service Crash

Had a strange one today.

Customer had tried previously to setup the connection between VMM and SCOM but made some mistake somewhere along the line and then uninstalled the SCOM console without removing the connection as they said the console was causing the VMM service to constantly crash.

Having not seen that behaviour before and slightly doubting it somewhat I re-installed the console and sure enough was prevented from accessing the VMM console as the service was crashing.

Just as an added check, I tried running some PowerShell commands to check it wasn't a GUI issue only to be created by error messages complaining that the VMM service wasn't running or accessible.

So I uninstalled the console again which allowed me access back to VMM and running the Get-SCOpsMgrConnection showed me the broken connection.  However attempts to remove via the console or PowerShell were both met by errors telling me I needed the SCOM console installed first in order to manage the connection.  Arh.. slight problem...

After checking everything I could think of (SPN's, SCP's, Service Accounts etc etc) and not finding anything that stood out (Including nothing useful in the event logs) I thought I'd try a timing trick.

So I opened up a SCVMM PowerShell window ready, kicked off the SCOM console install again and then repeatedly spammed Remove-SCOpsMgrConnection -Force and wouldn't you know it after a few messages saying the SCOM console must be installed, just before the install completed the command completed successfully and removed the broken connection.  More to the point the SCOM console installation completed and the VMM service didn't crash.

After recreating the connection everything remained stable, but even though the create connection job ran successfully, the following error was present in the connector details:

“Operations Manager discovery failed with error: “Exception of type ‘Microsoft.VirtualManager.EnterpriseManagement.common.discoverydatainvalidrelationshipsourceexceptionOM10’ was thrown.

This is because the SCOM connection was created with the PRO-Tips enabled but without a SCOM monitoring agent deployed to the VMM Server.
Easily fixable, just untick the PRO and Maintenance Mode connection settings, deploy a SCOM agent to the VMM server and once the agent is installed and reporting, re-enable the options.

Problems Clustering Virtual Machines on Windows Server 2012 Hyper-V

I was re-building our lab environment at work the other week in preparation for our big Summit13 event, that and the lab had been trashed over the last year...

As part of the re-build I had decided to implement a couple of virtual machine clusters, one for a scale-out file server and one as a SQL cluster.

I'd deployed the virtual machines for the cluster nodes using Service Templates in SCVMM and as part of that template chosen to use an availability set to ensure the VM's were separated across hosts (a cluster doesn't provide much High Availability if they all reside on the same host that's failed!).

When I started to create the cluster I ran straight into a problem with the Failover Cluster Manager reporting problems due to timeouts when creating the cluster.

Creating a single node cluster worked fine, but then would again fail when trying to add in another node.

I happened to put one of the Hyper-V hosts into maintenance mode for something and it migrated all the VM's onto the same host, at which point creating the cluster worked flawlessly, yay!

However, when the Hyper-V host came back out of maintenance mode and the availability sets kicked in during optimisation forcing a VM node back onto a separate physical host, the clusters broke again, not yay :(

So after some Googling Binging about and a shout on Twitter (Thanks @hvredevoort and @WorkingHardInIT) an issue with Broadcom NICs was brought to my attention and I came across this TechNet Forum post talking about the same issue.

Sophia_whx suggested trying to use Disable-NetAdapterChecksumOffload on the NICs to help with the issue.

So, first things first.  Use the Get-NetAdapterChecksumOffload to see just what the configuration was and sure enough Checksum Offload is enabled for just about all services across the majority of the NICs.

I then used the Disable-NetAdapterChecksumOffload * -TcpIPv4 command which resulted in this:

 

A reboot later and then perform it on the second host and whoa....

For some reason, the virtual switch really didn't like having that done to it.

I wish I had some screenshots, but I went into "get it fixed fast" mode.

 

Basically the switch via powershell was showing as up, the NIC Teaming GUI was showing it down and all the bound adapters as failed. SCVMM had lost all configuration for the switch altogether.

Deleting the switch from SCVMM didn't delete it from the host, but brought it back to life on the host but was missing in SCVMM.  SCVMM then wouldn't redetect it or let you build it again as it was still there, apparently???

 

I had to manually remove the team from a remote NIC Teaming GUI (I could of PowerShell'd it I know!) and then recreated via SCVMM.
 

Anyway... at first this looked to have fixed the clustering within virtual machine issues, but it only delayed the symptoms i.e. it took longer to evict nodes and randomly brought them back online.

 

So next I tried disabling Checksum Offload for all services, being careful not to touch the Virtual Switch this time.

 

Rather than going adapter by adapter I used the following command:

 

Get-NetAdapter | Where-Object {$_.Name -notlike "Converged*"} | Disable-NetAdapterChecksumOffload

 

This resulted in the Checksum Offload being disabled for the various services as shown, except for my virtual switch.

After doing this on the other host and giving them a reboot, my clustered virtual machines appear to be nice and stable when split across physical hosts. Yay! Problem fixed.

Just as another side note about Broadcom adapters, there have also been reports of performance issues when the Virtual Machine Queue (VMQ) setting is enabled, despite it being a recommended setting.

A quick check of my hosts showed it was enabled:

Another quick PowerShell line later and it wasn't:

Get-NetAdapterVmq -InterfaceDescription Broad* | Disable-NetAdapterVmq

SCVMM - Delete IP Pool

Quick PowerShell snippet for handy reference when I'm playing in the lab and need to delete an IP Pool:

##Display all IP's and the VM's they are assigned to:
$ippool = Get-SCStaticIPAddressPool "Internal Network"
Get-SCIPAddress -StaticIPAddressPool $ippool | ft -property Address,Description,State

##Return all the IP's for that pool ready to remove the pool
$ip = Get-SCIPAddress -StaticIPAddressPool $ippool
$ip | Revoke-SCIPAddress

Installing the SCVMM (Non SP1) console on Windows 8

If you find yourself running the new swanky Windows 8 OS on your desktop that you use for administration, you might run into a problem with trying to install the System Center 2012 Virtual Machine Manager console for remote admin purposes.

When Service Pack 1 is released (very soon) this isn't too much of an issue, except when you may want to administer a non-SP1 SCVMM setup.

So, there's a "hack" to install the SCVMM console, without SP1, on a Windows 8 machine.

** Firstly - disclaimer - This is in no way shape or form supported and you are to run this at your own risk, it's not my responsibility if it kills your infrastructure!!! **


Steps:

1. Grab the SP1 Beta install folder for SCVMM and copy it locally to a folder called VMMSP1
2. Copy the Non-SP1 install files locally to a folder too called VMMNONSP1
3. Rename the setup folder in either i386 or amd64 depending on your OS in the SP1 media to setupOrig.  In the screen shot below I've done this for the i386 folder.
   
   
   
4. Copy the setup folder for the relevant os (x86/x64) from the non-sp1 folder to the SP1 folder
   
   
   
5. Edit the PrerequisiteInputFile1033.xml in the setup folder copied across to the SP1 folder and look for the Win7 section below:
   
   <LogicDelegate LogicType="or" DelegateId="OSVersion-win7-Fail"
    DelegateNameSpace="Microsoft.VirtualManager.SetupFramework.BuiltInDelegates,SetupFramework"
   DelegateName="RequiredOperatingSystem">
   6.1.7600.0,6.1.9999.9999,Win32NT,WinNT,2
   </LogicDelegate>
6. Add this new section underneath the Win7 section:
   
   <LogicDelegate LogicType="or" DelegateId="OSVersion-win8-Fail"
    DelegateNameSpace="Microsoft.VirtualManager.SetupFramework.BuiltInDelegates,SetupFramework"
   DelegateName="RequiredOperatingSystem">
   6.2.9200.0,6.2.9999.9999,Win32NT,WinNT,2
   </LogicDelegate>
   
   
   
7. Go back to the root level for the OS "bitness" you're doing this for and run SetupVMM.exe
   
   
   
8. Don't worry when you see the splash screen as this will still say SP1, just click "Install"
   
   
9. This will then launch the Non-SP1 install and allow you to step through the options as normal to install the admin console.
   
   
   

As I said before, use this at your own risk and test fully in a lab environment first.  I will not be held responsible for what may go wrong, there's probably a reason Microsoft blocks the install, but I've always hated to be told no so had to try this.

Thanks to Dirk Flakowski for getting me side-tracked on this one today!
http://social.technet.microsoft.com/Forums/en-US/virtualmachinemanager/thread/07576dd4-bce2-442a-8af8-40213228406c

SCVMM 2012 SP1 - Missing properties and unable to refresh VM

Giddy as a kid in a sweet shop I migrated my testlab cluster to Server 2012, migrated my VM's across and installed System Center 2012 Virtual Machine Manager SP1 to manage it.

Straight away I ran into some slightly strange issues.

1. Majority of the Auto Discovered VM's refused to see hardware configuration properties such as:
   Virtual Hard Disks and Controllers
   Virtual Network Cards
2. Majority of the Auto Discovered VM's left hardware configuration properties at defaults, such as:
   Processor Count and Memory Amount
   High Availability Status
3. Every time I tried to refresh a VM I would get the error:
   Error (2923)
   
   A malformed response was received while trying to contact the VMM agent on hyperv01.domain.local.
   Unspecified error (0x80004005)
   
   Recommended Action
   1) Ensure the VMM agent is installed and that the SCVMMAgent service is running.
   2) If the SCVMMAgent service is running, restart the service, and then verify that WS-Management is installed correctly and that the WinRM service is running.
   3) Check the manufacturers' Web sites to ensure that you have the latest drivers for your network adapters and other devices.

Looking on the Microsoft Connect site I found that someone called egilsk was having the same issues.

Thankfully, last Friday (05/10/12) Mark at Microsoft posted the following suggestion to try:

On the Hyper-V hosts run the following PowerShell commands

Import-Module Hyper-V

Get-WmiObject –Namespace “root\virtualization\v2” Msvm_PlannedComputerSystem | foreach{Get-VM –Id $_.Name} | Remove-VM –Force

Once this has been run on your hosts, refreshing the VM's should work successfully and all of the missing/default hardware config should be updated and displayed.

It looks like this is to do with the hosts having the VM stuck in a temporary state, even if brought across as a planned import or migration.

The problem is now at least known about and should be fixed/handled in the next release of the Service Pack.

System Center 2012 Update Rollup 1 - SCVMM & App Controller

Sneaking in quietly under the radar is the first update for the just released System Center 2012.

N.B. Don't be alarmed that there are updates available already for a product that has only been released, the development cycle and the aligning of the different System Center components mean that Virtual Machine Manager was slightly ahead in development which means they've been able to focus on updates and service packs already (a CTP Preview of SP1 is already available, see the bottom of this post)

This update focuses purely on Virtual Machine Manager, both server side and console as well as App Controller.

The knowledge base for Update Rollup 1 can be found here:
http://support.microsoft.com/kb/2686249

The downloads can be found on Microsoft Update or manually here:

Virtual Machine Manager Server Update (KB2663959)http://catalog.update.microsoft.com/v7/site/Search.aspx?q=2663959
Virtual Machine Manager Console Update (KB2663960)http://catalog.update.microsoft.com/v7/site/Search.aspx?q=2663960
Virtual Machine Manager Self-Service Portal Update (KB2663961)http://catalog.update.microsoft.com/v7/site/Search.aspx?q=2663961
App Controller Update (KB2677596)
http://catalog.update.microsoft.com/v7/site/Search.aspx?q=2677596

In total we're looking at about 16 issues that have been fixed in this rollup.

It's also worth noting that after performing the server side update, the VMM agents on your hosts will also require updating.

To update the VMM agent on the managed computers, follow these steps:

1. Open the VMM Console.
2. Click Fabric, and then select Servers.
3. Under Managed Computers, select the servers, and then click Update Agent

How to move a SCVMM 2012 database to another SQL server

I'm not exactly re-inventing the wheel here as it's the same process for Virtual Machine Manager 2012 as it was for 2008 R2, but I thought I'd post about it anyway.

This isn't a common thing to need to do, but I found myself in the position today where I wanted to move the database that was on a local SQL install on a SCVMM 2012 server up onto a new SQL Cluster I had just finished building.

So these are the steps I took to move the database:

1. Take a snapshot of the server (assuming it's a VM)
2. Take a backup of the existing VMM DB
3. Uninstall VMM and choose the Retain Database option
4. Detach the VMM DB from the local SQL instance
5. Copy the VMM DB files across to the new SQL Server
6. On the remote SQL server, attach the VMM db
7. Re-install SCVMM 2012 and when prompted, point it at the new SQL server, selecting to use an existing database (the one you imported) 

Easy!  Now you're free to uninstall SQL as and when.