Now that System Center Operations Manager no longer has that pesky Root Management Server role; a server role that in larger environments quickly became the choking point and made creating a fully Highly-Available SCOM-environment both complex and frustrating to maintain and with little gain at that. With that gone and the SDK Service, or Data Access Service, thriving on all of the Management Servers HA suddenly became pretty simple. All you have to do in SCOM2012 to make sure your management groups keep on kicking is to have at-least two Management Servers and your databases clustered.

This new distributed architecture does not only give easy HA, it also makes it possible to connect to the SDK-service—be it using the Operations Console or powershell to name two options—on any Management Server. This, in turn, provides for a completely new level of scalability. Choked on sessions? Deploy a new Management Server!

Anyway… given all this scalability and HA, would it not be nice if you could load-balance all these SDK-sessions you are going to be running from System Center Virtual Machine Manager, System Center Service Manager, System Center Orchestrator, regular scheduled powershell scripts and what-not?

Of course it would! And you can! The simple solution is to use the built-in Network Load Balancer (NLB for short) feature in Windows Server and that’s what we’re going to discuss in this post.
Before we go, I’d like to point to a great article written by Justin Cook that is covering most bases but in a less for-dummies manner. So, yeah… I suppose this is the for-dummies version then.

Enjoy!

Prerequisites

We need to have the Network Load Balancing feature installed on all our targeted Management servers. The quick way to do this is using command-line (Windows Server 2008 R2 or later?).

dism /online /enable-feature /featurename:NetworkLoadBalancingFullServer 

You also need a plan and some information about your new cluster. Make sure you have identified the following parameters before starting the configuration:

You can use this pre-flight table to take note of your IP-address, DNS Name and Server List.

Create a New Cluster

Open the Network Load Balancing Manager and create a new cluster.

In the “New Cluster” dialogue, connect to one of your Management Servers.

1. Enter the name of a management server
2. Click Connect
3. Select the network interface to use
4. Click Next

Select the settings on your first host in the cluster.

1. Make sure it’s the correct IP-address.
2. Click Next

Set the Cluster IP-address.

1. Click Add
2. Enter your Dedicated Cluster IP-Address and Subnet mask
3. Click OK
4. Click Next

If another IP-address is needed, like an IPv6 address, you simply repeat step 1-3 before proceeding to step 4.

Edit DNS Names and Cluster Operation Mode.

1. Select your Dedicated Cluster IP-address
2. Enter your chosen Dedicated Cluster DNS Name
3. Select Multicast mode
4. Click Next

Note: We are not going to delve into the Cluster Operation Mode in this guide, but this is what I use for Operations Manager 2012.
If you are interested, here’s the KB on the various settings: http://support.microsoft.com/kb/323437

Set your Port Rules and Affinity Settings.

1. Verify that Affinity is set to “Single”. If not, Click “Edit…” and adjust.
2. Click Finish

Note: “Single” affinity tells the cluster to always direct the same client to the same host if possible. This is required to be able to maintain sessions.
In the world of NLB, a “client” is an IP-address.

Post-Configuration

Now that you have a cluster configured you have to make sure your SDK-clients are able to resolve the dedicated cluster DNS-name. The one you picked in the pre-flight table.

To enable name resolution you have to add your cluster DNS-name to your DNS-zone and point it to your dedicated cluster IP-address. Make it an A-record and it should work.

If you intend to use the cluster name from outside the local network or subnet—Operation Consoles or Powershell sessions for example—you would also need to verify that the router is able to handle the multicast packages. I am by no means a network guy, but asking the person behind that “Don’t blame the network” sign to help you access a NLB cluster on network X from network Y usually works. One way to troubleshoot this is to ping the cluster DNS-name from one of the hosts. If that works but you are still unable to ping from another network or subnet, then it might be a router setting.

Adding Hosts to the Cluster

With the cluster configured and up-and-running you need to add the rest of the Management Servers. Repeat this section for each Management Server you wish to add to the load-balancing cluster.

In the Network Load Balancing Manager, right-click your cluster and select “Add Host To Cluster”.

Connect to your next Management Server to be added

1. Enter the servername of the Management Server (“host” in cluster terminology)
2. Click Connect
3. Select the IP-address of the host
4. Click Next

Verify your Host Parameters

1. Doublecheck the IP-address
2. Click Next

Verify the Port Rules

1. Make sure that Load is Equal and Affinity is Single
2. Click Finish

Final Verification

After each added host it would be proper to check if it was added correctly. The easiest way is to check their statuses in the Network Load Balancing Manager. Green is generally considered good and you want your hosts to be “Converged”.

Another way to verify functionality is to point your Operations Manager console to the Cluster DNS-name instead and connect. If you are in a lab or in an environment where it happens to be OK to shut down Management Servers you could try that as well.

Considering my note on routers in the Cluster Post-Configuration I guess it would be prudent to point out that you should test to launch SDK-sessions from all networks you intend to connect from to make sure that your routers are configured to handle these kinds of sessions.

Postlude

Now; as easy this may be I would personally argue that you should involve your network team before starting to deploy your load-balanced clusters. A little heads-up is always a good thing—I have noticed that network people rarely like surprises—and they might actually be able to help you all the way is you ask nicely. And maybe they’ll tell you right away that the routers need to be configured before-hand instead of giggling frantically in a corner at your feeble attempts to troubleshoot your fresh little cluster.

Soooo… have fun!

And remember; with great powers come great responsibility.

[Sheesh! This post got out of hand!]

A Disclimer

A Disclaimer

I have had serious doubts about actually writing this article for almost a year now for reasons that I will explain further on. But as others have discovered this “feature” as well–maybe “hack” would be a better name for it–I feel the need to explain how it works and also why you should not use it. Knowledge is power, and even if I advice against using this technique it is also a good way to understand how SCOM uses display-strings in management packs.

The Good News

The Good News

Yes, you can use parameter replacement in you AlertName.

With “parameter replacement” i mean using some kind of substitute text, or mnemonic if you like, that at run-time get translated into something useful. If you have written any kind of alert generating rules or monitors, you most like included something like $Data/Context/Property[@Name='SomeDataFromAPropertyBag']$ into your alert description.

In this dialog, you also have the possibility to set the Alert Name. And if you are lazy, like I am, you probably also noticed that it is impossible to insert any kind of dynamic data into that field as well. This is especially annoying when you are writing a management pack that needs to look different in the Alert Views in the console, and you want to monitor 50 different Events or Performance counters or Log entries that are pretty much the same apart from a Name or ID.
Of course I could not refrain from copy-pasting a $Data/Context…$ into the alert name only to realize that it simply is not being parsed and translated into the value of the specified parameter. Over time I have settled for a stand-point that it’s probably a performance issue and I have also used that as an argument for this apparent lack of simplicity that some of my customers have been questioning.

Two, maybe three, years later. Microsoft releases an update to the core agent monitoring packs. Much to my surprise, one performance monitor suddenly generated alerts with a dynamic performance value in the Alert Name. You know, that field that is not gettingt parsed I was mentioning in the earlier paragraph. It actually looked pretty bad and made it very much impossible to practice any kind of alert supression, but still. It actually had a parsed value in the Alert Name.
As the lack of this feature had me irked before, I exported the core MP and started reading through the XML to find out how they did it. To my surprise, it was actually pretty simple if you ditched the Authoring Console and used your trusty text-editor instead.

How To Do It

How To Do It

In simple terms, if you know your SCOM XML out-side-in, you add the parameters to your “Alert” and modify your DisplayString, the one under LanguagePacks, to call that parameter by it’s relative ID.

Just in case you happen to be a regular mortal, here’s the step-by-step guide.

In this example I have created a silly-simple Alert Generating EventLog rule in wich I have added some parameters to the description. It pretty much looks like this:

Saving the MP at this point I will have a new WriteAlert action with a bunch of parameters and a displaystring with a unique ID.
The resulting XML-code for the WriteAlert action looks like this:

<WriteAction ID="Alert" TypeID="Health!System.Health.GenerateAlert">
  <Priority>1</Priority>
  <Severity>2</Severity>
  <AlertName />
  <AlertDescription />
  <AlertOwner />
  <AlertMessageId>$MPElement[Name="TestMP.MyEventlogAlertRule.AlertMessage"]$</AlertMessageId>
  <AlertParameters>
    <AlertParameter1>$Data/EventDescription$</AlertParameter1>
    <AlertParameter2>$Data/LoggingComputer$</AlertParameter2>
    <AlertParameter3>$Data/UserName$</AlertParameter3>
    <AlertParameter4>$Data/EventNumber$</AlertParameter4>
  </AlertParameters>
  <Suppression />
  <Custom1 />
  <Custom2 />
  <Custom3 />
  <Custom4 />
  <Custom5 />
  <Custom6 />
  <Custom7 />
  <Custom8 />
  <Custom9 />
  <Custom10 />
</WriteAction>

In this slab of code, you’ve got the Parameters, their “relative ID” (the number in the Tag), and also the which is what you would need to search for among the displaystrings. Take note of the you want to use in your AlertName since you are going to need it later. A quick search for “TestMP.MyEventlogAlertRule.AlertMessage” will lead you to the actual message, which should looke similar to this:

<DisplayString ElementID="TestMP.MyEventlogAlertRule.AlertMessage">
  <Name>I want Parameter Replacement here:</Name>
  <Description>Event Description: {0}
{1}
{2}
{3}
  </Description>
</DisplayString>

You should have one of these for every language you have configured in your management pack.
As you may notice, the Description does not contain any XPaths of any kind but rather references to the parameters defined in the WriteAction. That would be the {X} thingies within the description tags. You may also notice that the AlertParameters starts from “1″ while the parameter references begin with “0″. In practice, that means that {0} in the displaystring definition equals to AlertParameter1 and so forth.

Now, with this in mind, adding a Parameter to AlertName if very simple and straight-forward. Simply decide which parameter your want to use and add it’s reference to the Name definition.
So, let’s say I have the above alert message and I want to include UserName in the Alert Name, I would change the <Name> tag to:

<Name>I want Parameter Replacement here:{2}</Name>

That’s it folks!
Save the XML, make sure you verify it and import it into your Operations Manager LAB and watch the magic. Just make sure you continue reading this post as the next part about the downsides is pretty important.

What's the Downside?

What’s the Downside?

The negative effects of using Parameter Replacement in AlertName really boils down to two main perspectives. These are, if I may say to, pretty major and is the reason I have been reluctant on posting this, sort of, how-to.

You are breaking your subscriptions and connectors!

Apparently, parameter replacement is not working in subscriptions and connectors, be it 3:rd-party, Universal, Orchestrator IPs et.c. Basically anything other than the Operations Console will not be able to handle your parameters in AlertName. This will give you a “{1} in {2} has received an error” instead of “veryimportant.log in C:\Logs\ImportantSystem\ has received an error” in anything except the Operations Console. Now, this might not be a bad thing if the Operations Console is your only interface to Operations Manager, but if you would like to forward these alerts to a mobile phone, IM user or a mail recipient it will be utterly useless. I have not verified this, but I am not sure if Alert Suppression is actually parsing the parameters as well. Also. If you, by means of connectors or System Center Orchestrator, are generating incidents from alerts the AlertName forwarded will be equally useless for anyone receiving it.

It’s killing your health perspective.

Might sound drastic, but I’m quite serious about this. Using Parameter Replacement in AlertName might be useful for alert generating rules, but should never be used in a monitor. The reason is pretty simple actually. If you are using a monitor, you are without a doubt interested in it’s state. Otherwise, you would not use a monitor, am I right? I cannot see how it would make sense to create a monitor that does not reflect the health of a specific checkpoint. Much less having a monitor generating alerts that might confuse any service desk operator about what monitor that caused the alert. Also–I have not tested this–I assume that a monitor would not update it’s AlertName once the alert are generated making it more difficult to troubleshoot. Alert generating rules are, to me, something I tend to use only when all other options fail for some reason and I do not like to monitor things to cannot have health. And since parameters in AlertName is nothing for monitors I also question whether it actually is usable at all.

Conclusion

The Conclusion

While this is an interesting techique and a good way to enjoy MP Authoring and learn about the behaviour of display strings and languages in you Management Packs, I would definitely consider it a mostly academical exercise. Only in extreme use-cases and situations would I consider it a viably option and while it might look good in the console, do be aware of the limitations. I have also not seen anything about it on MSDN, and I have serious doubts about it being a supported feature. I have personally been using it in transition to a “better” management pack to get alerting running quickly, yet a prefered solution would actually be using a custom datasouce and a decent macro-enabled text-editor–not to mention snippets–to bulk-generate your monitors and/or rules.

MP Authoring
is fun and powerful
– be resposible

Me and my apprentice is currently decommissioning an entire Management Group with a thousand (-ish) agents. Long story short, we got a new Management Group, migrated all the agents, added a couple of hundreds more, deployed a bunch of gateways and now we are shutting down the old one.

Now, uninstalling the old Management Group from all the agents is a breeze using SCCM and handling the few 20-ish servers that are left is not a biggie either. Shutting down ACS, however, is a different matter.

Although you do configure your forwarders using Operations Manager, removing the management group you were running ACS in does not mean the agents will shut down and disable the AdtAgent service or stop trying to forward audit events to your collector. Now, selecting 10 agents at the time and running the “Disable Audit Collection” task–in case you did not know, there’s a limitation on how many agents you can run a task on in the Operations Console–is not my idea of a jolly good day and since Powershell is a bucket of joy in comparison; here’s a script for you all!

DisableACSForwarders

It is zipped to avoid security alerts, but as with any script found on the internet I implore to to read the code before actually running it.

Anyway, you can use it in a couple of ways.

To run it interactively, just go to the directory where you unpacked it and run it. You will be requested to enter the FQDN of you Root Management Server and a wildcard search for ACS Forwarders.
For example:

PS C:\..\Scripts> .\DisableACSForwarders.ps1
Root Management Server: rms.teknoglot.local
ACS Forwarder name (wildcard): *.teknoglot.local

You can also run it with parameters (for scheduling?) like this:

PS C:\..\Scripts> .\DisableACSForwarders.ps1 rms.teknoglot.local *.teknoglot.local

If you need to run the task with different credentials there’s a switch for that. Just add -UseCredentials to the command and you will be prompted for it.
Like this:

PS C:\..\Scripts> .\DisableACSForwarders.ps1 -UseCredentials

As you might already have realized, the wildcard search does not require actual wildcards. If you only want to disable the ACS forwarder on a single machine, just enter it’s FQDN. As for what wildcards it will accept; anything supported by the powershell -like operator is valid.

[UPDATE!] You might get false failures when running the script on clustered machines because of a bug in the AC Management Pack

For the source code, read on!

## Using parameters for RMS and wildcard search
param(
 [string]$rootManagementServer = $(Read-Host -Prompt "Root Management Server"),
 [string]$filterAgents = $(Read-Host -Prompt "ACS Forwarder name (wildcard)"),
 [switch]$UseCredentials
)

## Make sure the $rootManagementServer variable is defined
while (-not $rootManagementServer) {
 Write-Host "`nRoot Management Server MUST be defined!" -ForegroundColor Red
 [string]$rootManagementServer = $(Read-Host -Prompt "Root Management Server")
}

if ($UseCredentials -eq $true) {
 $taskCredentials = Get-Credential
}

$startLocation = Get-Location
$checkPSSnapin = Get-PSSnapin | where {$_.Name -eq "Microsoft.EnterpriseManagement.OperationsManager.Client"}
if ($checkPSSnapin -eq $null) {
 Add-PSSnapin "Microsoft.EnterpriseManagement.OperationsManager.Client"
}
$result = Set-Location "OperationsManagerMonitoring::"
$result = New-ManagementGroupConnection -ConnectionString:$rootManagementServer
$result = Set-Location $rootManagementServer

$acsForwarderClass = Get-MonitoringClass -Name "Microsoft.SystemCenter.ACS.Forwarder"
$agentNames = Get-MonitoringObject -monitoringclass:$acsForwarderClass | where {$_.Path -like $filterAgents} | Select Path
$successfulTasks = @()
$skippedTasks = @()
$failedTasks = @()

$monitoringClass = Get-MonitoringClass -Name "Microsoft.SystemCenter.Agent"
if ($agentNames -ne $null) {
 Write-Host "`nFound"$agentNames.Count"ACS forwarders with wildcard $filterAgents"
 Write-Host "`n`nWaiting for 5 seconds to give you a chance to abort!`n"
 Start-Sleep -Seconds 5
 Write-Host "Time's UP!`n`n"
 $agentNames | ForEach-Object {
 $agentName = $_.Path
 Write-Host "Disabling Audit Collection on" $agentName
 $monitoringObject = Get-MonitoringObject -monitoringclass:$monitoringClass | where {$_.DisplayName -like $agentName}
 $agentTask = $monitoringObject | Get-Task | where {$_.Name -eq "Microsoft.SystemCenter.DisableAuditCollectionService"}
 if ($monitoringObject.IsAvailable -eq $true) {
 if ($credentials -eq $null) {
 $result = Start-Task -Task $agentTask -TargetMonitoringObject $monitoringObject
 } else {
 $result = Start-Task -Task $agentTask -TargetMonitoringObject $monitoringObject -Credential $taskCredentials
 }
 if ($result.Status -eq "Succeeded") {
 Write-Host "Operation Successful!" -ForegroundColor Green
 Write-Host `n
 $successfulTasks += $agentName
 } else {
 Write-Host "Operation failed." -ForegroundColor Red
 Write-Host $result.ErrorMessage
 Write-Host `n
 $failedTasks += $agentName
 }
 } else {
 Write-Host "The agent is unavailable, Skipping!"
 Write-Host `n
 $skippedTasks += $agentName
 }
 }
 Write-Host "Summary"
 Write-Host "`tSuccessful:`t"$successfulTasks.count
 Write-Host "`tFailed:`t`t"$failedTasks.count
 Write-Host "`tSkipped:`t"$skippedTasks.count
} else {
 Write-Host "`nNo ACS Forwarders found using wildcard $filterAgents"
}

Set-Location $startLocation

Enjoy!

I’ve had this little visio drawing lying around on my desktop for a while now and I thought that it might be a nice thing to share.

It is nothing ground breaking at all and all the information is available at the Operations Manager 2007 R2 Supported Configurations page on Technet, but I find the visual map easier to read and I use it personally to quickly look up all port openings for the most common components in Operations Manager.

It is missing a few components like ACS, AEM and XPlat, but I usually just look them up when needed.

Have fun!

For quite some time now I’ve had this idea spinning around in my head to write a couple of blog-posts about some of the more useful techniques available when building management packs. Many of these techniques are already described on MSDN and Technet- or other blogs as well as on various forums, but often no more than small bits and pieces of them and I have yet to see some humanly readable information about how to tie them together into a useful management pack. I say “humanly readable” because the information you do find online so far may be clear and somewhat easy to understand for someone with a system development background and a pretty good idea of how object oriented development models tend to work. But the real life System Center Operations Manager engineer–you know the one who get those “do you think we could monitor our …-system too?” questions a couple of times a week, you know… you (most likely, being here)–tend to have a completely different background. Yet as their OpsMgr environment grows, so does the demand for custom monitoring and all of a sudden the former server engineer are now also a developer. A developer who has never before had the need to grasp such abstract concepts as classes, instances, inheritance and who probably never before have had any reason whatsoever to write any XML code.

Purpose

My idea for this series of posts is to shed some light into the world of the authoring console and modules and cookdown and so forth. I am by no means an accredited author, but I will do my best to stay human in this venture and in plain english try to explain why and how you do certain things when going from Management Pack templates, rules, monitors and the safe haven that is authoring in the Operations Console into making your scripts resuable, easy to extend and prime for cookdown using the Authoring Console and XML.

The TG WinAutoSvc Management Pack

To give the series some kind of context and at the same time not only be a matter of examples I will base them on a fully functional management pack that discovers and monitors all Windows services that are set to automatic startup. I know there is other similar management packs out there but I haven’t fancied any one of them yet, and since I had the idea of writing this series I decided that building a new one would be a good way to go. Some of the interesting features with this management pack is:

• You will get an instance of the service classes for each and every service.
• It uses different classes for Own Process services and Shared Process services (svchost for example).
• Every service have a health state (you can use them in distributed applications).
• The service state monitors are inherited from their base classes, no coding neccesary.
• There is only one discovery script for all kinds of windows services.
• Extending the discovery to include different kinds of windows services, like kernel processes, is a matter of filtering.
• It is Open Source and licensed under the Eclipse Public License v1.

Most of these features will be described thoroughly in later posts in the series and as development of it progresses I will document what I do, how I do it and why I do it in certain ways. Hopefully you will learn something new through this and get closer to becoming that MP Dev the organization asks for.
In the mean time, feel free to download, look at the source code (which it by no means perfect) and try it out.

The TG WinAutoSvc monitoring management pack is available for download here:
http://code.google.com/p/tg-winautosvc/downloads/detail?name=TG.WinAutoSvc.xml

The latest revision of the source code is located here:
http://code.google.com/p/tg-winautosvc/source/browse/trunk/TG.WinAutoSvc.xml

A small word of advice though. If you implement this in your environment, remember that you probably have alot more automatic services than you would expect. Because of this, discovery is disabled by default.

Best of luck, and enjoy!

To make it clear right away, this is not going to be a “Building an SNMP Management Pack Tutorial” since there’s plentiful good ones out there already, and to be extra helpful I’m gonna include a few links right away:

• SNMP Setup and Simple Custom SNMP Discovery – Pretty much the basics
• SNMP Management Pack Example: NetApp Management Pack – Part 4 actually, but has the links to the other parts
• Creating SNMP Probe Based Monitors – No custom discovery, but a good and simple guide to SNMP Probes

It’s the second, the NetApp one, I’ve used as a guide to building the UPS management pack since it goes through the process of building your own filtered discovery using SystemOID to identify your hardware-classes and then building the monitors on top of those.

Let’s get to it

When building the discovery of my hardware classes I ran into problems. The discovery simply did not work. At first I got some strange errors about “invalid queries”, something that turned out to be related to me reading two guides–seriously though, pick one guide that is closest to what you want to achieve and stick to it–and mixing up the XPathQuery variables. Silly me.
I got those errors to go away and I was able to get a few objects to my base-class, but none of the hardware classes who was populated through the return value of an SNMP OID got discovered.
The only error I got this time was the following:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          2010-09-02 11:19:12
Event ID:      11001
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CENSORED
Description:
Error sending an SNMP GET message to IP Address XX.XX.XX.XX, Community String:=CENSORED, Status 0x6c.

One or more workflows were affected by this.

Workflow name: CENSORED.MP.CLASS.DISCOVERY
Instance name: CENSORED_DEVICENAME
Instance ID: {5C7EFB30-D885-8843-0DD7-EA86B4FD2311}
Management group: CENSORED

<IsWriteAction>false</IsWriteAction>
<IP>$Config/IP$</IP>
<CommunityString>$Config/CommunityString$</CommunityString>
<Version>1</Version>
<SnmpVarBinds>
	<SnmpVarBind>
		<OID>1.3.6.1.4.1.534.1.1.1.0</OID>
		<Syntax>0</Syntax>
		<Value VariantType="8"></Value>
	</SnmpVarBind>
	<SnmpVarBind>
		<OID>1.3.6.1.4.1.534.1.1.2.0</OID>
		<Syntax>0</Syntax>
		<Value VariantType="8"></Value>
	</SnmpVarBind>
	<SnmpVarBind>
		<OID>1.3.6.1.4.1.534.1.1.3.0</OID>
		<Syntax>0</Syntax>
		<Value VariantType="8"></Value>
	</SnmpVarBind>
</SnmpVarBinds>

The situation is basically that we’re monitoring customers through gateway servers connected to our central Operations Manager environment. To have a bit of redundancy we always put two (or more) gateway servers per site (or customer, really) and they, in turn, talks to a couple of central management servers. I guess a drawing would be nice, but I have no Visio on this computer. The gateways are manually configured to talk to different management servers and have the others configured for fail-over (through powershell) and since we’re talking about no more than a few handfuls (say 20-ish) it’s not a problem handling it that way.

Agents, on the other hand, are a different matter. Even though we try to spread them out somewhat evenly at deployment between the gateway servers at each site we still end up looking at a 3:2 ratio after a while and since agents do not automatically fail-over between gateway servers we need a way to fix that too.
So I wrote a little powershell script that takes a bunch of gateway servers (or management servers) as parameters, gathers all connected agents, spreads the agents evenly between the servers and configures the others as fail-over servers while at it.

It’s all pretty crude, but it works and you can download it from here: DistributeAgents.ps1
Save it somewhere on disk and call it from the Operations Manager Shell like this:

C:DistributeAgents.ps1 gateway01.customer.local,gateway02.customer.local,gateway03.customer.local

Yes, you should replace “C:” with whatever path you decided to save the script to and “gatewayXX.customer.local” with a real servername.

Ok, I’m a powershell freshman and I’m pretty sure you could do this a prettier way, but here’s the script:

Param([array]$CSVServerList)

$arrServerObject = @()
$arrAgentObject = @()

foreach($Server in $CSVServerList)
{
	$arrServerObject += Get-ManagementServer | where {$_.Name -eq $Server}
	echo "Looking for $Server"
}
$ServerCount = $arrServerObject.Count
if ($ServerCount -gt 1)
{
	echo "Found $ServerCount management servers"
} else {
	echo "Found only 1 (or less) management servers. Aborting..."
	Exit
}

echo "Getting agents..."
foreach ($Server in $arrServerObject)
{
	$arrAgentObject += Get-Agent | where {$_.PrimaryManagementServerName -eq $Server.Name}
}
$AgentCount = $arrAgentObject.Count
if ($AgentCount -gt 1)
{
	echo "Found $AgentCount agents"
	Start-Sleep -m 200
} else {
	echo "Found only 1 (or less) agents. Aborting..."
	Exit
}
$i = 0
foreach ($Agent in $arrAgentObject)
{
	if ($i -ge $ServerCount)
	{
		$i = 0
	}
	$arrTemp = @($arrServerObject | Where-Object {$_ -ne $arrServerObject[$i]})
	# $FailoverServers = $arrTemp -join ","
	Set-ManagementServer -AgentManagedComputer: $Agent -PrimaryManagementServer: $arrServerObject[$i] -FailoverServer: $arrTemp

	$arrTemp = $null
	$i++
}

I have used it on a couple of occasions now and have only discovered a problem with an error when one of the servers don’t have any agents at all (probably a new one), but the script still works so I haven’t really dived into it.
Now, as with all scripts you download on the ‘net it’s up to you to test it in a lab before shooting wildly among your in-production systems. I really can’t give any warranties that it won’t FSU royally at your place.

This time I’ve basically put parameter support in it to make it easier to use.

Here’s the script anyway.

Param($OldGW,$NewGW)

$OldMS= Get-ManagementServer | where {$_.Name -eq $OldGW}
$NewMS = Get-ManagementServer | where {$_.Name -eq $NewGW}
$agents = Get-Agent | where {$_.PrimaryManagementServerName -eq $OldGW}
$agents = $agents
"Moving " + $agents.count + " agents from " + $OldMS.Name + " to " + $NewMS.Name
Start-Sleep -m 200
Set-ManagementServer -AgentManagedComputer: $agents -PrimaryManagementServer: $NewMS -FailoverServer: $OldMS

To use it, create a textfile called ChangeGW.ps1 and paste the code into it. Save the file somewhere neat (maybe C:Scripts) for easy access. If you don’t feel like copy/pasting, you can download the script here.

To use it, open the Operations Manager Command Shell and type:
C:ScriptsChangeGW.ps1 <old.gatewayserver.dns.name> <new.gatewayserver.dns.name>

For example:

C:ScriptsChangeGW.ps1 gwserver01.domainname.local gwserver02.domainname.local

This seems to be  most common on Windows 2008 and i guess it’s because of the UAC and the fact that opening the Control Panel isn’t running in administrative mode.

To work around this you need to run the msiexec command on the correct installation GUID from an administrative command prompt.

Besides running through the registry to find the GUID, one of the easier ways is this:

1. Open an administrative command prompt.
2. run wmic product
3. Locate your product by its name, the GUID (looks a bit like this {25097770-2B1F-49F6-AB9D-1C708B96262A}) directly after that is the one you want. Copy it.
4. run msiexec /i <PASTEYOURGUIDHERE>
5. Modify the agent as pleased

That’s pretty much it. Good luck.

System Center Pack for: Message Queuing 3.0
Version: 6.0.6615.0
Released on: 12/14/2009

Message Queuing (also known as MSMQ) is a server application that enables applications to communicate across heterogeneous networks and systems that may be temporarily offline or otherwise inaccessible. Instead of an application communicating with a service on another computer, it sends its information to Message Queuing, which sends the information to a Message Queuing service on the target computer where it is made available to the other application. Message Queuing provides guaranteed delivery, efficient routing, security, and priority based messaging.

Now, what’s really interesting is what you will find in the MP Guide under “Supported Configurations”.

The Message Queuing Management Pack for Operations Manager 2007 is designed to monitor Message Queuing version 3 only.

The Message Queuing Management Pack supports the following platforms:

· Windows Server 2003

· Windows XP

The Message Queuing Management Pack also supports monitoring clustered MSMQ components.

Text coloration is obviously added by me to highlight the interesting part.

Finally MSMQ monitoring seems to be cluster aware, which might mean that the home-made pack i did to have those (numerous) queues covered could be passed on to the scrap-heap. This is also confirmed under “Changes in This Update”.

The December 2009 update to this management pack includes the following change:

· Fixed a problem when working with an instance of MSMQ in a Cluster. The MP is now able to discover and monitor public and private queues in a cluster.

· Fixed a problem when discovering the local and cluster instance of MSMQ. The MP is now able to discover and monitor both instances.

The confusing double RunAs profiles seems to have been cleaned up too (you only have to worry about one now) as well as fixing some sloppy mistakes in the previous scripts (no Option Explicit? C’mon Microsoft! You write the best practices, try to stick to them.) and generally improving display and documentation.

Gonna import this to our staging environment today and let it roll during the holidays.

Cheers! Oh, and happy holidays!

Download and documentation:
http://www.microsoft.com/downloads/details.aspx?FamilyId=1D2B4398-8BC2-4A43-850C-852EBB0D983B&displaylang=en&displaylang=en

<stdout>Generating certificate with hostname="COMPUTERNAME"

[/home/serviceb/TfsCoreWrkSpcRedhat/source/code/tools/scx_ssl_config/scxsslcert.cpp:198]

Failed to allocate resource of type random data: Failed to get random data - not enough entropy

</stdout><stderr>error: %post(scx-1.0.4-248.i386) scriptlet failed, exit status 1

</stderr><returnCode>1</returnCode>

<DataItem type="Microsoft.SSH.SSHCommandData" time="2009-08-05T11:15:01.5800358-04:00" sourceHealthServiceId="0EB1D6DA-202C-7FC5-3D46-BDBB9208547D"><SSHCommandData><stdout>Generating certificate with hostname="COMPUTERNAME"

[/home/serviceb/TfsCoreWrkSpcRedhat/source/code/tools/scx_ssl_config/scxsslcert.cpp:198]

Failed to allocate resource of type random data: Failed to get random data - not enough entropy

</stdout><stderr>error: %post(scx-1.0.4-248.i386) scriptlet failed, exit status 1

</stderr><returnCode>1</returnCode></SSHCommandData></DataItem>

But first, a little background on the actual “problem”. To generate the certificate, the entropy needs to be high enough to generate random data for the certificate creation. Without the certificate, the OpsMgr agent won’t be able to open up communications with the MS. So, what creates this entropy we need? Bluntly put, a selection of hardware components that are likely to produce non-predictable data. Like a keyboard, mouse and a monitor or videocard. Of course, there’s a lot more to it, but we really don’t need to know this. What we need to know is that there has to be a “bit bucket” of more than 256bytes of entropy for the certificate creation process to succeed. We also need to know that more enterprise-ish servers, like rack- or blade-servers tend to be void of things like directly attached keyboards, mouses and monitors that the linux kernel needs to be able to generate entropy. And herein lies the problem. If you have a new server that is not in full service (likely since we are trying to deploy the monitoring on it) which means that there’s not much random data flowing through the hardware and there’s no keyboard or mouse or monitor connected to it there is quite the risk that the system entropy is going to be very low. Of the linux systems that I have been deploying OpsMgr agents to, about half have failed because of “Not enough entropy”. So, here’s the steps I usually takes to ensure that discovery works. I use PuTTY to connect to the soon-to-be-monitored servers. This guide also assumes that you have SU rights on the system since all of these steps (except #1) needs it.

1. Check you current entropy

   cat /proc/sys/kernel/random/entropy_avail

   Is it less than, or close to, 256? It probably is. If you don’t feel like connecting a mouse and start wiggling it around—not really feasible in a data center—and see if the entropy increases, you can generate your own random data.

2. Generate you own random data.
   Be advised that this forced entropy will not be as random as the system-created on and thus not as secure. How much more insecure it is, I don’t know, and quite frankly I prefer to have my systems monitored yet slightly less secure than not monitored at all. Anyway, you can force your own random data by running:

   dd if=/dev/urandom of=~/.rnd bs=1 count=1024

   This creates a .rnd file with 1024B of random data that the certificate creation process will use instead of the system entropy if the file exists.

3. Uninstall and re-discover
   The first failed attempt of discovery will most likely leave a non-working agent installation that we have to remove. Otherwise we will just be stuck with an “Access Denied” error. Run:

   rpm –e scx

   Now, try to discover the system again.

4. Failed again?
   Try generating the certificate manually by running:

   /opt/microsoft/scx/bin/tools/scxsslconfig -f –v
   /opt/microsoft/scx/bin/tools/scxadmin –restart

   Retry discovery again.

5. Still fails?
   Uninstall the agent once more as instructed in step 3.

Stese steps have solved my problems 100% on both SUSE and RedHat and hopefully they will help you too.

Interestingely enough, these problems seems to be connected to some changes in the 2.6 kernel and basically everything that uses SSL-ish certificates will be affected. Even though the symptoms may be a bit more subtle, like time-outs and disconnects. For “headless” servers like those I usually to administer where the random data tend to be much lower, there’s even specialised hardware whose sole purpose is to generate random data, like the Entropy Key. I have also been told that new servers is likely to be equipped with entropy chipsets to make sure that there’s chaos enough to avoid these new-found oddities.

Sources:
http://social.technet.microsoft.com/Forums/en-US/crossplatformsles/thread/f94ec905-23ac-4444-b9f8-644fec3ae357

http://www.askrenzo.com/oracle/SCOM/SCOM_discovering_nodes.html

The update basically contains support for x64 that was missing in the previous release.

The Configuration Manager 2007 SP2 Management Pack adds support for monitoring Configuration Manager 2007 SP2 in a 64-bit environment with Operations Manager 2007 R2 or Operations Manager 2007 SP1 with hotfix (KB971541) installed. This enables the Configuration Manager 2007 SP2 Management Pack to work with either the 32-bit or the 64-bit Operations Manager 2007 agent. Except for the 64-bit support, the other features and guidance for Configuration Manager 2007 Management Packs remain intact.

(coloration added by me)

Read more and download here:
http://www.microsoft.com/downloads/details.aspx?FamilyID=a8443173-46c2-4581-b3b8-ce67160f627b

Besides SUSE 11 support, here’s the short overview.

The System Center Operations Manager 2007 R2 Cross Platform Update adds fixes for a defunct process issue on Unix/Linux Servers, as well as, adds support for SUSE Linux Enterprise Server 11 (both 32-bit and 64-bit versions) and Solaris Zone support.
Feature Summary:
The System Center Operations Manager 2007 R2 Cross Platform Update supports the monitoring of Unix/Linux Servers including:

• Monitoring of SUSE Linux Enterprise Server 11 servers (both 32-bit and 64-bit versions)
• Support of Solaris Zones
• Fix for defunct Process issue
• The Cross Platform Agent may not discover soft partitions on Solaris systems. Therefore, the disk provider may be unloaded, and the Cross Platform Agent may stop collecting information from the system disks.
• The Cross Platform Agent may not restart after the AIX server reboots.

The latest versions of all the Operations Manager 2007 R2 Unix/Linux agents are included in this update.

Perfect timing, I must say, since I really need this today.

Update:
This is no small MP-update, which probably is the reason that we do not find it in the MP Catalog, but a ~250MB OpsMgr R2 Software Update. You need to run this on all Operations Manager Servers (RMS/MS, GW?) since it actually updates many of the agent Cross Platform binaries. It does add a new MP för SUSE 11 that you have to import from disk if you need it.

So, the installation goes somewhat like this:

1. Install the Software Update (pick the right Architecture) on all OpsMgr R2 Servers
2. Import the SUSE 11 MP if necessary
3. Re-discover your Unix/Linux machines.

Files updated in this update for R2:

• .Microsoft.Enterprisemanagement.UI.Administration.dll (Version 6.1.7043.1)
• .AgentManagementUnixAgentsscx-1.0.4-248.aix.5.ppc.lpp.gz
• .AgentManagementUnixAgentsscx-1.0.4-248.aix.6.ppc.lpp.gz
• .AgentManagementUnixAgentsscx-1.0.4-248.hpux.11iv2.ia64.depot.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.hpux.11iv2.parisc.depot.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.hpux.11iv3.ia64.depot.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.hpux.11iv3.parisc.depot.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.rhel.4.x64.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.rhel.4.x86.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.rhel.5.x64.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.rhel.5.x86.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.sles.10.x64.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.sles.10.x86.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.sles.9.x86.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.solaris.10.sparc.pkg.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.solaris.10.x86.pkg.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.solaris.8.sparc.pkg.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.solaris.9.sparc.pkg.Z

Files added:

• Microsoft.Linux.SLES.11.MP

All in all, the update contains the following fixes:

• KB969342
• KB973583
• Q954049
• Q956240

Most significant updates, according to me, would seem to be:

Fixed an issue that was previously preventing all rules related to agentless exception monitoring from generating alerts.

Added the rule “Collects Opsmgr SDK ServiceClient Connections” to collect the number of connected clients for a given management group. This data is shown in the view “Console and SDK Connection Count” under the folder “Operations ManagerManagement Server Performance”.

Updated a number of monitors and rules to ensure that data is reported to the correct management group for multihomed agents.

Fixed the configuration of the rule “IIS Discovery Probe Module Execution Failure” to so that the parameter replacement will now work correctly for alert suppression and generating the details of the alert’s description.

The rest is mostly polishing, fine-tuning and complementary updates. Nothing really ground-breaking here, but still a welcome update.

Download at: http://www.microsoft.com/downloads/details.aspx?FamilyID=61365290-3c38-4004-b717-e90bb0f6c148

Here’s basically what happens:

• You tell Operations Manager to change Primary Management Server for AGENTX from GW1 to GW2.
• The SDK Service (i guess) tells GW1 that “You’re no longer the Primary Management Server for AGENTX”
• GW1 acknowledges this and stops talking to AGENTX. And I mean Completely stops talking to AGENTX.
• OpsMgr then tells GW2 to start accepting communication from AGENTX.
• OpsMgr tries to tell AGENTX that it should talk to GW2 since GW1 won’t listen.

Spotted the problem?
This modus operandi probably works when agents are on the same network and in the same domain where fail-over is sort of automatic. The problem we are facing now is that the server are telling the Gateway to stop accepting communications to and from the agent before the agent is notified that there is a new Gateway server to talk to. The agent will continue to talk to GW1 but will be completely ignored and you will probably start seeing events in the Operations Manager eventlog on GW1 with EventID 20000.

How do I get around this little feature then?

No matter if you found this article after running into the mentioned troubles or if you are googling ahead of time to be prepared, the fix is the same and consists of a few powershell scripts. These scripts are out there allready, but in different contexts, hence this post.

First step: Install the new Gateway

Documentation on this from Microsoft is good enough, but here’s the short version.

1. Verify name resolution to and from Gateway server and Management Server
2. Create certificate for the Gateway server
3. Approve the Gateway server
4. Install Gateway server
5. Import certificates on Windows system
6. Run MOMCertImport.exe on Gateway server to add the certificate into Gateway server configuration
7. Wait

The wait is for the gateway server to get all needed configuration from RMS and to download all neccesary management packs, run all the discovery scripts and so on. When the Operations Manager event log has calmed down a bit, move to step two.

Second step: Configure Agent Failover

Connect to an Operations Manager Command Shell. Any will do, as long as it’s connected to the correct Management Group.
Then run the following script:

$primaryGW= Get-ManagementServer | where {$_.Name -eq 'GW2.domain.local'}
$failoverGw = Get-ManagementServer | where {$_.Name -eq 'GW1.domain.local'}
$agents = Get-Agent | where {$_.primarymanagementservername -eq 'GW1.domain.local'}
Set-ManagementServer -AgentManagedComputer: $agents -PrimaryManagementServer: $primaryGW -FailoverServer: $failoverGw

Remember to change “GW1.domain.local” to you OLD Gateway servername and “GW2.domain.local” to your NEW Gateway servername.
If you don’t know powershell, this script basically configures all agents using the old Gateway to use the new one as primare, but keep the old one as a fail-over server. The Gateways will still get to know the changes before the agents, but since the old on is still listening to the agents (though, as the fail-over host) it will be able to tell them to go to the new one, GW2.