OpsMgr « teknoglot:

Tag Archives: OpsMgr

Load-balanced SCOM2012 SDK Services for Network Illiterates [#opsmgr, #nlb]

Posted on May 18, 2012 by Sam T

Prelude

Now that System Center Operations Manager no longer has that pesky Root Management Server role; a server role that in larger environments quickly became the choking point and made creating a fully Highly-Available SCOM-environment both complex and frustrating to maintain and with little gain at that. With that gone and the SDK Service, or Data Access Service, thriving on all of the Management Servers HA suddenly became pretty simple. All you have to do in SCOM2012 to make sure your management groups keep on kicking is to have at-least two Management Servers and your databases clustered.

This new distributed architecture does not only give easy HA, it also makes it possible to connect to the SDK-service—be it using the Operations Console or powershell to name two options—on any Management Server. This, in turn, provides for a completely new level of scalability. Choked on sessions? Deploy a new Management Server!

Anyway… given all this scalability and HA, would it not be nice if you could load-balance all these SDK-sessions you are going to be running from System Center Virtual Machine Manager, System Center Service Manager, System Center Orchestrator, regular scheduled powershell scripts and what-not?

Of course it would! And you can! The simple solution is to use the built-in Network Load Balancer (NLB for short) feature in Windows Server and that’s what we’re going to discuss in this post.
Before we go, I’d like to point to a great article written by Justin Cook that is covering most bases but in a less for-dummies manner. So, yeah… I suppose this is the for-dummies version then.

Enjoy!

Prerequisites

We need to have the Network Load Balancing feature installed on all our targeted Management servers. The quick way to do this is using command-line (Windows Server 2008 R2 or later?).

dism /online /enable-feature /featurename:NetworkLoadBalancingFullServer

You also need a plan and some information about your new cluster. Make sure you have identified the following parameters before starting the configuration:

A Dedicated Cluster IP-Address:
A Dedicated Cluster DNS Name:
A list of SCOM2012 Management Servers:

You can use this pre-flight table to take note of your IP-address, DNS Name and Server List.

Create a New Cluster

Open the Network Load Balancing Manager and create a new cluster.

In the “New Cluster” dialogue, connect to one of your Management Servers.

Enter the name of a management server
Click Connect
Select the network interface to use
Click Next

Select the settings on your first host in the cluster.

Make sure it’s the correct IP-address.
Click Next

Set the Cluster IP-address.

Click Add
Enter your Dedicated Cluster IP-Address and Subnet mask
Click OK
Click Next

If another IP-address is needed, like an IPv6 address, you simply repeat step 1-3 before proceeding to step 4.

Edit DNS Names and Cluster Operation Mode.

Select your Dedicated Cluster IP-address
Enter your chosen Dedicated Cluster DNS Name
Select Multicast mode
Click Next

Note: We are not going to delve into the Cluster Operation Mode in this guide, but this is what I use for Operations Manager 2012.
If you are interested, here’s the KB on the various settings: http://support.microsoft.com/kb/323437

Set your Port Rules and Affinity Settings.

Verify that Affinity is set to “Single”. If not, Click “Edit…” and adjust.
Click Finish

Note: “Single” affinity tells the cluster to always direct the same client to the same host if possible. This is required to be able to maintain sessions.
In the world of NLB, a “client” is an IP-address.

Post-Configuration

Now that you have a cluster configured you have to make sure your SDK-clients are able to resolve the dedicated cluster DNS-name. The one you picked in the pre-flight table.

To enable name resolution you have to add your cluster DNS-name to your DNS-zone and point it to your dedicated cluster IP-address. Make it an A-record and it should work.

If you intend to use the cluster name from outside the local network or subnet—Operation Consoles or Powershell sessions for example—you would also need to verify that the router is able to handle the multicast packages. I am by no means a network guy, but asking the person behind that “Don’t blame the network” sign to help you access a NLB cluster on network X from network Y usually works. One way to troubleshoot this is to ping the cluster DNS-name from one of the hosts. If that works but you are still unable to ping from another network or subnet, then it might be a router setting.

Adding Hosts to the Cluster

With the cluster configured and up-and-running you need to add the rest of the Management Servers. Repeat this section for each Management Server you wish to add to the load-balancing cluster.

In the Network Load Balancing Manager, right-click your cluster and select “Add Host To Cluster”.

Connect to your next Management Server to be added

Enter the servername of the Management Server (“host” in cluster terminology)
Click Connect
Select the IP-address of the host
Click Next

Verify your Host Parameters

Doublecheck the IP-address
Click Next

Verify the Port Rules

Make sure that Load is Equal and Affinity is Single
Click Finish

Final Verification

After each added host it would be proper to check if it was added correctly. The easiest way is to check their statuses in the Network Load Balancing Manager. Green is generally considered good and you want your hosts to be “Converged”.

Another way to verify functionality is to point your Operations Manager console to the Cluster DNS-name instead and connect. If you are in a lab or in an environment where it happens to be OK to shut down Management Servers you could try that as well.

Considering my note on routers in the Cluster Post-Configuration I guess it would be prudent to point out that you should test to launch SDK-sessions from all networks you intend to connect from to make sure that your routers are configured to handle these kinds of sessions.

Postlude

Now; as easy this may be I would personally argue that you should involve your network team before starting to deploy your load-balanced clusters. A little heads-up is always a good thing—I have noticed that network people rarely like surprises—and they might actually be able to help you all the way is you ask nicely. And maybe they’ll tell you right away that the routers need to be configured before-hand instead of giggling frantically in a corner at your feeble attempts to troubleshoot your fresh little cluster.

Soooo… have fun!

And remember; with great powers come great responsibility.

[Sheesh! This post got out of hand!]

Categories: OpsMgr 2012 | Tags: Highly-Available, How-To, NLB, OpsMgr, Tutorial

Parameter Replacement in AlertName

Posted on April 9, 2012 by Sam T

No Comments

…and why you should not use it.

Bulk disable ACS Forwarders (with wildcards)

Posted on July 7, 2011 by Sam T

2 Comments

Here’s a little something-something for the wicked.

Me and my apprentice is currently decommissioning an entire Management Group with a thousand (-ish) agents. Long story short, we got a new Management Group, migrated all the agents, added a couple of hundreds more, deployed a bunch of gateways and now we are shutting down the old one.

Now, uninstalling the old Management Group from all the agents is a breeze using SCCM and handling the few 20-ish servers that are left is not a biggie either. Shutting down ACS, however, is a different matter.

Although you do configure your forwarders using Operations Manager, removing the management group you were running ACS in does not mean the agents will shut down and disable the AdtAgent service or stop trying to forward audit events to your collector. Now, selecting 10 agents at the time and running the “Disable Audit Collection” task–in case you did not know, there’s a limitation on how many agents you can run a task on in the Operations Console–is not my idea of a jolly good day and since Powershell is a bucket of joy in comparison; here’s a script for you all!

DisableACSForwarders

It is zipped to avoid security alerts, but as with any script found on the internet I implore to to read the code before actually running it.

Anyway, you can use it in a couple of ways.

To run it interactively, just go to the directory where you unpacked it and run it. You will be requested to enter the FQDN of you Root Management Server and a wildcard search for ACS Forwarders.
For example:

PS C:\..\Scripts> .\DisableACSForwarders.ps1
Root Management Server: rms.teknoglot.local
ACS Forwarder name (wildcard): *.teknoglot.local

You can also run it with parameters (for scheduling?) like this:

PS C:\..\Scripts> .\DisableACSForwarders.ps1 rms.teknoglot.local *.teknoglot.local

If you need to run the task with different credentials there’s a switch for that. Just add -UseCredentials to the command and you will be prompted for it.
Like this:

PS C:\..\Scripts> .\DisableACSForwarders.ps1 -UseCredentials

As you might already have realized, the wildcard search does not require actual wildcards. If you only want to disable the ACS forwarder on a single machine, just enter it’s FQDN. As for what wildcards it will accept; anything supported by the powershell -like operator is valid.

[UPDATE!] You might get false failures when running the script on clustered machines because of a bug in the AC Management Pack

For the source code, read on!

OpsMgr 2007 Connectivity Map

Posted on May 17, 2011 by Sam T

No Comments

I’ve had this little visio drawing lying around on my desktop for a while now and I thought that it might be a nice thing to share.

It is nothing ground breaking at all and all the information is available at the Operations Manager 2007 R2 Supported Configurations page on Technet, but I find the visual map easier to read and I use it personally to quickly look up all port openings for the most common components in Operations Manager.

It is missing a few components like ACS, AEM and XPlat, but I usually just look them up when needed.

Have fun!

Categories: OpsMgr 2007 | Tags: OpsMgr, Quick Reference

Introduction to TG WinAutoSvc v1

Posted on April 29, 2011 by Sam T

2 Comments

This entry is part 1 of 1 in the series MP Dev: TG WinAutoSvc

Background

For quite some time now I’ve had this idea spinning around in my head to write a couple of blog-posts about some of the more useful techniques available when building management packs. Many of these techniques are already described on MSDN and Technet- or other blogs as well as on various forums, but often no more than small bits and pieces of them and I have yet to see some humanly readable information about how to tie them together into a useful management pack. I say “humanly readable” because the information you do find online so far may be clear and somewhat easy to understand for someone with a system development background and a pretty good idea of how object oriented development models tend to work. But the real life System Center Operations Manager engineer–you know the one who get those “do you think we could monitor our …-system too?” questions a couple of times a week, you know… you (most likely, being here)–tend to have a completely different background. Yet as their OpsMgr environment grows, so does the demand for custom monitoring and all of a sudden the former server engineer are now also a developer. A developer who has never before had the need to grasp such abstract concepts as classes, instances, inheritance and who probably never before have had any reason whatsoever to write any XML code.

Purpose

My idea for this series of posts is to shed some light into the world of the authoring console and modules and cookdown and so forth. I am by no means an accredited author, but I will do my best to stay human in this venture and in plain english try to explain why and how you do certain things when going from Management Pack templates, rules, monitors and the safe haven that is authoring in the Operations Console into making your scripts resuable, easy to extend and prime for cookdown using the Authoring Console and XML.

The TG WinAutoSvc Management Pack

To give the series some kind of context and at the same time not only be a matter of examples I will base them on a fully functional management pack that discovers and monitors all Windows services that are set to automatic startup. I know there is other similar management packs out there but I haven’t fancied any one of them yet, and since I had the idea of writing this series I decided that building a new one would be a good way to go. Some of the interesting features with this management pack is:

You will get an instance of the service classes for each and every service.
It uses different classes for Own Process services and Shared Process services (svchost for example).
Every service have a health state (you can use them in distributed applications).
The service state monitors are inherited from their base classes, no coding neccesary.
There is only one discovery script for all kinds of windows services.
Extending the discovery to include different kinds of windows services, like kernel processes, is a matter of filtering.
It is Open Source and licensed under the Eclipse Public License v1.

Most of these features will be described thoroughly in later posts in the series and as development of it progresses I will document what I do, how I do it and why I do it in certain ways. Hopefully you will learn something new through this and get closer to becoming that MP Dev the organization asks for.
In the mean time, feel free to download, look at the source code (which it by no means perfect) and try it out.

The TG WinAutoSvc monitoring management pack is available for download here:
http://code.google.com/p/tg-winautosvc/downloads/detail?name=TG.WinAutoSvc.xml

The latest revision of the source code is located here:
http://code.google.com/p/tg-winautosvc/source/browse/trunk/TG.WinAutoSvc.xml

A small word of advice though. If you implement this in your environment, remember that you probably have alot more automatic services than you would expect. Because of this, discovery is disabled by default.

Best of luck, and enjoy!

Categories: OpsMgr 2007 | Tags: Management Pack, MP Development, OpsMgr

SNMP GET Errors in OpsMgr EventLog

Posted on September 2, 2010 by Sam T

2 Comments

I’ve been building a little SNMP Management Pack in the past few days to discover and monitor a bunch of PowerWare UPS’s, which turned out to take quite a lot more energy and time than expected. Mostly due to the facts that I am really bad with SNMP and how it works, I’ve never really looked into the inner working of building an SNMP management pack and also because we ran into a couple of errors preventing the discovery process to work alright.

To make it clear right away, this is not going to be a “Building an SNMP Management Pack Tutorial” since there’s plentiful good ones out there already, and to be extra helpful I’m gonna include a few links right away:

SNMP Setup and Simple Custom SNMP Discovery – Pretty much the basics
SNMP Management Pack Example: NetApp Management Pack – Part 4 actually, but has the links to the other parts
Creating SNMP Probe Based Monitors – No custom discovery, but a good and simple guide to SNMP Probes

It’s the second, the NetApp one, I’ve used as a guide to building the UPS management pack since it goes through the process of building your own filtered discovery using SystemOID to identify your hardware-classes and then building the monitors on top of those.

Let’s get to it

When building the discovery of my hardware classes I ran into problems. The discovery simply did not work. At first I got some strange errors about “invalid queries”, something that turned out to be related to me reading two guides–seriously though, pick one guide that is closest to what you want to achieve and stick to it–and mixing up the XPathQuery variables. Silly me.
I got those errors to go away and I was able to get a few objects to my base-class, but none of the hardware classes who was populated through the return value of an SNMP OID got discovered.
The only error I got this time was the following:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          2010-09-02 11:19:12
Event ID:      11001
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CENSORED
Description:
Error sending an SNMP GET message to IP Address XX.XX.XX.XX, Community String:=CENSORED, Status 0x6c.

One or more workflows were affected by this.

Workflow name: CENSORED.MP.CLASS.DISCOVERY
Instance name: CENSORED_DEVICENAME
Instance ID: {5C7EFB30-D885-8843-0DD7-EA86B4FD2311}
Management group: CENSORED

I went through all the other logical steps of troubleshooting an error like that which include double-checking firewall settings, OIDs, IP-addresses, allowed hosts and so forth. It wasn’t until I loaded the PowerMIB into a MIB Browser installed on the proxy machine (in this case a Management Server) I realized that there was no problem sending an SNMP GET to the UPS from that server. I launched Wireshark and had it listen to SNMP traffic between the UPS and the Management Server. The thing that struck me right-away was the fact that I could see the a bunch of “SNMP Get-Request” but no “SNMP Get-Response” which means that Operations Manager did send an SNMP GET but there was no response.

After a bit of intense staring i noticed what you see in the screenshot.

For some reason Operations Manager does not care about what SNMP version you configure when you do the initial discovery of a network device. Even if you do specify SNMP v1, you probes may very well be using SNMP v2c instead and in many cases that will result in these SNMP GET errors in the Operations Manager event log.

To avoid this, you haves to specify which SNMP version to use in your System.SnmpProbe according to the information provided here: http://msdn.microsoft.com/en-us/library/ee809331.aspx

Since I am such a nice guy, here’s an example of the working probe with the added line highlighted.

<IsWriteAction>false</IsWriteAction>
<IP>$Config/IP$</IP>
<CommunityString>$Config/CommunityString$</CommunityString>
<Version>1</Version>
<SnmpVarBinds>
	<SnmpVarBind>
		<OID>1.3.6.1.4.1.534.1.1.1.0</OID>
		<Syntax>0</Syntax>
		<Value VariantType="8"></Value>
	</SnmpVarBind>
	<SnmpVarBind>
		<OID>1.3.6.1.4.1.534.1.1.2.0</OID>
		<Syntax>0</Syntax>
		<Value VariantType="8"></Value>
	</SnmpVarBind>
	<SnmpVarBind>
		<OID>1.3.6.1.4.1.534.1.1.3.0</OID>
		<Syntax>0</Syntax>
		<Value VariantType="8"></Value>
	</SnmpVarBind>
</SnmpVarBinds>

That’s it. Working perfectly now.

Best of luck to you too.

Categories: OpsMgr 2007 | Tags: Errors, Management Pack, OpsMgr, TroubleShooting

“Load Balancing” Powershell Script for Operations Manager

Posted on April 15, 2010 by Sam T

3 Comments

I’ve been looking for at way to evenly distribute agents between Gateway Servers (or Management Servers for that matter, but I’ll stick to GWs this time) for some time but haven’t really got to fixing it myself until now.

The situation is basically that we’re monitoring customers through gateway servers connected to our central Operations Manager environment. To have a bit of redundancy we always put two (or more) gateway servers per site (or customer, really) and they, in turn, talks to a couple of central management servers. I guess a drawing would be nice, but I have no Visio on this computer. The gateways are manually configured to talk to different management servers and have the others configured for fail-over (through powershell) and since we’re talking about no more than a few handfuls (say 20-ish) it’s not a problem handling it that way.

Agents, on the other hand, are a different matter. Even though we try to spread them out somewhat evenly at deployment between the gateway servers at each site we still end up looking at a 3:2 ratio after a while and since agents do not automatically fail-over between gateway servers we need a way to fix that too.
So I wrote a little powershell script that takes a bunch of gateway servers (or management servers) as parameters, gathers all connected agents, spreads the agents evenly between the servers and configures the others as fail-over servers while at it.

It’s all pretty crude, but it works and you can download it from here: DistributeAgents.ps1
Save it somewhere on disk and call it from the Operations Manager Shell like this:

C:DistributeAgents.ps1 gateway01.customer.local,gateway02.customer.local,gateway03.customer.local

Yes, you should replace “C:” with whatever path you decided to save the script to and “gatewayXX.customer.local” with a real servername.

Ok, I’m a powershell freshman and I’m pretty sure you could do this a prettier way, but here’s the script:

Param([array]$CSVServerList)

$arrServerObject = @()
$arrAgentObject = @()

foreach($Server in $CSVServerList)
{
	$arrServerObject += Get-ManagementServer | where {$_.Name -eq $Server}
	echo "Looking for $Server"
}
$ServerCount = $arrServerObject.Count
if ($ServerCount -gt 1)
{
	echo "Found $ServerCount management servers"
} else {
	echo "Found only 1 (or less) management servers. Aborting..."
	Exit
}

echo "Getting agents..."
foreach ($Server in $arrServerObject)
{
	$arrAgentObject += Get-Agent | where {$_.PrimaryManagementServerName -eq $Server.Name}
}
$AgentCount = $arrAgentObject.Count
if ($AgentCount -gt 1)
{
	echo "Found $AgentCount agents"
	Start-Sleep -m 200
} else {
	echo "Found only 1 (or less) agents. Aborting..."
	Exit
}
$i = 0
foreach ($Agent in $arrAgentObject)
{
	if ($i -ge $ServerCount)
	{
		$i = 0
	}
	$arrTemp = @($arrServerObject | Where-Object {$_ -ne $arrServerObject[$i]})
	# $FailoverServers = $arrTemp -join ","
	Set-ManagementServer -AgentManagedComputer: $Agent -PrimaryManagementServer: $arrServerObject[$i] -FailoverServer: $arrTemp

	$arrTemp = $null
	$i++
}

I have used it on a couple of occasions now and have only discovered a problem with an error when one of the servers don’t have any agents at all (probably a new one), but the script still works so I haven’t really dived into it.
Now, as with all scripts you download on the ‘net it’s up to you to test it in a lab before shooting wildly among your in-production systems. I really can’t give any warranties that it won’t FSU royally at your place.

Categories: OpsMgr 2007, PowerShell | Tags: OpsMgr, PowerShell, Script

Load-balanced SCOM2012 SDK Services for Network Illiterates [#opsmgr, #nlb]

Prelude

Prerequisites

Create a New Cluster

Post-Configuration

Adding Hosts to the Cluster

Final Verification

Postlude

Parameter Replacement in AlertName

…and why you should not use it.

Bulk disable ACS Forwarders (with wildcards)

OpsMgr 2007 Connectivity Map

Introduction to TG WinAutoSvc v1

Background

Purpose

The TG WinAutoSvc Management Pack

SNMP GET Errors in OpsMgr EventLog

Let’s get to it

“Load Balancing” Powershell Script for Operations Manager

kaTWEET!

Categories

Recent Posts

Recent Comments