For quite some time now I’ve had this idea spinning around in my head to write a couple of blog-posts about some of the more useful techniques available when building management packs. Many of these techniques are already described on MSDN and Technet- or other blogs as well as on various forums, but often no more than small bits and pieces of them and I have yet to see some humanly readable information about how to tie them together into a useful management pack. I say “humanly readable” because the information you do find online so far may be clear and somewhat easy to understand for someone with a system development background and a pretty good idea of how object oriented development models tend to work. But the real life System Center Operations Manager engineer–you know the one who get those “do you think we could monitor our …-system too?” questions a couple of times a week, you know… you (most likely, being here)–tend to have a completely different background. Yet as their OpsMgr environment grows, so does the demand for custom monitoring and all of a sudden the former server engineer are now also a developer. A developer who has never before had the need to grasp such abstract concepts as classes, instances, inheritance and who probably never before have had any reason whatsoever to write any XML code.

Purpose

My idea for this series of posts is to shed some light into the world of the authoring console and modules and cookdown and so forth. I am by no means an accredited author, but I will do my best to stay human in this venture and in plain english try to explain why and how you do certain things when going from Management Pack templates, rules, monitors and the safe haven that is authoring in the Operations Console into making your scripts resuable, easy to extend and prime for cookdown using the Authoring Console and XML.

The TG WinAutoSvc Management Pack

To give the series some kind of context and at the same time not only be a matter of examples I will base them on a fully functional management pack that discovers and monitors all Windows services that are set to automatic startup. I know there is other similar management packs out there but I haven’t fancied any one of them yet, and since I had the idea of writing this series I decided that building a new one would be a good way to go. Some of the interesting features with this management pack is:

• You will get an instance of the service classes for each and every service.
• It uses different classes for Own Process services and Shared Process services (svchost for example).
• Every service have a health state (you can use them in distributed applications).
• The service state monitors are inherited from their base classes, no coding neccesary.
• There is only one discovery script for all kinds of windows services.
• Extending the discovery to include different kinds of windows services, like kernel processes, is a matter of filtering.
• It is Open Source and licensed under the Eclipse Public License v1.

Most of these features will be described thoroughly in later posts in the series and as development of it progresses I will document what I do, how I do it and why I do it in certain ways. Hopefully you will learn something new through this and get closer to becoming that MP Dev the organization asks for.
In the mean time, feel free to download, look at the source code (which it by no means perfect) and try it out.

The TG WinAutoSvc monitoring management pack is available for download here:
http://code.google.com/p/tg-winautosvc/downloads/detail?name=TG.WinAutoSvc.xml

The latest revision of the source code is located here:
http://code.google.com/p/tg-winautosvc/source/browse/trunk/TG.WinAutoSvc.xml

A small word of advice though. If you implement this in your environment, remember that you probably have alot more automatic services than you would expect. Because of this, discovery is disabled by default.

Best of luck, and enjoy!

To make it clear right away, this is not going to be a “Building an SNMP Management Pack Tutorial” since there’s plentiful good ones out there already, and to be extra helpful I’m gonna include a few links right away:

• SNMP Setup and Simple Custom SNMP Discovery – Pretty much the basics
• SNMP Management Pack Example: NetApp Management Pack – Part 4 actually, but has the links to the other parts
• Creating SNMP Probe Based Monitors – No custom discovery, but a good and simple guide to SNMP Probes

It’s the second, the NetApp one, I’ve used as a guide to building the UPS management pack since it goes through the process of building your own filtered discovery using SystemOID to identify your hardware-classes and then building the monitors on top of those.

Let’s get to it

When building the discovery of my hardware classes I ran into problems. The discovery simply did not work. At first I got some strange errors about “invalid queries”, something that turned out to be related to me reading two guides–seriously though, pick one guide that is closest to what you want to achieve and stick to it–and mixing up the XPathQuery variables. Silly me.
I got those errors to go away and I was able to get a few objects to my base-class, but none of the hardware classes who was populated through the return value of an SNMP OID got discovered.
The only error I got this time was the following:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          2010-09-02 11:19:12
Event ID:      11001
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CENSORED
Description:
Error sending an SNMP GET message to IP Address XX.XX.XX.XX, Community String:=CENSORED, Status 0x6c.

One or more workflows were affected by this.

Workflow name: CENSORED.MP.CLASS.DISCOVERY
Instance name: CENSORED_DEVICENAME
Instance ID: {5C7EFB30-D885-8843-0DD7-EA86B4FD2311}
Management group: CENSORED

<IsWriteAction>false</IsWriteAction>
<IP>$Config/IP$</IP>
<CommunityString>$Config/CommunityString$</CommunityString>
<Version>1</Version>
<SnmpVarBinds>
	<SnmpVarBind>
		<OID>1.3.6.1.4.1.534.1.1.1.0</OID>
		<Syntax>0</Syntax>
		<Value VariantType="8"></Value>
	</SnmpVarBind>
	<SnmpVarBind>
		<OID>1.3.6.1.4.1.534.1.1.2.0</OID>
		<Syntax>0</Syntax>
		<Value VariantType="8"></Value>
	</SnmpVarBind>
	<SnmpVarBind>
		<OID>1.3.6.1.4.1.534.1.1.3.0</OID>
		<Syntax>0</Syntax>
		<Value VariantType="8"></Value>
	</SnmpVarBind>
</SnmpVarBinds>

Both seem to be fully Cluster aware and pretty much holds the same monitoring as the the latest MSMQ 3 MP.

Message Queuing 4.0 Management Pack for Operations Manager 2007

Quick Details

Version: 6.0.6700.83

Date Published: 4/5/2010

Language: English

Download here: http://www.microsoft.com/downloads/details.aspx?FamilyID=cfc103b8-7185-4721-8098-110885fe9e9e&displaylang=en

Message Queuing 5.0 Management Pack for Operations Manager 2007

Quick Details

Version: 6.0.6700.88

Date Published: 4/5/2010

Language: English

Download here: http://www.microsoft.com/downloads/details.aspx?FamilyID=28349b78-8329-44aa-8a1f-81f4e3f84d0c&displaylang=en

System Center Pack for: Message Queuing 3.0
Version: 6.0.6615.0
Released on: 12/14/2009

Message Queuing (also known as MSMQ) is a server application that enables applications to communicate across heterogeneous networks and systems that may be temporarily offline or otherwise inaccessible. Instead of an application communicating with a service on another computer, it sends its information to Message Queuing, which sends the information to a Message Queuing service on the target computer where it is made available to the other application. Message Queuing provides guaranteed delivery, efficient routing, security, and priority based messaging.

Now, what’s really interesting is what you will find in the MP Guide under “Supported Configurations”.

The Message Queuing Management Pack for Operations Manager 2007 is designed to monitor Message Queuing version 3 only.

The Message Queuing Management Pack supports the following platforms:

· Windows Server 2003

· Windows XP

The Message Queuing Management Pack also supports monitoring clustered MSMQ components.

Text coloration is obviously added by me to highlight the interesting part.

Finally MSMQ monitoring seems to be cluster aware, which might mean that the home-made pack i did to have those (numerous) queues covered could be passed on to the scrap-heap. This is also confirmed under “Changes in This Update”.

The December 2009 update to this management pack includes the following change:

· Fixed a problem when working with an instance of MSMQ in a Cluster. The MP is now able to discover and monitor public and private queues in a cluster.

· Fixed a problem when discovering the local and cluster instance of MSMQ. The MP is now able to discover and monitor both instances.

The confusing double RunAs profiles seems to have been cleaned up too (you only have to worry about one now) as well as fixing some sloppy mistakes in the previous scripts (no Option Explicit? C’mon Microsoft! You write the best practices, try to stick to them.) and generally improving display and documentation.

Gonna import this to our staging environment today and let it roll during the holidays.

Cheers! Oh, and happy holidays!

Download and documentation:
http://www.microsoft.com/downloads/details.aspx?FamilyId=1D2B4398-8BC2-4A43-850C-852EBB0D983B&displaylang=en&displaylang=en

The update basically contains support for x64 that was missing in the previous release.

The Configuration Manager 2007 SP2 Management Pack adds support for monitoring Configuration Manager 2007 SP2 in a 64-bit environment with Operations Manager 2007 R2 or Operations Manager 2007 SP1 with hotfix (KB971541) installed. This enables the Configuration Manager 2007 SP2 Management Pack to work with either the 32-bit or the 64-bit Operations Manager 2007 agent. Except for the 64-bit support, the other features and guidance for Configuration Manager 2007 Management Packs remain intact.

(coloration added by me)

Read more and download here:
http://www.microsoft.com/downloads/details.aspx?FamilyID=a8443173-46c2-4581-b3b8-ce67160f627b

Besides SUSE 11 support, here’s the short overview.

The System Center Operations Manager 2007 R2 Cross Platform Update adds fixes for a defunct process issue on Unix/Linux Servers, as well as, adds support for SUSE Linux Enterprise Server 11 (both 32-bit and 64-bit versions) and Solaris Zone support.
Feature Summary:
The System Center Operations Manager 2007 R2 Cross Platform Update supports the monitoring of Unix/Linux Servers including:

• Monitoring of SUSE Linux Enterprise Server 11 servers (both 32-bit and 64-bit versions)
• Support of Solaris Zones
• Fix for defunct Process issue
• The Cross Platform Agent may not discover soft partitions on Solaris systems. Therefore, the disk provider may be unloaded, and the Cross Platform Agent may stop collecting information from the system disks.
• The Cross Platform Agent may not restart after the AIX server reboots.

The latest versions of all the Operations Manager 2007 R2 Unix/Linux agents are included in this update.

Perfect timing, I must say, since I really need this today.

Update:
This is no small MP-update, which probably is the reason that we do not find it in the MP Catalog, but a ~250MB OpsMgr R2 Software Update. You need to run this on all Operations Manager Servers (RMS/MS, GW?) since it actually updates many of the agent Cross Platform binaries. It does add a new MP för SUSE 11 that you have to import from disk if you need it.

So, the installation goes somewhat like this:

1. Install the Software Update (pick the right Architecture) on all OpsMgr R2 Servers
2. Import the SUSE 11 MP if necessary
3. Re-discover your Unix/Linux machines.

Files updated in this update for R2:

• .Microsoft.Enterprisemanagement.UI.Administration.dll (Version 6.1.7043.1)
• .AgentManagementUnixAgentsscx-1.0.4-248.aix.5.ppc.lpp.gz
• .AgentManagementUnixAgentsscx-1.0.4-248.aix.6.ppc.lpp.gz
• .AgentManagementUnixAgentsscx-1.0.4-248.hpux.11iv2.ia64.depot.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.hpux.11iv2.parisc.depot.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.hpux.11iv3.ia64.depot.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.hpux.11iv3.parisc.depot.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.rhel.4.x64.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.rhel.4.x86.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.rhel.5.x64.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.rhel.5.x86.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.sles.10.x64.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.sles.10.x86.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.sles.9.x86.rpm
• .AgentManagementUnixAgentsscx-1.0.4-248.solaris.10.sparc.pkg.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.solaris.10.x86.pkg.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.solaris.8.sparc.pkg.Z
• .AgentManagementUnixAgentsscx-1.0.4-248.solaris.9.sparc.pkg.Z

Files added:

• Microsoft.Linux.SLES.11.MP

All in all, the update contains the following fixes:

• KB969342
• KB973583
• Q954049
• Q956240

In case you wonder why on earth I post when there’s allready a “solution” out there; Pagerank, baby!
Not for me, but for the forum post making it show up earlier on google.

Most significant updates, according to me, would seem to be:

Fixed an issue that was previously preventing all rules related to agentless exception monitoring from generating alerts.

Added the rule “Collects Opsmgr SDK ServiceClient Connections” to collect the number of connected clients for a given management group. This data is shown in the view “Console and SDK Connection Count” under the folder “Operations ManagerManagement Server Performance”.

Updated a number of monitors and rules to ensure that data is reported to the correct management group for multihomed agents.

Fixed the configuration of the rule “IIS Discovery Probe Module Execution Failure” to so that the parameter replacement will now work correctly for alert suppression and generating the details of the alert’s description.

The rest is mostly polishing, fine-tuning and complementary updates. Nothing really ground-breaking here, but still a welcome update.

Download at: http://www.microsoft.com/downloads/details.aspx?FamilyID=61365290-3c38-4004-b717-e90bb0f6c148

http://blogs.technet.com/momteam/archive/2009/09/25/sql-management-pack-survey-live-on-connect.aspx

Quick Details

Feature Summary

• Monitor the NLB Node status.
• Based on the status of individual cluster nodes, determine the overall state of the cluster.
• Where an integration management pack exists, determine the health state of a cluster node by looking at the health state of the load balanced application, such as IIS.
• Alert on errors and warnings that are reported by the NLB driver, such as an incorrectly configured NLB cluster.
• Take the node out of the NLB cluster if the underlying load-balanced application becomes unhealthy, and add the node back to the cluster when the application becomes healthy again.

Requires OpsMgr 2007 SP1 or later, the Base Operating System MP for 2008, the QFEs for Windows Server 2008 and that you are not running the converted 2003 NLB MP. If you are running the old converted NLB MP, upgrade first. As an additional recommendation, Microsoft recommends in the MP Guide that you install the QFE for wmiprvse.exe problems on Windows Server 2008.

No support for Mixed-mode (2008 and 2003) clusters though.

Error doing IIS Discovery

Error: 0x80070002
Details: The system cannot find the file specified.

One or more workflows were affected by this.

Workflow name: Microsoft.Windows.InternetInformationServices.2003.DiscoverBase
Instance name: Microsoft.Windows.InternetInformationServices.2003.ServerRole
Instance ID: {A81E4808-4D05-9BFE-4043-DC668527F2D0}
Management group: MASKED

Or…

Error doing IIS Discovery

Error: 0x80070006
Details: The handle is invalid.

One or more workflows were affected by this.

Workflow name: Microsoft.Windows.InternetInformationServices.2000.DiscoverWebSites26to50
Instance name: IIS Web Server
Instance ID: {D36DA76A-027F-8F3E-4160-115279A1E23A}
Management group: MASKED

I have been trying to figure out what file is missing and/or if the “invalid handle” is related. Possibly a file-handle? Could be but not neccesary since these two errors occur on different servers with increasing repeat-count (atleast once-a-day). The IIS MP does call the IIS*.VBS Scripts in %windir%System32 but as far as I can tell, on the systems I have tried it on, the scritps return valid data. This does by no means mean that there is no error and evidently I am missing something. But what? Does anyone have a clue to this?

References and other victims:

• “Error Doing IIS Discovery” on Google
• “SCOM: IIS Discovery Probe Module Failed Execution” on Codejnki’s Blog
• “Error doing IIS Discovery Error: 0×80070006″ on TechNet Forums
• “IIS Discovery Probe Module Execution Failure” on Digital Support Forum
• “IIS Discovery Probe Module Execution Failure” on EggHeadCafe
• “IISDiscoveryProbe Error – any ideas?” on Microsoft Discusson Groups

And no, neither of these provides even a hint to a working solution.

Quick Details

Release History

• 6/3/2008 – Initial Release, version 6.0.6278.23. Refer to the MP guide for further details.
• 4/24/2009 – Undated release, version 6.0.6587.0. Refer to the MP guide for further details.

The MP Guide does not really say much about what’s updated and I don’t know how much more than the reporting they have fixed. I can just state the fact that support for clustered MSMQ 3 instances is still missing.

Symptoms

This “research” comes from getting an obscene amounts of Script or Executable Failed to run in the Operations Console. Each time it was the NetworkAdapterCheck.vbs script that could not create PropertyBagData. The error message copied from one of the alerts looks like this:

The process started at 14:29:26 failed to create System.PropertyBagData, no errors detected in the output. The process exited with 0

Command executed: "C:WINNTsystem32cscript.exe" /nologo "NetworkAdapterCheck.vbs" MASKEDCOMPUTERNAME 0 false true false
Working Directory: C:Program FilesSystem Center Operations Manager 2007Health Service StateMonitoring Host Temporary Files 2882781

One or more workflows were affected by this.

Workflow name: Microsoft.Windows.Server.2000.NetworkAdapter.NetworkAdapterConnectionHealth
Instance name: 0
Instance ID: {F4C478D3-38E5-8C29-3957-E3B7F486216E}
Management group: MASKED

This error repeats almost as often as the script is scheduled to run and appears on almost every Windows 2000 server.

Cause?

I am not really sure, but after a quite a bit of troubleshooting I am pretty sure it all boils down to a malformed WMI-query. What I basically did was to extract the script from the MP and dry-run it to see if I could find anything obvious, which I didn’t.
Since I didn’t have a good debugging too available, like in PrimalScript, I added the VBS equivalent of old-school printf debugging. I basically added

wscript.echo "Line XX:" & Err.Number

after each object/function call in the Main Sub.
The problem seems to be at line 118:

Set WMISet = WMIGetInstance("winmgmts:" + TargetComputer + "rootcimv2", query)

At line 119 I had a debug line that never executes and it would seem like the script is unable to process the query and exits without any error codes.

The query variable contains the following:

Win32_NetworkAdapter Where DeviceID='0' And ( NetConnectionStatus = '0' )

When I was debugging this I started the script like this:

cscript.exe /nologo "NetworkAdapterCheck.vbs" MASKED 0 false true false

I did find that if I change the parameters to this:

cscript.exe /nologo "NetworkAdapterCheck.vbs" MASKED 0 false false false

Then the script runs smoothly and produces an XML-output like this:

<Collection>
<DataItem type="System.PropertyBagData" time="2009-04-22T15:03:44.6495202+02:00" sourceHealthServiceId="C9F5A9F7-00A3-8A86-57B5-34FB634471A8">
<Property Name="State" VariantType="8">GOOD</Property>
</DataItem>
</Collection>

Comments

There’s quite the lot of similar errors posted on various forums on the internet but most of them ends up pointing towards the Antivirus Software as the perp and we have tried various exceptions on the server like excluding HealthExplorer.exe, MonitoringHost.exe and also the entire Health Service State folder with no improvement. But since I was able to run the scripts from other locations (not excluded) on the same server I find it hard to believe that the AV would be blocking it.