SolarWinds Orion NPM: Configuring Cisco Hardware Health Pollers

While troubleshooting an issue with a 4506 switch recently, I discovered an interesting discrepancy: Orion was reporting a critical hardware issue on the switch, but the switch itself didn’t seem to have any idea that anything was wrong.

The exact error I was seeing was that a supply voltage sensor for a fiber switchport interface was in a critical state.

A bit of Google-fu revealed multiple publications from the SolarWinds Customer Success Center which explained, in short, that Cisco devices may trigger false or invalid hardware health state alerts in the Orion Web Console due to a misconfiguration of the Hardware Health Poller settings.

By default, it seems that Orion uses the CISCO-ENTITY-SENSOR-MIB.  The publications from SolarWinds state that this issue exists on the Cisco 3850 and 4500-series “and many more,” which is unhelpfully vague.  There is a link to another article hidden behind a login form that seems to insinuate that there is also a known problem with Cisco 3650 switches incorrectly reporting temperatures because of the same issue.

The fix is to configure Orion to poll hardware health using the CISCO-ENVMON-MIB instead.  I attempted to research why this fixes the issue without much success.  The closest I got to finding a reason why the ENVMON MIB works when the ENTITY-SENSOR MIB does not was this sentence from an undated and deprecated Cisco document for the UBR series router:

In Cisco IOS Release 12.2BC and later releases, use this MIB instead of CISCO-ENTITY-SENSOR-MIB for temperature monitoring.

This seems to insinuate that the CISCO-ENTITY-SENSOR-MIB was deprecated following the release of Cisco IOS 12.2BC, but that’s largely an assumption on my part.  I have attempted further research without any success.

The Resolution

To reconfigure the Hardware Health Polling setting for a specific node, follow the steps below:

  1. Log into the Orion Web Console.
  2. Click Settings > Manage Nodes.
  3. Locate and select the desired node.
  4. Click Edit Properties.
  5. Scroll down to the Hardware Health Polling section near the bottom of the page.
  6. Click the Preferred Cisco MIB drop-down menu and select CISCO-ENVMON-MIB.
  7. Click Submit.

At this point you can either wait for Orion to poll the node itself automatically or initiate a manual poll.  This should resolve the issue on the node.  However, if you would prefer to change the default behavior of Orion to use the ENVMON MIB instead, follow these steps:

  1. From the Orion Web Console, click on the Settings > All Settings menu.
  2. Find the Thresholds & Polling section, then click Polling Settings.
  3. Scroll down to the Hardware Health Polling section near the bottom of the page.
  4. Click the Preferred Cisco MIB drop-down menu and select CISCO-ENVMON-MIB.
  5. Click Submit.

Alternatively there are two further workarounds that can be used if reconfiguring the MIB does not resolve the issue:

  • Disable the sensor(s)
  • Override the sensor(s)

To override the sensor thresholds:

  1. From the Orion Web Console, click on the Settings > All Settings menu.
  2. Find the Node & Group Management section, then click Manage Hardware Sensors.
  3. Locate and select the desired sensor(s), then click Edit Thresholds.

From this point, you can change the sensor status state and/or specify a new threshold.

Additional Thoughts

With no further information on whether this issue is caused by Cisco IOS version, hardware model, or both, I have elected to keep the global setting configured for its default value of CISCO-ENTITY-SENSOR-MIB and change the setting manually on a device-by-device basis until I have a better understanding of what exactly is going on.

It would be helpful if someone – anyone – would publish some sort of guidance about the details of this issue.

References

Inaccurate Cisco hardware health is in warning or critical state | SolarWinds Customer Success Center
https://support.solarwinds.com/SuccessCenter/s/article/Inaccurate-Cisco-hardware-health-is-in-warning-or-critical-state?language=en_US

Change the preferred Cisco MIB for hardware health in NPM | SolarWinds Customer Success Center
https://support.solarwinds.com/SuccessCenter/s/article/Change-the-preferred-Cisco-MIB-for-hardware-health-in-NPM?language=en_US

Change the MIB used for polling hardware health statistics | SolarWinds Customer Success Center
https://documentation.solarwinds.com/en/Success_Center/orionplatform/Content/Core-Changing-MIB-Used-for-Polling-Hardware-Health-Statistics-sw1607.htm

Chapter 3, MIB Specifications | Cisco
https://www.cisco.com/en/US/docs/ios/zz_trash/CableCmds/ubrmib3M.html#wp1029199