How to enable ESXi host alarms on the vCenter with PowerCLI (& how to automate it)

Posted on by Sahan Fernando

In a previous write-up I discussed how to check the Alarm status of ESX hosts in a cluster. Let’s take it a step further and check how we can enable the alarms on an ESX host in order to automate it – so that alarms will be enabled on ESX hosts for eternity, as a healthy environment should! No longer shall we be in doubt whether we forgot to enable alarms in a ESX host after working on it! This is actually very simple! Lets get to it!

Here I will be talking about enabling alarms on all hosts in the vCenter (not on a particular cluster). You can put this script on a task scheduler to keep it running on a schedule so that Alarms will be enabled periodically throughout the day.

Step 1: Getting the list of ESX hosts whose alarms are disabled

$alarm_disabled_hosts = Get-VMHost | Where-Object {$_.extensiondata.AlarmActionsEnabled -eq $false}

This was discussed in the previous post, so I won’t discuss this at length. Its a pretty self evident line anyway 😉 With Get-VMHost called raw and bare naked like that we loop through all hosts piped to the where-object clause which checks for the AlarmActionsEnabled extension data.

Step 2: Enable them alarms!

The Service Instance in charge of handling alarms in the vCenter is the Alarm manager. So lets get a view of that manager entity to call on its management capabilities. 

$alarmMgr = Get-View AlarmManager

Now that we have called on the Alarm Manager to the $alarmMgr variable, we can also call on EnableAlarmActions to enable alarms from our list of hosts whose alarms are disabled

$alarmMgr.EnableAlarmActions(<host_name>.Extensiondata.MoRef,$true)

Above line enables the alarms on the host that we specify

Let’s see how we can put all this together in one full script:

Connect-VIServer <vCenter>

$alarm_disabled_hosts = Get-VMHost | Where-Object {$_.extensiondata.AlarmActionsEnabled -eq $false}

$alarmMgr = Get-View AlarmManager


ForEach ($hosts in $alarm_disabled_hosts){

    $alarmMgr.EnableAlarmActions($hosts.Extensiondata.MoRef,$true)
    
}

This would enable alarms on all ESX hosts whose alarms are currently disabled.

Extra Credit : Skip hosts in maintenance mode

If the host is in maintenance mode, chances are that you are working on some error on the host and you will have to keep those alarms disabled. So for extra credit let’s see how we can modify the script to skip the hosts in maintenance mode:

Connect-VIServer <vCenter>

$alarmMgr = Get-View AlarmManager


ForEach ($hosts in $alarm_disabled_hosts){

    if ($hosts.ConnectionState -eq "Maintenance"){
        Write-Host "$hosts in maintenance mode, alarms will remain disabled"
    }

    else{
        Write-Host "Enabling alarms in $hosts"
        $alarmMgr.EnableAlarmActions($hosts.Extensiondata.MoRef,$true)
    }

}

And thats it! Keep this running on a schedule and you can be sure your environment has constant monitoring enabled.

If you found this useful, it helps me out if you share, comment or just generally spread the word (no pressure 😉 ). Social media links are below!

And as always, Happy scripting!

Changing the vCenter Alarm reporting frequency | PowerCLI

Posted on by Sahan Fernando

Recently we noticed that in vCenters while we receive SNMP or Email (SMTP) red alerts, we did not receive the green alerts that changed immediately.

For example if an ESX host had a memory utilization of 95%, it would trigger a red state alarm and would send out an email or SNMP trap notifying the admins “HELP SOMETHING IS WRONG“, but should that alarm flap immediately and return to green (below that 95% or 90% whichever is set) – the vCenter doesn’t send out an email or SNMP saying “okay false alarm calm down“. This would waste a vigilant system administrator’s time checking an alarm that was a fluke.

According to this VMware KB article the alarm actions like Email alerts/SNMP traps are not triggered for state changes within 5 minutes from initial alarm trigger. So it’s official. And there is no GUI way to change this, it’s all PowerCLI to the rescue.

We didn’t notice this until we built an application that shows real time vCenter alarms which was sensitive to flapping state changes. So we had to take action to set the alarming frequency from 5 minutes to 0 to get immediate alarms. Fortunately you can do this for individual alarms, which means its not a global setting that changes the alarm frequency of all alarms in the vCenter.

Enough chit chat – down to scripting! I will do a breakdown of the full script to explain what I’m doing and have the full script down below, if you’re lazy you can just copy the script and run it

#Credentials 
$user = "vcenter_username"
$pwd = "vcenter_password"

#the vcenter name goes here
$server = "vcenter_1"
	
Connect-VIServer $server -User $user -Password $pwd

#variable with the name of the alarm that you want to change the alarming frequency of
$alarm_check = "vCenter host alarm"
write-host "checking alarm $alarm_check"
	
$alarm_id = (Get-AlarmDefinition | where name -eq $alarm_check ).id
$alarm_view = Get-View -id $alarm_id

echo $alarm_view.info.setting

#create a spec variable that holds a copy of data to update
$spec_to_update = $alarm_view.info
$spec_to_update.Setting.ReportingFrequency = 0

Write-Host "Reconfiguring $alarm_check"
$alarm_view.ReconfigureAlarm($spec_to_update)

Write-Host "New Reporting frequency of $alarm_check"

$new_alarm_id = (Get-AlarmDefinition | where name -eq $alarm_check ).id
$new_alarm_view = Get-View -id $new_alarm_id

echo $new_alarm_view.info.setting

For those of you who want to know what just happened with this script here is a breakdown of the script:
First up we need to connect to the vCenter via Connect-VIServer

#Credentials 
$user = "vcenter_username"
$pwd = "vcenter_password"

#the vcenter name goes here
$server = "vcenter_1"
	
Connect-VIServer $server -User $user -Password $pwd

Next up we create a variable that has the name of the alarm which you want to change the reporting frequency of, we’re calling that $alarm_check. (I’m taking the vcenter host alarm example I took above for this too and assuming the alarm name is “vCenter host alarm”)

$alarm_check = "vCenter host alarm"
write-host "checking alarm $alarm_check"

Then we take the ID of the alarm (vCenter has a separate ID for every alarm to identify every alarm uniquely), then we get-view that alarm ID which shows all the settings of that alarm. We then echo the settings to see whether the reporting frequency is actually 5 minutes

$alarm_id = (Get-AlarmDefinition | where name -eq $alarm_check ).id
$alarm_view = Get-View -id $alarm_id

echo $alarm_view.info.setting

Here’s where things might get a liiiiiittle bit confusing. We create a copy of that alarm-view data, and then create a proxy copy of data that need to be there with the new alarm reporting frequency of 0. Basically we have created a variable with the setting that we want to be there..

#create a spec variable that holds a copy of data to update
$spec_to_update = $alarm_view.info
$spec_to_update.Setting.ReportingFrequency = 0

Then we take the original alarm-view and reconfigure the original alarm with the proxy copy of data that we created with the data that we need to be there

Write-Host "Reconfiguring $alarm_check"
$alarm_view.ReconfigureAlarm($spec_to_update)

Like the good Sys Admins that we are – then we double check whether the change did take place

$new_alarm_id = (Get-AlarmDefinition | where name -eq $alarm_check ).id
$new_alarm_view = Get-View -id $new_alarm_id

echo $new_alarm_view.info.setting
The output should look a little something like this

And that’s it! The vCenter should send alarm state changes immediately now!

The main issue is there is no GUI way of doing this, you have to rely on PowerCLI to do that for you. But if you do find a better, lazier way to do this please do let me know – I mean that’s why we script right? because we laaaaazy 😉