Introduction

I’ve been working on producing a daily report to audit who’s been modifying compute resources and while looking for some resources, found a great blog by Luc Dekens (@LucD22). However, one of the problems was that neither events or tasks keep original values, only retain new values.

In this blog post, instead of going through the script line by line, I would rather go through what I’ve done to overcome this constraint.

Mechanism

Initially, I thought about deploying a database (MySQL or MSSQL or whatever…) to store the compute resource data, i.e. CPU, memory and hard disk(s). But it wasn’t a good idea as it introduces another management layer. After a few minutes, one thing popped up in my head that how about using a .csv file to store the data? Did a quick brainstorm and came up with the following:

1. For the first run, get the list of VM, CPU, memory, hard disks and capacity and export the list as a .csv file (I will call vm_report.csv from now on).
2. For the second run and so on, for each event, generate an array to contain what has been modified by comparing the new value to old value from the .csv file. For instance, CPU 1->2.
3. Once the for loop against events is finished, send an email to administrators for review.
4. Export the new list to overwrite vm_report.csv.

Finished writing a script and started testing. One thing I found is that the script ran for the second time with no issues but for the third time, it failed. Well, it was obvious as the vm_report.csv was already up-to-date and I was trying to get the comparison where there will be none!

I was looking at the event object closely and figured out that each event has a unique element called key. Which means, if I can somehow filter the events out using the keys, there will be no overlap.

Eventually, I generated another .csv file called vm_events_keys.csv with the list of keys. Before the script queries the event information, it first looks at whether the vm_events_keys.csv contains the event key. If there is, it means it’s already been queried before so it could be ignored. Otherwise it’s a new event.

In summary:

1. Get the list of information and save it in vm_report.csv.
2. Get the list of keys of events and save it in vm_events_keys.csv.
3. For each event, ensure vm_events_keys.csv does not contain the key of the event. If the event is a new one, generate report
4. If there are new events, update vm_report.csv and vm_events_keys.csv. Otherwise do nothing.
5. Send an email with report to administrators for review.

Before I move on to walk through, some people might ask about the period of events the script queries for, i.e. begin time and end time. Let me make it simple. Literally, the script looks for the time between the users execute the script. For example, if you ran the script at 7pm and then run it again at 10pm, the period will be 3 hours. Then after 24 hours you run the script, it will query for 24 hours of events.

Time for walk through!

Walk Through

In this walk through, I will be running the script for the three times.

The screenshot below shows you it was the first time running the script. It outputs a several warnings saying there are no .csv files as well as no new events:

The following screenshot is after the vm_report.csv and vm_events_keys.csv were generated. For the second run, it shows you a nice report showing what’s been changed as well as old values and new values.

The last run produces a warning saying “No new events found” and the reason is simple, I already ran it before and the vm_keys_events.csv is updated!

Looking at the inbox, I also got the result in a table format:

Script

The following is the script written. Did minimal comments, feel free to leave a reply if you need any clarifications.

## report function that outputs the result
function report {
    param ($event, $change, $operation, $value)
    return $event | Select @{N="Event Time";E={[datetime]($_.CreatedTime).addHours(12)}}, @{N="Cluster";E={$_.ComputeResource.Name}}, @{N="VM";E={$_.VM.Name}}, @{N="Log";E={$_.FullFormattedMessage}}, @{N="Username";E={$_.UserName}}, @{N="Change";E={$change}}, @{N="Operation";E={$operation}}, @{N="Value";E={$value}}
}

## update_report function that updates the vm_report .csv file
function update_report {
    param($vm_report, $vm, $change, $operation, $value)
    
    switch (([string]$operation).ToLower()) {        
        add {
            $harddisk = $vm | %{ [String]::Join(":", ($_.HardDisks.ExtensionData.DeviceInfo.Label)) }
            $harddisk_capacity = $vm | %{ [String]::Join(":", ( ($_.HardDisks.ExtensionData.DeviceInfo.Summary -replace " KB"))) }
            
            ($vm_report | where {$_.Name -eq $vm.Name})."Hard Disk" = $harddisk
            ($vm_report | where {$_.Name -eq $vm.Name})."Capacity" = $harddisk_capacity
            
            break
        }
        
        remove {     
            $harddisk = $vm | %{ [String]::Join(":", ($_.HardDisks.ExtensionData.DeviceInfo.Label)) }
            $harddisk_capacity = $vm | %{ [String]::Join(":", ( ($_.HardDisks.ExtensionData.DeviceInfo.Summary -replace " KB"))) }
            
            ($vm_report | where {$_.Name -eq $vm.Name})."Hard Disk" = $harddisk
            ($vm_report | where {$_.Name -eq $vm.Name})."Capacity" = $harddisk_capacity
            
            break
        }
        
        edit { 
            ($vm_report | where {$_.Name -eq $vm.Name})."Capacity" = $value
            break
        }
        
        modify {
            if ($change -eq 'Memory') {
                ($vm_report | where {$_.Name -eq $vm.Name}).MemoryGB = $value
            } elseif ($change -eq 'CPU') {
                ($vm_report | where {$_.Name -eq $vm.Name}).CPU = $value
            }
            break
        }
        
        default { 
            break
        }
    }
    return $vm_report
}

## send_email function that sends the final result to the end user
function send_email {
    param ($event_report, $today)

    $header = @"
<style>
TABLE {border-width: 1px;border-style: solid;border-color: black;border-collapse: collapse;}
TH {border-width: 1px;padding: 3px;border-style: solid;border-color: black;background-color: #6495ED;}
TD {border-width: 1px;padding: 3px;border-style: solid;border-color: black;}
</style>
"@

    $body = "<h2>Today's Date: " + $today + "</h2>"
    
    if (!$event_report) {
        $body += "There are no new events"
    } else {
        $body += $event_report | Sort "Event Time" | ConvertTo-HTML -Head $header | Out-String
    }
    Send-MailMessage -From "Email Sender Address" -To "Email Receiver Address" -Subject "vSphere Audit Report" -BodyasHtml -Body $body -SmtpServer "SMTP Server"
}

## List of administrator users you would like to exclude. For example, exclude yourself as you know you have the right to modify
$admins = "com.vmware.vadm|administrator|YourAccount"

## Check for vm_report.csv & vm_events_keys.csv files and if they don't exist, generate one
if (Test-Path "vm_report.csv") {
    $vm_report = Import-Csv "vm_report.csv"
} else {
    $vm_report = Get-VM | Sort Name | Select Name, @{N="CPU";E={$_.NumCpu}}, 
    MemoryGB, 
    @{N="Hard Disk";E={[String]::Join(":", ($_.HardDisks.ExtensionData.DeviceInfo.Label))}}, 
    @{N="Capacity";E={[String]::Join(":", ( ($_.HardDisks.ExtensionData.DeviceInfo.Summary -replace " KB")))}}
    
    $vm_report | Export-CSV -UseCulture -NoTypeInformation "vm_report.csv"
    Write-Warning "No vm_report.csv found therefore, exported a new list. Run it again after at least one event occurs"
}

if (Test-Path "vm_events_keys.csv") {
    $events_keys = Import-Csv "vm_events_keys.csv"
} else {
    $events_keys = @{"Key" = ""}
    Write-Warning "No vm_events_keys.csv found, starting with an empty array"
}

## Create vCenter API object, EventManager
$event_manager = Get-View EventManager
$event_filter_spec = New-Object VMware.Vim.EventFilterSpec
$event_filter_spec.Type = "VmReconfiguredEvent"

$event_filter_spec.Type = "VmReconfiguredEvent"
$event_filter_spec.Time = New-Object VMware.Vim.EventFilterSpecByTime

## Define the Event Filter. 
## Start date is based on the last modified date of the vm_report.csv file.
## End date is now
$last_modified = (Get-Item vm_report.csv) | %{$_.LastWriteTime}
$today = Get-Date
$hour_difference = -($today - $last_modified).TotalDays

$event_filter_spec.Time.BeginTime = (Get-Date).AddDays($hour_difference)
$event_filter_spec.Time.EndTime = (Get-Date)

## Query events with the Event Filter defined above
$events = $event_manager.QueryEvents($event_filter_spec) | where {$_.Username -notmatch $admins} | Sort CreatedTime

## For each event, see if the event key is found in $vm_events_keys list
## If found, then skip it otherwise save the result to $event_report array
$event_report = foreach ($event in $events) {
  $event | Foreach-Object {        
        if (!(($events_keys | %{$_.Key}).Contains([string]($_.Key)))) {
            $vm = Get-VM -Name $_.VM.Name    
            if ($vm -and ($vm_report | where {$_.Name -eq $vm.Name})) {
                
                ## Memory modified
                if ($_.ConfigSpec.MemoryMB) { 
                    $change = "Memory"
                    $operation = "Modify"
                    $new_value = $_.ConfigSpec.MemoryMB    
                    $old_value = [int]($vm_report | where {$_.Name -eq $vm.Name} | %{$_.MemoryGB}) * 1024
                    
                    $value = [string]$old_value + 'MB->' + [string]$new_value + 'MB'
                    if ($new_value -ne $old_value) {
                        ## Update $vm_report array
                        $vm_report = update_report $vm_report $vm $change $operation ($new_value / 1024)
                        ## Save the change to $event_report array
                        report $event $change $operation $value
                    }
                }
                
                ## CPU modified
                if ($_.ConfigSpec.NumCPUs) {
                    $change = "CPU"
                    $operation = "Modify"
                    $new_value = $_.ConfigSpec.NumCPUs
                    $old_value = [int]($vm_report | where {$_.Name -eq $vm.Name} | %{$_.CPU})            
                    $value = [string]$old_value + '->' + [string]$new_value
                    if ($new_value -ne $old_value) {
                        $vm_report = update_report $vm_report $vm $change $operation $new_value
                        report $event $change $operation $value
                    }
                } 
                
                ## Hard Disk modified
                if ($_.ConfigSpec.DeviceChange) {
                    ## Use Foreach-Object as there might be more than 1 hard disk changes
                    $_.ConfigSpec.DeviceChange | where { [string]$_.Device -eq "VMware.Vim.VirtualDisk" } | Foreach-Object {
                        if ($_.Device.DeviceInfo.Label) {
                            $change = $_.Device.DeviceInfo.Label
                            $operation = $_.Operation
                            $new_value = $_.Device.CapacityInKB / 1024 / 1024
                            $harddisk = ($vm_report | where {$_.Name -eq $vm.Name} | %{$_."Hard Disk"}) -split ":"
                            $harddisk_capacity = ($vm_report | where {$_.Name -eq $vm.Name} | %{$_."Capacity"}) -split ":"            
                            
                            for ($i = 0; $i -lt $harddisk.length; $i++) {
                                if ($harddisk[$i] -eq $change) {
                                    $old_value = [int]$harddisk_capacity[$i] / 1024 / 1024
                                    $harddisk_capacity[$i] = ("{0:N0}" -f ($new_value * 1024 * 1024))
                                }
                            }                        
                            ## If the operation is not Remove, i.e. Edit
                            if ($operation -ne 'remove') {
                                if ($new_value -ne $old_value) {
                                    $value = [string]$old_value + 'GB->' + [string]$new_value + 'GB'
                                    $vm_report = update_report $vm_report $vm $change $operation ([string]::Join(":", ($harddisk_capacity)))
                                }
                            ## If the operation is Remove
                            } else {
                                $value = [string]$old_value + 'GB'                            
                                $vm_report = update_report $vm_report $vm $change $operation ""
                            }
                            ## Save the output to $event_report
                            report $event $change $operation $value        
                            
                        ## If there is no device label, it means there is a new hard disk
                        } else {
                            $change = "New Hard disk"
                            $operation = $_.Operation
                            $value = $_.Device.CapacityInKB / 1024 / 1024
                            $vm_report = update_report $vm_report $vm $change $operation ""
                            report $event $change $operation ([string]($value) + "GB")                        
                        }
                    }
                }
            }
            
            ## Reset parameters after 1 loop
            $change = '';
            $operation = '';
            $old_value = '';
            $new_value = '';
            $value = '';
        }
    }
}

## If there are no events, send an email with an empty result
if (!$event_report) {
    Write-Warning "No new events found"
    send_email "" $today
## If there are events found, update vm_events_keys.csv to prevent duplicate check
## Update vm_report.csv to ensure the data is up-to-date
## Send an email to the administrators with the changes
} else {
    $event_report
    $events | Select Key | Export-Csv -UseCulture -NoTypeInformation "vm_events_keys.csv"
    $vm_report | Export-Csv -UseCulture -NoTypeInformation -Force -Path "vm_report.csv"
    send_email $event_report $today
}

Hope this helps!