Enabling EVC (Enhanced vMotion Compatibility)

It has been a while since I have had to enable EVC, but I needed to the other day in the office.  I created a cluster with a HP DL380 G7 and an older HP DL380 G5.  When I tried to turn EVC on for the cluster, I ran into this error.  “The host cannot be admitted to the cluster’s current Enhanced vMotion Compatibility mode.  Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode.”  This message is telling you that the current machines that are powered on are using the technology from the newer processor and in order to turn on EVC for the cluster, the VMs need to be powered off.  So…I powered off all of my VMs on the DL380 G7 (newer host).EVC1

After all VMs are powered down, right click on the cluster and select Edit Settings.
evc2

Click the VMware EVC on the left pane and the click Change EVC Mode… button.EVC3

I have Intel processors so I selected Enable EVC for Intel Hosts.  Now I get a green check under the Compatibility pane.  Looking good!EVC4

Now depending on the processor generation, you have to change the EVC Mode.  For mine, I chose the Intel “Penryn” Generation” and I still had a green check box.  If your hosts don’t support the EVC mode, it will let you know in the Compatibility pane.  The processor support documentation can be found here.EVC5

We now see that Intel “Penryn” Generation is my EVC mode.   The only thing left to do is power on the VMs and start your migrations!EVC6

 

VMworld 2015 Schedule Builder is now live!

Time to sign up for the Ask the Expert vBloggers session first! #VMworld 2015 Schedule Builder is now live!

VMworld 2015 Schedule Builder is now live!

Search from over 400 unique sessions, sign up early for your favorites, and view customized recommendations based on your attendee profile! Are you ready for any?


VMware Advocacy

Who’s ready for the VMworld Party Already…

Looks like VMware is heading back to ATT stadium this year.

Join the VMworld community at AT&T park, home of the San Francisco Giants, for an evening of fun, feasting, and fantastic entertainment! VMworld will transform the park with exciting carnival rides and interactive midway games with the stunning backdrop of the San Francisco Bay. The party starts Wednesday, September 2 at 7:30PM .


VMware Advocacy

VMworld 2015 Content Catalog is now live!

What sessions will you attend? The #VMworld 2015 Content Catalog is now live!

VMworld 2015 Content Catalog is now live!

The Content Catalog allows prospective VMworld attendees access to the VMworld agenda, with the ability to peruse breakouts and note sessions of interest. You can search and filter to your heart’s content—by track, category, session format, industry, role, technical level, speaker name, location (US or Europe), and keyword search. You cannot schedule sessions in the catalog.


VMware Advocacy

Change Active Adapters and Load Balancing using ESXi Shell

Anyone ever had a network administrator make your hosts lose connectivity and the only option you have is to use the ESXi shell through an HP ILO port? I have. Here are the changes that were helpful to me at the time.

freakout

esxcli network vswitch standard list — This lists all of the vswitches on the host.
listvswitch

esxcli network vswitch standard uplink add -u vmnic1 -v vSwitch0 — Add an uplink to the vswitch. Just switch “add” to “remove” to get rid of a vmnic from the vswitch.
adduplinktovswitch

esxcli network vswitch standard policy failover get -v vSwitch0 — Get the load balancing policy for the vSwitch.
getvswitchpolicy

esxcli network vswitch standard policy failover set -a vmnic1 -v vSwitch0 — Be careful because when changing the active vmnics on a vswitch it will only add ones that you specify and if there are active ones that you specify it will move them to unused.
Incorrect
setactivevmniconvswitchfail
Correctsetactivevmniconvswitch

esxcli network vswitch standard portgroup policy failover get -p “Management Network” — This shows the policy information for the portgroup “Management Network”
getportgrouppolicy

esxcli network vswitch standard portgroup policy failover set -a vmnic1 -p “Management Network” — Change the active adapter on the Management Network.
setactivevmniconportgroup

esxcli network vswitch standard policy failover set -l iphash -v vSwitch0 — Set the load balancing on the vSwitch0 to iphash. Please be aware that this might change the portgroups to iphash if they are set to get their settings from the vswitch.
setloadbalanceonvswitch

esxcli network vswitch standard portgroup policy failover set -l iphash -p “Management Network” — Sets the load balancing policy on the Management Network portgroup to iphash. This will cause the load balancing policy on the vswitch to no longer apply to the Management Network portgroup.
setloadbalanceonportgroup

PowerCLI shutdown/startup script for remote sites.

In my environment I have 11 Remote Locations that each have one ESXi host.  I had a problem with ILO firmware updates the other day where I had to reboot these remote hosts.  I really didn’t want to use the VIC to click on each of the VMs to shut them down, so using a lot of help from mpreston’s site I came up with a script that will shut down all of my VMs and then when ready, start the VMs again.  My script shuts down “Front End” VMs first, then DB VMs, and then finally DCs.  I have a modified version that allows me to do the same for my Datacenters that I will share later.  You might have to edit the script if your database and or DCs are named differently.

add-pssnapin VMware.VimAutomation.Core
connect-viserver servername #Insert your vcenter server

#Variables
$datacenter = “datacenter”  #Add all datacenters here with a comma and quotations between each example “datacenter1”, “datacenter2”
$filename = “c:\vmshutdown\poweredonvms.csv”  #You must create this file first…it won’t create it for you.

#Get a list of the hosts being rebooted and set them to maintenance mode.  This keeps the VMs from powering on again.
Get-VMHost -location $datacenter | set-vmhost -State Maintenance

#Get a list of powered on VMs
get-vm -location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | Select Name | Export-CSV $filename

#Get a list of VMs that are not Database servers or DCs.  
Write-Host “”
Write-Host “Retrieving a list of powered on VMs…” -ForegroundColor Green
Write-Host “”
$poweredonguests = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -notlike “*DB*”} | where {$_.Name -notlike “*DC*”}

#Power off servers that are not Database and DCs
foreach ($guest in $poweredonguests)
{
Write-Host “Processing $guest…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $guest.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $guest, these will be powered off” -ForegroundColor Yellow
Stop-VM $guest -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $guest”
$vmshutdown = $guest | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for VMs that are not DB or DC to shutdown.”
Write-Host “”
Sleep 180

#Shut down DB Servers at remote sites
Write-Host “”
Write-Host “Shutting Down DB Servers”
Write-Host “”
#get our list of DB servers that are powered on…
$poweredondbs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -like “*DB*”} | where {$_.Name -notlike “*DC*”}
foreach ($dbs in $poweredondbs)
{
Write-Host “Processing $dbs…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $dbs.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $dbs, these will be powered off” -ForegroundColor Yellow
Stop-VM $dbs -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $dbs”
$vmshutdown = $dbs | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for Database VMs to shutdown.”
Write-Host “”
Sleep 180

#Shut down DCs
Write-Host “”
Write-Host “Shutting Down DCs”
Write-Host “”
#get our list of DB servers that are powered on…
$poweredondcs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -like “*DC*”}
foreach ($dcs in $poweredondcs)
{
Write-Host “Processing $dcs…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $dcs.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $dcs, these will be powered off” -ForegroundColor Yellow
Stop-VM $dcs -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $dcs”
$vmshutdown = $dcs | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for DCs to shutdown.”
Write-Host “”
Sleep 180

#Time to restart the hosts
get-vmhost -location $datacenter | restart-vmhost -runasync -confirm $false

Now that you have rebooted your hosts, it is time to start all the VMs again.  You are going to use the dame variable that we created above.

#Take hosts out of maintenance mode.  
get-vmhost -location $datacenter | set-vmhost -state connected

#Power On DCs
Write-Host “”
Write-Host “Powering On DCs”
Write-Host “”
#get our list of DCs servers that are powered off…
$poweredoffdcs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”} | where {$_.Name -like “*DC*”}
foreach ($dcs in $poweredoffdcs)
{
Start-VM $dcs -Confirm:$false
Sleep 5
}

#Power On DB Servers
Write-Host “”
Write-Host “Powering On DBs”
Write-Host “”
#get our list of DB servers that are powered off…
$poweredoffdbs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”} | where {$_.Name -like “*DB*”}
foreach ($dbs in $poweredoffdbs)
{
Start-VM $dbs -Confirm:$false
Sleep 5
}

#Power On the rest
Write-Host “”
Write-Host “Powering On Remaining VMs”
Write-Host “”
#get our list of VMs that are still powered off…
$poweredoffguest = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”}
foreach ($guest in $poweredoffguest)
{
Start-VM $guest -Confirm:$false
Sleep 5
}

 You remote locations should be back up!

False alerts after upgrading ILO firmware from 1.55 to 1.8

There are two separate alerts that I have experience from these upgrades.  Please note that after the update the ILO cards were rebooted, but our esxi hosts running vsphere 5.1 were not rebooted.

Error1:

Out of 31 hosts that had firmware updated, this error has appeared on 6 so far.  Some of them took a couple days before they presented with the alert.  The alarm triggered is: Host Memory Status.  Under the “Hardware Status” tab the alert shows for the System Board 8 Memory: Uncorrectable ECC.

memoryalarm

uncorrectable

 

Error2:

The second problem that I have encountered is the filling up of the IPMI SEL Log.  I am able to go in an clear the log which gets rid of the alert for a short time, but the log fills up again.  The alert shows as Host IPMI System Event Log Status.  Under the “Hardware Status” tab the “System Event Log” and “IPMI SEL” show as Unknown.  You can click on “Show Event Log” and then “Reset Event Log” and it will clear for a while…but the alert will return.  Notice the future date of 12/31/9998 which I am guessing might be when the world ends.

hostipmistatus

ipmisellog
thefuturelogs

How do I fix these problems?  After calling HP and VMware I was told that I needed to put each host into Maintenance Mode and then run a “Detailed Hardware Diagnostic”.  According to VMware this was the only way to clear the error (especially the memory one).  The solution that ended up working for me was to just reboot the host…:)  The VMware couldn’t believe that worked, but it did.  I know it isn’t a difficult fix, but maybe this might help others that get this alert.

Please let me know if you have encountered similar alerts from upgrading the firmware on your ILO ports.

Failing/Failed Disk And Detailed Information From HPACUCLI

Today I was alerted by vCenter with a yellow, “Host Storage Status” alarm.
diskalert

Clicking the ESXi host in the alert I find from the description that I can check the Hardware Status tab of the host for more details
diskalertdetail

I see that the disk is failing and that RAID is in “Interim Recovery”.  This doesn’t give me much information about the failing disk though.  Maybe in HP ILO I can find more information?  Nope, ILO doesn’t seem to have any information about the disks in the host.  According to the ILO System Information, everything looks just fine.
nodiskilo

I called to open a ticket with HP and while on the phone the support technician mentioned a diagnostic tool called HPACUCLI.  If you installed your host a HP customized ESXi image, then this should be there for you.

SSH into your host then type cd opt\hp\hpacucli\bin.  If you type ls you should see the script called “hpacucli”.  Type ./hpacucli and then press “Enter”.  You should see something about detecting controllers and you should be at a new prompt.
hpacucli1

Type controller all diag file=/tmp/adu1.zip and then press “Enter”.
hpacucli2

Now you will need a program like WinSCP that will get the newly created zip file off the host.  Connect to the host with WinSCP and then navigate to the /tmp folder.  Click on the adu1.zip file and the click “Download”.
tmpadu

Looking into the zip file we see four files; double click on “ADUReport.htm”.  The report should open into your browser and should have a lot of information.  Towards the top of my report I see that the drive has failed and if I scroll down I can actually get some in depth information about the drive.  Things like the model and Serial.
Drivefaileddrivefailed2

Can’t power on VM in Workstation 10 after Windows 8.1 October Rollup KB2995388

This issue has been fixed with the release of VMware Workstation 10.0.4.  VMware recommends that if you made changes to the config.ini that you remove the entry or change the value to “FALSE”.  The Workstation download can be found here.

When powering on a VM in Workstation 10 after rollup KB2995388 I see the following.

workstationerror

VMware is aware of the issue it would seem and there are three workarounds so far.  VMware official blog has this about it.

1.  Uninstall KB2995388 (This is an optional update).
2.  Edit the config.ini found in C:\ProgramData\VMware\VMware Workstation         and add the line:  vmmon.disableHostParameters = “TRUE”
3.  This is untested by me but it is reported that if you open Workstation in                   administrative mode (with admin rights), that it won’t have the problem.

Reboot your computer and your VMs should work again.

I did notice after making the change that the first time I start my VMs I get:
devicenotavailable notrespondingdevice

I chose “No” on both and the VMs now boot without issue.

Extending VMware VMDKs with System Partitions

hard-disk-drive
I have had to deal with extending System Partition VMDKs within our environment, but this can get tricky especially with 2003 Server and below.  Hope this helps…Note that Data drives can be extended without issue on 2012 and below.

2008 and above
1.  Open your VIC and edit the settings of the VM that you would like to modify.
Webeditsettings
2.  Change the system disk to the new size (Usually Hard disk 1).
Disk Change
3.   Log into your VM using the console or RDP.
4.  There are two ways to do the extend.  If it is 2008 R2 and above I usually do option 1.  2008 Standard and below I use option 2.

Option 1:
a.  Open Server Manager.
ServerManager
b.  Click Disk Management.
c.  Usually you will have to do a rescan to see the unallocated space.  Right click Disk Management and click Rescan Disks.
Rescandisks
d.  Right click the drive that you wish to expand (should show unallocated space) and click Extend Volume.
ExtendVolume
e.  Click OK on the Extend Volume Wizard and then again click OK on the Select Disks screen.  Now click Finish to complete the wizard.
f.  You should now see the drive is larger.

Option 2:
a.  Click Start –> Run and then type cmd.  Right click on the command prompt application and choose Run as Administrator.
runadmincmd
b.  Type diskpart.
diskpartsmall
c.  Type list volume.
diskpartlistvolume
d.  Select the system partition by typing select volume #.
diskpartselectvolume
e.  You should see an asterisk by the selected volume.  Type extend.
diskpartextend
f.  The drive should now be extended.  Type exit.

2003 Server and Below
1.   Have a 2008 R2 Helper machine that you will use to expand the drive.
2.  Power down the 2003 Server VM.
3.  Edit the settings and add space to the system partition.
Disk Change
4.  Power on the 2003 Server VM.
5.  Click Start –>All Programs –> Administrative Tools –> Computer Management.
ServerManager03
6.  Click Disk Management.  You should see the unallocated space, but if you do not then right click Disk Management and click Rescan Disks.   Just verify…don’t do anything with the space yet.
Rescandisks
7.  Shutdown the 2003 Server VM.
8.  Edit the 2003 Server VM within the VIC and note the location of the disk on the datastore.  Click remove on the drive that you would like extended by clicking on the X next to the hard disk.  DO NOT CHOOSE DELETE FILES FROM DATASTORE or REMOVE FROM VIRTUAL MACHINE AND DELETE FILES FROM DISK.  This is different wording depending on the Full VIC or the Web VIC.  Click OK.
RemoveDisk
diskremoved
9.  Now find your 2008 R2 helper VM within the VIC and then edit the settings.
10.  Where it says “New Device” choose Existing Hard Disk from the menu.  Click Add.
Addexistingdisk
11.  Using the information that you noted before, find the location of the 2003 Server disk that you removed before, select the disk, click OK.
choosevmdk
12.  Cick OK and this will mount the 2003 Server disk on your 2008 R2 helper VM.
13.  Log into the 2008 R2 helper VM with the console or RDP.
14. Click Start –> Run and then type cmd.  Right click on the command prompt application and choose Run as Administrator.
runadmincmd
15.  Type diskpart.
diskpartsmall
16.  Type list volume.
diskpartlistvolume
17.  Select the system partition by typing select volume #.
diskpartselectvolume
18.  You should see an asterisk by the selected volume.  Type extend.
diskpartextend
19.  The drive should now be extended.  Type exit.
20.  Go back into the VIC and edit the 2008 R2 helper VM.
21.  Click the X next to the Hard disk that you added earlier to this machine.  Again, DO NOT CHOOSE DELETE FILES FROM DATASTORE or REMOVE FROM VIRTUAL MACHINE AND DELETE FILES FROM DISK. 
RemoveDisk
diskremoved

22.  Edit the 2003 Server and then from the “New device” menu click Existing Hard Disk and then click Add.
Addexistingdisk
23.  Find the vmdk that you just extended then click OK.  Click again on the settings screen.
choosevmdk
24.  Power on the 2003 Server and you should see the additional space.  You might get a message indicating that a reboot is needed.  If requested, do the reboot.

Random Issues I Have Run Across
I did have an issue the other day with one our our 2008 Standard Servers.  When I went to extend the drive it gave me a weird error.  The parameter is incorrect.
DiskGrowParameterIncorrect

I give thanks to the Lord every day for the googles.  I found a great post by Jonathan Medd that gave the simple fix for this.  When you are in diskpart, select the volume # that you are extending and then use the command extend filesystem.