VMworld 2015 Schedule Builder is now live!

Time to sign up for the Ask the Expert vBloggers session first! #VMworld 2015 Schedule Builder is now live!

VMworld 2015 Schedule Builder is now live!

Search from over 400 unique sessions, sign up early for your favorites, and view customized recommendations based on your attendee profile! Are you ready for any?


VMware Advocacy

Who’s ready for the VMworld Party Already…

Looks like VMware is heading back to ATT stadium this year.

Join the VMworld community at AT&T park, home of the San Francisco Giants, for an evening of fun, feasting, and fantastic entertainment! VMworld will transform the park with exciting carnival rides and interactive midway games with the stunning backdrop of the San Francisco Bay. The party starts Wednesday, September 2 at 7:30PM .


VMware Advocacy

VMworld 2015 Content Catalog is now live!

What sessions will you attend? The #VMworld 2015 Content Catalog is now live!

VMworld 2015 Content Catalog is now live!

The Content Catalog allows prospective VMworld attendees access to the VMworld agenda, with the ability to peruse breakouts and note sessions of interest. You can search and filter to your heart’s content—by track, category, session format, industry, role, technical level, speaker name, location (US or Europe), and keyword search. You cannot schedule sessions in the catalog.


VMware Advocacy

Change Active Adapters and Load Balancing using ESXi Shell

Anyone ever had a network administrator make your hosts lose connectivity and the only option you have is to use the ESXi shell through an HP ILO port? I have. Here are the changes that were helpful to me at the time.

freakout

esxcli network vswitch standard list — This lists all of the vswitches on the host.
listvswitch

esxcli network vswitch standard uplink add -u vmnic1 -v vSwitch0 — Add an uplink to the vswitch. Just switch “add” to “remove” to get rid of a vmnic from the vswitch.
adduplinktovswitch

esxcli network vswitch standard policy failover get -v vSwitch0 — Get the load balancing policy for the vSwitch.
getvswitchpolicy

esxcli network vswitch standard policy failover set -a vmnic1 -v vSwitch0 — Be careful because when changing the active vmnics on a vswitch it will only add ones that you specify and if there are active ones that you specify it will move them to unused.
Incorrect
setactivevmniconvswitchfail
Correctsetactivevmniconvswitch

esxcli network vswitch standard portgroup policy failover get -p “Management Network” — This shows the policy information for the portgroup “Management Network”
getportgrouppolicy

esxcli network vswitch standard portgroup policy failover set -a vmnic1 -p “Management Network” — Change the active adapter on the Management Network.
setactivevmniconportgroup

esxcli network vswitch standard policy failover set -l iphash -v vSwitch0 — Set the load balancing on the vSwitch0 to iphash. Please be aware that this might change the portgroups to iphash if they are set to get their settings from the vswitch.
setloadbalanceonvswitch

esxcli network vswitch standard portgroup policy failover set -l iphash -p “Management Network” — Sets the load balancing policy on the Management Network portgroup to iphash. This will cause the load balancing policy on the vswitch to no longer apply to the Management Network portgroup.
setloadbalanceonportgroup

PowerCLI shutdown/startup script for remote sites.

In my environment I have 11 Remote Locations that each have one ESXi host.  I had a problem with ILO firmware updates the other day where I had to reboot these remote hosts.  I really didn’t want to use the VIC to click on each of the VMs to shut them down, so using a lot of help from mpreston’s site I came up with a script that will shut down all of my VMs and then when ready, start the VMs again.  My script shuts down “Front End” VMs first, then DB VMs, and then finally DCs.  I have a modified version that allows me to do the same for my Datacenters that I will share later.  You might have to edit the script if your database and or DCs are named differently.

add-pssnapin VMware.VimAutomation.Core
connect-viserver servername #Insert your vcenter server

#Variables
$datacenter = “datacenter”  #Add all datacenters here with a comma and quotations between each example “datacenter1”, “datacenter2”
$filename = “c:\vmshutdown\poweredonvms.csv”  #You must create this file first…it won’t create it for you.

#Get a list of the hosts being rebooted and set them to maintenance mode.  This keeps the VMs from powering on again.
Get-VMHost -location $datacenter | set-vmhost -State Maintenance

#Get a list of powered on VMs
get-vm -location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | Select Name | Export-CSV $filename

#Get a list of VMs that are not Database servers or DCs.  
Write-Host “”
Write-Host “Retrieving a list of powered on VMs…” -ForegroundColor Green
Write-Host “”
$poweredonguests = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -notlike “*DB*”} | where {$_.Name -notlike “*DC*”}

#Power off servers that are not Database and DCs
foreach ($guest in $poweredonguests)
{
Write-Host “Processing $guest…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $guest.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $guest, these will be powered off” -ForegroundColor Yellow
Stop-VM $guest -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $guest”
$vmshutdown = $guest | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for VMs that are not DB or DC to shutdown.”
Write-Host “”
Sleep 180

#Shut down DB Servers at remote sites
Write-Host “”
Write-Host “Shutting Down DB Servers”
Write-Host “”
#get our list of DB servers that are powered on…
$poweredondbs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -like “*DB*”} | where {$_.Name -notlike “*DC*”}
foreach ($dbs in $poweredondbs)
{
Write-Host “Processing $dbs…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $dbs.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $dbs, these will be powered off” -ForegroundColor Yellow
Stop-VM $dbs -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $dbs”
$vmshutdown = $dbs | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for Database VMs to shutdown.”
Write-Host “”
Sleep 180

#Shut down DCs
Write-Host “”
Write-Host “Shutting Down DCs”
Write-Host “”
#get our list of DB servers that are powered on…
$poweredondcs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOn”} | where {$_.Name -like “*DC*”}
foreach ($dcs in $poweredondcs)
{
Write-Host “Processing $dcs…” -ForegroundColor Green
Write-Host “Checking for VMware Tools Install” -ForegroundColor Green
$guestinfo = get-view -Id $dcs.Id
if ($guestinfo.config.Tools.ToolsVersion -eq 0)
{
Write-Host “No VMware tools detected in $dcs, these will be powered off” -ForegroundColor Yellow
Stop-VM $dcs -Confirm:$false
}
else
{
Write-Host “VMware tools detected. I will attempt to gracefully shutdown $dcs”
$vmshutdown = $dcs | Shutdown-VMGuest -Confirm:$false
Sleep 5
}
}

#Waiting for these shutdowns to complete
Write-Host “”
Write-Host “Giving VMs 3 minutes for DCs to shutdown.”
Write-Host “”
Sleep 180

#Time to restart the hosts
get-vmhost -location $datacenter | restart-vmhost -runasync -confirm $false

Now that you have rebooted your hosts, it is time to start all the VMs again.  You are going to use the dame variable that we created above.

#Take hosts out of maintenance mode.  
get-vmhost -location $datacenter | set-vmhost -state connected

#Power On DCs
Write-Host “”
Write-Host “Powering On DCs”
Write-Host “”
#get our list of DCs servers that are powered off…
$poweredoffdcs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”} | where {$_.Name -like “*DC*”}
foreach ($dcs in $poweredoffdcs)
{
Start-VM $dcs -Confirm:$false
Sleep 5
}

#Power On DB Servers
Write-Host “”
Write-Host “Powering On DBs”
Write-Host “”
#get our list of DB servers that are powered off…
$poweredoffdbs = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”} | where {$_.Name -like “*DB*”}
foreach ($dbs in $poweredoffdbs)
{
Start-VM $dbs -Confirm:$false
Sleep 5
}

#Power On the rest
Write-Host “”
Write-Host “Powering On Remaining VMs”
Write-Host “”
#get our list of VMs that are still powered off…
$poweredoffguest = Get-VM -Location $datacenter | Where-Object {$_.PowerState -eq “PoweredOff”}
foreach ($guest in $poweredoffguest)
{
Start-VM $guest -Confirm:$false
Sleep 5
}

 You remote locations should be back up!

False alerts after upgrading ILO firmware from 1.55 to 1.8

There are two separate alerts that I have experience from these upgrades.  Please note that after the update the ILO cards were rebooted, but our esxi hosts running vsphere 5.1 were not rebooted.

Error1:

Out of 31 hosts that had firmware updated, this error has appeared on 6 so far.  Some of them took a couple days before they presented with the alert.  The alarm triggered is: Host Memory Status.  Under the “Hardware Status” tab the alert shows for the System Board 8 Memory: Uncorrectable ECC.

memoryalarm

uncorrectable

 

Error2:

The second problem that I have encountered is the filling up of the IPMI SEL Log.  I am able to go in an clear the log which gets rid of the alert for a short time, but the log fills up again.  The alert shows as Host IPMI System Event Log Status.  Under the “Hardware Status” tab the “System Event Log” and “IPMI SEL” show as Unknown.  You can click on “Show Event Log” and then “Reset Event Log” and it will clear for a while…but the alert will return.  Notice the future date of 12/31/9998 which I am guessing might be when the world ends.

hostipmistatus

ipmisellog
thefuturelogs

How do I fix these problems?  After calling HP and VMware I was told that I needed to put each host into Maintenance Mode and then run a “Detailed Hardware Diagnostic”.  According to VMware this was the only way to clear the error (especially the memory one).  The solution that ended up working for me was to just reboot the host…:)  The VMware couldn’t believe that worked, but it did.  I know it isn’t a difficult fix, but maybe this might help others that get this alert.

Please let me know if you have encountered similar alerts from upgrading the firmware on your ILO ports.

Failing/Failed Disk And Detailed Information From HPACUCLI

Today I was alerted by vCenter with a yellow, “Host Storage Status” alarm.
diskalert

Clicking the ESXi host in the alert I find from the description that I can check the Hardware Status tab of the host for more details
diskalertdetail

I see that the disk is failing and that RAID is in “Interim Recovery”.  This doesn’t give me much information about the failing disk though.  Maybe in HP ILO I can find more information?  Nope, ILO doesn’t seem to have any information about the disks in the host.  According to the ILO System Information, everything looks just fine.
nodiskilo

I called to open a ticket with HP and while on the phone the support technician mentioned a diagnostic tool called HPACUCLI.  If you installed your host a HP customized ESXi image, then this should be there for you.

SSH into your host then type cd opt\hp\hpacucli\bin.  If you type ls you should see the script called “hpacucli”.  Type ./hpacucli and then press “Enter”.  You should see something about detecting controllers and you should be at a new prompt.
hpacucli1

Type controller all diag file=/tmp/adu1.zip and then press “Enter”.
hpacucli2

Now you will need a program like WinSCP that will get the newly created zip file off the host.  Connect to the host with WinSCP and then navigate to the /tmp folder.  Click on the adu1.zip file and the click “Download”.
tmpadu

Looking into the zip file we see four files; double click on “ADUReport.htm”.  The report should open into your browser and should have a lot of information.  Towards the top of my report I see that the drive has failed and if I scroll down I can actually get some in depth information about the drive.  Things like the model and Serial.
Drivefaileddrivefailed2

Can’t power on VM in Workstation 10 after Windows 8.1 October Rollup KB2995388

This issue has been fixed with the release of VMware Workstation 10.0.4.  VMware recommends that if you made changes to the config.ini that you remove the entry or change the value to “FALSE”.  The Workstation download can be found here.

When powering on a VM in Workstation 10 after rollup KB2995388 I see the following.

workstationerror

VMware is aware of the issue it would seem and there are three workarounds so far.  VMware official blog has this about it.

1.  Uninstall KB2995388 (This is an optional update).
2.  Edit the config.ini found in C:\ProgramData\VMware\VMware Workstation         and add the line:  vmmon.disableHostParameters = “TRUE”
3.  This is untested by me but it is reported that if you open Workstation in                   administrative mode (with admin rights), that it won’t have the problem.

Reboot your computer and your VMs should work again.

I did notice after making the change that the first time I start my VMs I get:
devicenotavailable notrespondingdevice

I chose “No” on both and the VMs now boot without issue.

Extending VMware VMDKs with System Partitions

hard-disk-drive
I have had to deal with extending System Partition VMDKs within our environment, but this can get tricky especially with 2003 Server and below.  Hope this helps…Note that Data drives can be extended without issue on 2012 and below.

2008 and above
1.  Open your VIC and edit the settings of the VM that you would like to modify.
Webeditsettings
2.  Change the system disk to the new size (Usually Hard disk 1).
Disk Change
3.   Log into your VM using the console or RDP.
4.  There are two ways to do the extend.  If it is 2008 R2 and above I usually do option 1.  2008 Standard and below I use option 2.

Option 1:
a.  Open Server Manager.
ServerManager
b.  Click Disk Management.
c.  Usually you will have to do a rescan to see the unallocated space.  Right click Disk Management and click Rescan Disks.
Rescandisks
d.  Right click the drive that you wish to expand (should show unallocated space) and click Extend Volume.
ExtendVolume
e.  Click OK on the Extend Volume Wizard and then again click OK on the Select Disks screen.  Now click Finish to complete the wizard.
f.  You should now see the drive is larger.

Option 2:
a.  Click Start –> Run and then type cmd.  Right click on the command prompt application and choose Run as Administrator.
runadmincmd
b.  Type diskpart.
diskpartsmall
c.  Type list volume.
diskpartlistvolume
d.  Select the system partition by typing select volume #.
diskpartselectvolume
e.  You should see an asterisk by the selected volume.  Type extend.
diskpartextend
f.  The drive should now be extended.  Type exit.

2003 Server and Below
1.   Have a 2008 R2 Helper machine that you will use to expand the drive.
2.  Power down the 2003 Server VM.
3.  Edit the settings and add space to the system partition.
Disk Change
4.  Power on the 2003 Server VM.
5.  Click Start –>All Programs –> Administrative Tools –> Computer Management.
ServerManager03
6.  Click Disk Management.  You should see the unallocated space, but if you do not then right click Disk Management and click Rescan Disks.   Just verify…don’t do anything with the space yet.
Rescandisks
7.  Shutdown the 2003 Server VM.
8.  Edit the 2003 Server VM within the VIC and note the location of the disk on the datastore.  Click remove on the drive that you would like extended by clicking on the X next to the hard disk.  DO NOT CHOOSE DELETE FILES FROM DATASTORE or REMOVE FROM VIRTUAL MACHINE AND DELETE FILES FROM DISK.  This is different wording depending on the Full VIC or the Web VIC.  Click OK.
RemoveDisk
diskremoved
9.  Now find your 2008 R2 helper VM within the VIC and then edit the settings.
10.  Where it says “New Device” choose Existing Hard Disk from the menu.  Click Add.
Addexistingdisk
11.  Using the information that you noted before, find the location of the 2003 Server disk that you removed before, select the disk, click OK.
choosevmdk
12.  Cick OK and this will mount the 2003 Server disk on your 2008 R2 helper VM.
13.  Log into the 2008 R2 helper VM with the console or RDP.
14. Click Start –> Run and then type cmd.  Right click on the command prompt application and choose Run as Administrator.
runadmincmd
15.  Type diskpart.
diskpartsmall
16.  Type list volume.
diskpartlistvolume
17.  Select the system partition by typing select volume #.
diskpartselectvolume
18.  You should see an asterisk by the selected volume.  Type extend.
diskpartextend
19.  The drive should now be extended.  Type exit.
20.  Go back into the VIC and edit the 2008 R2 helper VM.
21.  Click the X next to the Hard disk that you added earlier to this machine.  Again, DO NOT CHOOSE DELETE FILES FROM DATASTORE or REMOVE FROM VIRTUAL MACHINE AND DELETE FILES FROM DISK. 
RemoveDisk
diskremoved

22.  Edit the 2003 Server and then from the “New device” menu click Existing Hard Disk and then click Add.
Addexistingdisk
23.  Find the vmdk that you just extended then click OK.  Click again on the settings screen.
choosevmdk
24.  Power on the 2003 Server and you should see the additional space.  You might get a message indicating that a reboot is needed.  If requested, do the reboot.

Random Issues I Have Run Across
I did have an issue the other day with one our our 2008 Standard Servers.  When I went to extend the drive it gave me a weird error.  The parameter is incorrect.
DiskGrowParameterIncorrect

I give thanks to the Lord every day for the googles.  I found a great post by Jonathan Medd that gave the simple fix for this.  When you are in diskpart, select the volume # that you are extending and then use the command extend filesystem.

SRM Testing…What do I do with my Domain Controllers? Part 2

Now that you have successfully prepared for the script…here it is. This script will power down the previous clone if there is one, delete the old clone, create a new clone, change the network, and then power on the clone.  Please make sure you test this in first in a test environment before deploying to production.

RedNotice

 

########################################################################
# Clone DC VM and set network to Test Bubble (clonedc.ps1)
#
# This script is used to clone a DC and then set the network port group # to the #Test Bubble group.
# The purpose of this script is to have a replica of our primary DC for # use #when testing an SRM failover. This will remove the previous clone # before #creating a new one.
#
# Created By: Matt Jones, 2014
#
# Variables: $newVMName – Name of the new clone made from the original VM
# $sourceVMName – Name of the target DC VM that will be cloned
# $VM — Name of VM that was cloned earlier
# $DestGroup — This is the port group that the network will be changed to.
#
# Usage: ./clonedc.ps1
#
#######################################################

add-pssnapin VMware.VimAutomation.Core
Connect-viserver (Your vCenter Server)

#Clone Script#
$newVMName = “DCClone”
$sourceVMName = “SourceDC”

Get-VM $newVMName | Stop-VM -Confirm:$false
Start-Sleep -s 5
Get-VM -Name $newVMName | Remove-VM -DeletePermanently -Confirm:$false
Start-Sleep -s 10
New-VM -Name $newVMName -VM $sourceVMName -Location Foldernamecreatedearlier -vmhost hostname
Start-Sleep -s 120

#Set Test Network#
$VM = get-vm “DCClone”
$DestGroup = “Test Bubble”
$VM | get-networkadapter | set-networkadapter -NetworkName $DestGroup -confirm:$false
Start-Sleep -s 30
$VM | Start-VM

taskkill /F /IM powershell.exe

#######################################################

Once you have created the clonedc.ps1 script we need to get it set up as a scheduled task on the physical machine. Log in using the script AD account that you created earlier, then go to Control Panel –> System and Security –> Administrative Tools –> Task Scheduler.  Click Create Task.

TaskScheduler

TaskCreate

Under the General Tab Name your new task, change this to Run whether user is logged on or not, and check the box Run with highest privileges.  Click the Triggers tab

TaskGeneral

Click New and then choose the frequency that you would like this to run.  Make sure to check the Stop task if it runs longer than box and then set it for 30 minutes.  This makes sure that the script stops and doesn’t continually run.  Make sure you check the Enabled box or the script won’t run automatically!  Click the Actions Tab.

TaskTrigger

Click New and then from the drop down menu choose Start a program.  In the Program/Script box type C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe and in the Add arguments (optional) box type -psc “C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\vim.psc1” -noe -c “. \”C:\scripts\clonedc.ps1\””.  Click the Settings Tab.

TaskAction

If you want to allow this to run on demand then check the Allow task to be run on demand box.  Check the Stop the task if it runs longer than box and choose 1 hour from the drop down.  Check the box If the running task does not end when requested, force it to stop.  Click OK.  You will be prompted to enter the credentials for the account running the task and then click OK.

TaskSettings

RunTaskAccount

 

You might have to do a refresh for your new task to appear.  I am guessing this might be a bug.  Manually run your task and monitor the VIC to make sure it is working properly.

Refresh

That should be it.  Now you can run an SRM test fail over and have a DC available.  Make sure that you have removed all of your DCs from SRM if you had them in there.

Yes