Tag Archives: vSphere 5.1

Enabling EVC (Enhanced vMotion Compatibility)

It has been a while since I have had to enable EVC, but I needed to the other day in the office.  I created a cluster with a HP DL380 G7 and an older HP DL380 G5.  When I tried to turn EVC on for the cluster, I ran into this error.  “The host cannot be admitted to the cluster’s current Enhanced vMotion Compatibility mode.  Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode.”  This message is telling you that the current machines that are powered on are using the technology from the newer processor and in order to turn on EVC for the cluster, the VMs need to be powered off.  So…I powered off all of my VMs on the DL380 G7 (newer host).EVC1

After all VMs are powered down, right click on the cluster and select Edit Settings.

Click the VMware EVC on the left pane and the click Change EVC Mode… button.EVC3

I have Intel processors so I selected Enable EVC for Intel Hosts.  Now I get a green check under the Compatibility pane.  Looking good!EVC4

Now depending on the processor generation, you have to change the EVC Mode.  For mine, I chose the Intel “Penryn” Generation” and I still had a green check box.  If your hosts don’t support the EVC mode, it will let you know in the Compatibility pane.  The processor support documentation can be found here.EVC5

We now see that Intel “Penryn” Generation is my EVC mode.   The only thing left to do is power on the VMs and start your migrations!EVC6


Enable Copy and Paste Through the VMware Console

I was setting up a new host the other day and I received a call from one of my admins letting me know that they could not copy/paste within the console; but they can copy/paste in RDP.

This is a simple fix found in KB1026437.  You can make the change on an individual VM, but I think it is best to change it on the host (which applies to all VMs).  I really wish the default would have this enabled.

Open a Putty session…if you don’t have putty then get it here.

1. Log into the ESXi host that you want to change.
2. Type vi /etc/vmware/config
3. Arrow down to the last line and type which stands for “insert”.
4. Add the lines:
         vmx.fullpath = “/bin/vmx”
5. Press the ESC key and then type :wq which stands for “write and quit”.

The next time each VM is power cycled it will enable the copy/paste functionality.  Keep in mind that if you ever upgrade this host to a new ESXi version that this setting will go back to the default of disabled and you will have to add this line again.

Deploying VMware Support Assistant v5.5

The other day I set up and configured VMware’s new Support Assistant 5.5.  I have used older versions to do things like open tickets and pull log files, but the new version has proactive support built in.  You configure when you want your log files sent to VMware and they compare them to know issues.  VMware even includes the ability to scrub the files before sending them out.

Here is how to deploy.

  1. Download the .OVA from www.myvmware.com. I am using build 1549662.
  2. Open up your VIC.
  3. Click File and then Deploy OVF Template.
  4. Select the OVA that you downloaded and click Open.
    1. Click Next on the Source Location window and then Next again on the details screen.
    2. Click Accept and then Next.
    3. Name your new support appliance and choose a folder if applicable then click Next.
    4. Select the host that the appliance will run on and then click Next.
    5. If applicable, choose a resource pool for this appliance and then click Next.
    6. Choose the Destination Network and then click Next.
    7. Enter in the Gateway, DNS, IP, and Subnet Mask and then click Next.
    8. On the Ready to Complete screen check the Power on after deployment box and then click Finish.
    9. You should see a progress bar indicating the status of the deployment.
  5. Open a console screen and you should see the appliance boot and eventually tell you to browse to the appliance IP to finish configuring. Open up a browser and go to that address.
  6. Accept the EULA…actually relatively short. Click Next.
  7. The lookup service address is the SSO server. Enter your SSO server and then click Next.
  8. Enter your SSO credentials which will usually be either admin@System-Domain if you installed SSO in 5.1 or administrator@vsphere.local if you upgraded from 5.1 to 5.5. Click Finish.
  9. Add an account that has rights to vCenter and then check the box to Assign log collection permissions for the following vCenter Server instances. Click Next.
  10. Enter your proxy information if needed. This allows the appliance to talk with VMware and send the log files. Click the Test Connectivity button to ensure things are working properly and then click Next.
  11. Add an email address to receive update about your environment then click Finish.
  12. Hopefully you see that the Service is ready…log gathering is disabled though. We will fix that in just a minute, but first let’s not forget to change the root password. Click VA Settings. You will see a place to put in the current password and then a new password. This is for the root account. The default password is vmare and make it something that you will remember. Click Save.
  13. The Support Assistant only works in the new Web VIC. Open a browser and navigate to your vCenter server and login.
  14. You should see a new icon now that looks like a life preserver called vCenter Support Assistant. Double click the icon.
  15. Click the link Configure data collection.
  16. Here you can change when your appliance will upload logs to VMware for analysis. The default is never, but that won’t help us! In the example mine will upload every Monday at 10AM. Click Entity Selection.
  17. Select the vCenter servers (linked mode supported) and hosts that you want logs from that will be sent to VMware. Click Data Scrubbing.
  18. This is great that VMware includes the ability to scrub the log files. Note that if you check these boxes, it will have an effect on the virtual appliance as it looks through and redacts the log files before sending to VMware. Check the boxes that you require and then click OK.
  19. Click the Monitor tab. This tab shows the status of your support uploads. Click the Manage tab.
  20. This gives a summary of your proactive support settings, which you just changed. Click the Support Requests button.
  21. After logging in with your Myvmware.com username and password, you can check the status of open and closed tickets and upload logs if needed.



Proliant DL380 embedded NICs missing after firmware update

I ran into a very strange issue today when I went to redeploy an old Proliant DL380 G5. The first thing I did was use the most current service pack DVD to update the firmware. The most current is from 2/2014 and has the number 2013.02.0. After installing ESXi 5.1 U1 I noticed that I was only showing 4 NICs and not the 6 I started with.

The two embedded NICs were missing!!

After a quick google search or twelve I stumbled upon an HP discussion with exactly the same problem that I was having. I followed the instructions from the HP discussion and here is what it took to fix (Most of this is copied from user hase3d’s post).

1. Download all necessary tools
     – download FreeDOS
     – download XDIAG.exe 
     – download bc08c740.bin 
     – read all information in setup.txt

2. Prepare the FreeDOS.iso
     -After downloading open the iso with a tool like UltraISO. I used Magic ISO.
-Add the XDIAG.exe and the bc08c740.bin to the iso – I these files to the          root so that I wouldn’t need to add a path later.
-Save the iso with a new name.
-Burn it or mount it with ilo.

3. Boot from FreeDOS
     -Select Install to harddisk
     -Press 1
     -Select your language and press Enter
    -Press ESC
    -Select run FreeDOS from CD-ROM

4. Mine booted to f:\freedos. Do a cd\ to get back to the root of f:
5. Run xdiag in engineering mode by typing xdiag -b06eng
6. type device 1
7. nvm fill 0 0x600 0
8. nvm upgrade -bc bc08c740.bin
9. nvm cfg
     -Press q
     -Type default
     -Press q again
     -Type 16=10 wich sets the BAR size to 32
     -Press q for the third time
     -Type save and then exit out to the main menu

10. Type device 2  and repeat steps 7-9, run the command 1=00:00:18:xx:xx:xx <— change the last digit for different mac on device 2.

I did not do anything else from the setup.txt file.

I powered down the host and then when I rebooted I had 6 NICs again!

The authentication server returned an unexpected error

I came in this morning only to be greeted by my web client telling me that I can’t login because it can’t create SAML 2.0. I am not sure that I really want it creating SAML 2.0….I don’t know SAML 1.0. Ok, bad joke. Here was the message…

I found KB2034798 at which point I remoted into my SSO server and checked the imsTrace.log for “NetUserGetLocalGroups”. I didn’t find it…so the KB didn’t apply to me…L

After some more googling I found this blog post that indicated that references KB2043070. The idea is that there is a local identity source within SSO that it is trying to authenticate the users to. You have to login with the admin@system-domain account and password. Hopefully you saved this when setting up your SSO server. The only problem I had was that I didn’t have this local identity source to remove.

I thought to myself, that there might be a stale identity source on the list that it is authenticating to. I was talking to a coworker and they mentioned that there was a domain that was deleted the day before. AHAH!! I clicked on the identity source of the domain that had been removed and then clicked “Test Connection”. There was an error that didn’t tell me much.

3-12-2014 2-42-32 PMI cancelled out and was back at my list of identity sources. I selected the identity source that had been removed from AD and I hit the red X, “Delete Identity Source”. You will get a prompt asking for you to confirm. One thing to note is that the identity source that I deleted was not one of the default domains at the bottom. If you haven’t set a default domain up, I would do that now. I am wondering if there might be a bug that uses the identity source at the top of the list instead of the default at the bottom. After deleting the state Identity Source I was able to login again.

vSphere HA detected that host is in a different network partition than the master

Target: Host
Previous Status: Green
New Status: Red
Alarm Definition:
([Event alarm expression: vSphere HA agent on a host has an error; Status = Red] OR [Event alarm expression: vSphere HA detected a network isolated host; Status = Red] OR [Event alarm expression: vSphere HA detected a network-partitioned host; Status = Red] OR [Event alarm expression: vSphere HA detected a host failure; Status = Red] OR [Event alarm expression: Host has no port groups enabled for vSphere HA; Status = Red] OR [Event alarm expression: vSphere HA agent is healthy; Status = Green])
Event details:
vSphere HA detected that host (host) is in a different network partition than the master (Cluster) in Datacenter

I had been getting this message randomly over the last couple months on some of my datacenter hosts. These alerts didn’t seem to be causing any problems within the cluster, but I wanted to get to the bottom of this. I opened a ticket with VMware and uploaded the logs from both the host and vCenter, but they didn’t see anything out of the ordinary. On the second webex with VMware I noticed a couple strange things with the management network that might be the cause.

  1. The first thing I noticed was that the NICs were set for “Auto Negotiate”. I originally set up our environment on ESXi 4 before upgrading to ESXi 5.1. When I initially set this up I hard coded (KB1004089) these to 1000GB/Full. I am wondering if at some point during the upgraded that they defaulted back. On our switches it was set at 1000GB/Full so it is important that we set this on the host NICs to 1000GB/Full as well.
  2. The second thing that I noticed that in the Management network that I had the Load Balancing set to “Route based on IP hash”. The problem here is that for this to work correctly you need a port channel configured (I do not have this configured this way). This might be the cause of the HA problem if the traffic is going across these NICs is getting confused because of the Load Balancing configuration. I changed this to “Route based on the originating virtual port ID”, which makes the traffic go out on the port that it came in on. There is a good read found here…http://blogs.vmware.com/kb/2013/03/troubleshooting-network-teaming-problems-with-ip-hash.html.

This case is still ongoing with VMware and I should know in the next couple weeks if this solves my problem; my gut tells me it will.

Creating a template for Server 2012 R2 – Part 1

I have borrowed items from http://www.boche.net/blog/index.php/2012/08/16/microsoft-windows-server-2012-tips/ to create this post. I encourage you to take a moment to check out that post.

This is the step by step document that I used to build my 2012 and 2012 R2 VMs.

VM values will start at:

Hardware: Value:
Memory 4 GB
CPU’s 1
Video card Auto-detect video settings
VMCI device None
SCSI Controller 0 VMware Paravirtual
Hard disk 1 40 GB, Thin
CD/DVD Drive 1 Client Device
Floppy Drive 1 Removed (when done)
Network Adapter 1 VMXNET3
General Options OS: Microsoft Windows Server 2012 and 2012 R2
VMware Tools Default Settings
Virtual Machine Version 9

Creating the VM

  1. In vCenter click “File” then “New” then “Virtual Machine”
  2. Choose the “Custom” radio button and then “Next”.
  3. Name the SM and choose the folder location.
  4. Choose the datastore for the VM and then click “Next”.
  5. Make sure that the “Virtual Machine Version: 8” radio button is selected. (5.1 is using version 9 so I am not sure why this can’t be selected here. We will change this later.
  6. Select the “Windows” radio button and then choose “Microsoft Windows Server 2012 (64-bit).
  7. Take the default of 1 virtual socket and 1 core per virtual socket.
  8. Take the default of 4GB memory and click “Next”
  9. Choose your Network and change the adapter to “VMXNET 3” then click “Next”.
  10. Change the SCSI controller to “VMware Paravirtual” and then click “Next”.
  11. Select the “Create new virtual disk” radio button and then “Next”.
  12. Take the default of 40GB and click “Next”.
  13. Take the default virtual device node. For the system partition you want this to be SCSI 0:0. Click “Next”.
  14. On the summary screen click the “Edit the virtual machine settings before completion” box and then click “Continue”.
  15. Click on the Video Card and then change the radio button to “Auto-detect settings”.
  16. Click on the CD/DVD and then choose the datastore location that you have the 2012 R2 install ISO. Make sure under Device Status that “Connect at power on” is checked. Now click “Finish”.
  17. Right click on your newly created VM and click “Edit Settings”.
  18. Click on the Floppy drive 1 and choose the “Use existing floppy image in datastore” radio button. Then click “Browse”. At the bottom of the datastores you should see a folder called “vmimages”. Double click this folder. (For some reason until the VM is created this folder does not show up and that is why we had to create the VM and then go back into the settings to change this).
  19. Double click on the “floppies” folder.
  20. Choose the “pvscsi-Windows2008.flp” and then “OK”.
  21. The Floppy drive 1 settings should look like this and then click “OK”.
  22. On the list of VMs click the one you are building and then click the “Power On” button.
  23. Now click the “Open Console” button.
  24. The VM should boot into the 2012 R2 setup screen. Choose your language and then “Next”.
  25. Click “Install”.
  26. Choose the version of server that you are using. We use the Datacenter here. Then click “Next”.
  27. Agree to give up your first born to Microsoft by clicking the “I accept the license terms” box and then click “Next”.
  28. Choose “Custom: Install Windows only (advanced).
  29. Uh oh, there is no location to install Windows. Luckily you configured the floppy drive 1 earlier right? Click the “Load Driver” button.
  30. Click “Browse”.
  31. Look for the Floppy Disk Drive and then double click “amd64”.
  32. Select the “VMware PVSCSI Controller (A:\amd64\pvscsi.inf) and click “Next”.
  33. Hey look there is our drive!! Click “Next”.
  34. Windows should now be installing.
  35. Enter a password for your admin account. Do not lose this password!
  36. You should have the login screen now.

Installing APC Network Shutdown for ESXi – Part 3

In Part 3 we are going to install Powerchute Network Shutdown on the OVA that we deployed, then we are going to configure it to shut down the VMs in case of a problem.

  • See APC pdf FA159776. Open Putty.exe, insert the name or IP of the VMA you just deployed, and then click “Open“. Click “Yes” if you get a security alert. Login with vi-admin and your password that you set earlier.
  • Create a temp directory in opt using the command (You will be prompted for the vi-admin password): sudo mkdir /opt/temp
  •   Next we need to change the permissions to this temp directory: sudo chmod 777 /opt/temp
  • Now to check the permissions: ls -la /opt The permissions should now read drwxrwxrwx
  • Now using WINSCP we need to transfer the .tar.gz file that we downloaded earlier up to the ESXi host. Enter the appropriate information and then click “Login“. Click “Yes” or “Proceed” if prompted with a security warning.
  • Check the “Never show this banner again” box and then click “Continue“. You should now see a screen with two windows. The window on the left is your local computer and the screen on the right is the VMA. Navigate on the left window until you find the .tar.gz file.
  • On the right window the drop down where it says “vi-admin“. Change this to /<root>. Then navigate to “opt–>temp 
  • Drag the .tar.gz file from the left window to the right window. Click “Copy” when prompted.
  • Verify that the file has been copied successfully.
  • Now go back to Putty.exe and we are going to uncompress the file. The commands are: gunzip pcnsname.tar.gz then: tar -xvf pcnsname.tar
  • Use the ls-la command and you should see a new ESXi folder. Use the command cd ESXi to change to this folder.
  • List the contents of ESXi with the ls -la command. We need to change the permissions for the installation file: sudo chmod 777 install_en.sh
    Now do another ls -la to see that the permissions have changed to rwxrwxrwx.
  • Now we are ready to install PCNS. Use the command: sudo ./install_en.sh
    Press “Enter” and then use the “z” key to scroll to the end of the agreement. If you agree then type “yes” and then press “Enter“.
  • Accept the default installation path (or insert a different one if you prefer). Press “Enter“. Type “yes” and “Enter” that you are sure about the path.
  • Take the default for the java directory. Press “Enter“.
  • Next the installation looks for the ESXi host that will be shut down. First add the IP of the host and then it will ask for the username and password for the host to make this change.  Update:  Almost all of the deployments failed to add the ESXi host here, so I would choose “q” to skip and then at the command line do: sudo vifp addserver <hostname/ IP address of ESXi host>
  • Verify that the server has been added with the command: vifp listservers
  • To ensure Powerchute can shutdown the VMs on the host, we need to add the ESXi host to the fasspass. Use the command: vifptarget -s <server name or ipaddress>
    Now type the command: vicfg-nics -l
    You should see a list of nics on the ESXi host.
  • One the server has been added you should be able to open a browser and go to the powerchute configuration wizard: https://vmahostnameorip:6547
  • Click “Next” and you should see the Configuration Wizard: Security page. Insert the username and password and the authentication phrase. This must match the card in your APC device. By default this is apc/apc with the passphrase: “admin user phrase” then click “Next“.
  • On the UPS Electrical Configuration page choose the correct configuration for your company and then click “Next“.
  • On the UPS Details page choose the protocol, port, and IP for the APC network card.
  • On the Miscellaneous page check the box for “Automatically check for updates to PCNS” and then click “Next“.
  • Confirm the details and then click “Apply“.
  • Hopefully you see that the computer is now protected. Click “Next“.
  •  You should now see that the wizard is complete, now click “Finish”.
  • You will now see the main page for the Network Shutdown. Click “Configure Events” and then click the check box for “Shutdown System” on “UPS: On Battery“.
  • The “Shut Down Operating System” page will display and input 300 into the “Shut down the PCNS operating system only when the event lasts this long (seconds)
  • Finally, we need to set up the virtual machine shutdown options on the ESXi host. Open the vSphere Client, select the host, and then choose the “Configuration” tab. Under the “Software” pane click on “Virtual Machine Startup/Shutdown“.
  • In the top right corner click “Properties“. Click the box “Allow virtual machines to start and stop automatically with the system“. Set the shutdown delay (120 default) and then set the shutdown action to “Guest Shutdown“.
  • Leaving VMs under the Manual startup will make it so when the host turns back on, the VMs will not start up by themselves. Usually you want to make sure power is restored and stable before bringing up VMs. You can change your VMs to start automatically if you really wanted to. 


Installing APC Network Shutdown for ESXi – Part 2

In Part 2 we are going to configure the OVA that we just deployed

  • Click on your new VMA and then click the “Open Console” button.
  •  There should be a Network Configuration menu. I have found that if I set the gateway first that it will not save when I set the IP. I am going to set that last. Choose option “3” to set the hostname.
  •   Make your hostname match your VM name.
  •  Select option “4” to set the DNS servers. Type the appropriate primary (Server 1) DNS IP and then press “Enter“. If used, also add your secondary (Server 2) DNS IP and press “Enter” again.
  • Select option “6” to set the IP for eth0. I only use IPv4 so type “n” to not configure IPv6, then “y” to configure IPv4, and then “n” to not use DHCP. Type the IP and Subnet for your VMA and then “y” to confirm it is correct.
  • Now I set the Gateway. Choose option “2” and the press “Enter” to set the gateway for eth0. Type the IP of your IPv4 Default Gateway and the press “Enter“. Press “Enter” again to skip the gateway for IPv6.
  • Choose option “1” to “Exit this program“. This will boot the VMA with the network settings that we just configured.
  • Next the VMA will ask for the old password for the vi-admin account. Press “Enter” for the Old Password. Then type your new password “Enter” and then retype it when prompted. “Enter” again.
  • The VMA should boot and you should see the following screen. Browse to https://VMA-IP:5480 to verify connectivity. 

Great, now you have configured the new VMA, it is now time to install Powerchute in Part 3.

Installing APC Network Shutdown for ESXi – Part 1


  •  Create a password for the vi-admin account.
  • Download VMA OVA from VMware.
  • Download the Powerchute Network Shutdown for ESXi from www.apc.com. The most current version at the time of writing this is v3.0.1.
  • Download and install putt.exe.
  • Install WinSCP on your local machine. This will be used to put the tar.gz file that you just downloaded from APC on your Esxi host.
  • Make sure the vSphere Client is installed on your machine.


  •  On your vCenter server click “File–>Deploy OVF Template“.  
    Choose the location of your ovf. Click “Next“.

  • Verify the details and click “Next“.
  •  Click “Accept” and then “Next“.
  •  Name your VM and then choose the inventory location for the VM. Click “Next“.
  •  Choose the host you wish to deploy to and then click “Next“.
  •  We don’t use resource pools. Select the top level cluster and then click “Next“.
  •  Select the datastore to deploy the VM to and then click “Next“.
  •  I prefer thin provisioning…especially for the VMA’s. Click the “Thin Provisioning” radio button and then click “Next“.
  •  Choose the appropriate source network and destination network and then click “Next“.
  •  Choose the “Fixed” radio button and then click “Next“.
  •  Enter the IP address for the new VMA and then click “Next“.
  •  Verify all of the settings, click the “Power on after deployment” box, and then click “Finish“.
  •  You will see the OVF start to deploy.

Everything look good? Proceed to Part 2