Category Archives: VMware Vcenter Server

Part 1 — vCenter 5.1 U1 — Creating and installing SSL certs for SSO.

There is a lot of information out there for installing vCenter 5.1, but the information is lacking for getting SSL certs working properly.  I first want to thank Derek Seaman over at www.derekseaman.com for his posts regarding what to do.  I have tried to shorten this a little bit for my own recollection.  Here is what I did to get SSL certs working.

Preparation

  1. Make sure you have installed the SSO Server, Inventory Service Server, and vCenter Server.  I used three separate machines for my environment, but you can use just one if you wanted to.
  2. Download and install the Visual C++ 2008 Redistributables (x64) and Win64 OpenSSL v0.9y from http://slproweb.com/products/Win32OpenSSL.html on your SSO server.
    Create a c:\certs folder on the SSO server containing the following subfolders:
    certsfolders
  3. Download the SSL Certificate Automation Tool from https://my.vmware.com/group/vmware/get-download?downloadGroup=SSL-TOOL-10.  Unzip this to c:\vmwarecerttool folder.
  4. Following Derek Seaman’s blog post http://www.derekseaman.com/2012/09/create-vmware-windows-ca-certificate.html , create a VMware-SSL template on your CA server.
  5. You will need the following information during the install:

SSO Administrator

Username: admin@System-Domain

Password:

 vCenter Administrator

Username:

Password:

 Original Database Password

 

Creating Certificates

 1.  Edit the ssl-environment.bat file located in c:\vmwarecerttool and fill in the appropriate information:

set sso_cert_chain=c:\certs\sso\chain.pem
set sso_private_key=c:\certs\sso\rui.key
set sso_node_type=single
set sso_admin_is_behind_lb=no

set is_cert_chain=c:\certs\inventory\chain.pem
set is_private_key_new=c:\certs\inventory\rui.key

set vc_cert_chain=c:\certs\vCenter\chain.pem
set vc_private_key=c:\certs\vCenter\rui.key

set ngc_cert_chain=c:\certs\WebClient\chain.pem
set ngc_private_key=c:\certs\WebClient\rui.key

set logbrowser_cert_chain=c:\certs\LogBrowser\chain.pem
set logbrowser_private_key=c:\certs\LogBrowser\rui.key

set vco_cert_chain=c:\certs\Orchestrator\chain.pem
set vco_private_key=c:\certs\Orchestrator\rui.key

set vum_cert_chain=c:\certs\UpdateManager\chain.pem
set vum_private_key=c:\certs\UpdateManager\rui.key

set sso_admin_user=admin@system-domain
set vc_username=corp\vminstaller

2.  Next, create the following configuration files in their respective folders.  Make sure that you name the files correctly.  Do not include the .cfg filename in the .cfg file.  I have done this…J  Do not change the organizationalUnitName!  I have created an example of the Inventory.cfg.  I got these from http://www.derekseaman.com/2012/09/vmware-vcenter-51-installation-part-2.html .

EXAMPLE: Inventory.cfg
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:VCINV1, DNS:VCINV1.DOMAIN.LOC

[ req_distinguished_name ]
countryName = US
stateOrProvinceName = Missouri
localityName = Saint Louis
0.organizationName = IT
organizationalUnitName = vCenterInventoryService
commonName = VCINV1.DOMAIN.LOC


Inventory.cfg
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:(Your Inventory Server), DNS:(FQDN of your Inventory Server)

[ req_distinguished_name ]
countryName = (Country Code)
stateOrProvinceName = (State)
localityName = (City)
0.organizationName = (Organization)
organizationalUnitName = vCenterInventoryService
commonName = (FQDN of your Inventory Server)

SSO.cfg
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:(Your SSO Server), DNS:(FQDN of your SSO Server)

[ req_distinguished_name ]
countryName = (Country Code)
stateOrProvinceName = (State)
localityName = (City)
0.organizationName = (Organization)
organizationalUnitName = vCenterSSO
commonName = (FQDN of your SSO Server)

vCenter.cfg

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:(Your vCenter Server), DNS:(FQDN of your vCenter Server)

[ req_distinguished_name ]
countryName = (Country Code)
stateOrProvinceName = (State)
localityName = (City)
0.organizationName = (Organization)
organizationalUnitName =vCenterServer
commonName = (FQDN of your vCenter Server)

WebClient.cfg

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:(Your vCenter Server), DNS:(FQDN of your vCenter Server)

[ req_distinguished_name ]
countryName = (Country Code)
stateOrProvinceName = (State)
localityName = (City)
0.organizationName = (Organization)
organizationalUnitName =vCenterWebClient
commonName = (FQDN of your vCenter Server)

VUM.cfg

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:(Your vCenter Server), DNS:(FQDN of your vCenter Server)

[ req_distinguished_name ]
countryName = (Country Code)
stateOrProvinceName = (State)
localityName = (City)
0.organizationName = (Organization)
organizationalUnitName =VMwareUpdateManager
commonName = (FQDN of your vCenter Server)

LogBrowser.cfg

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:(Your vCenter Server), DNS:(FQDN of your vCenter Server)

[ req_distinguished_name ]
countryName = (Country Code)
stateOrProvinceName = (State)
localityName = (City)
0.organizationName = (Organization)
organizationalUnitName =vCenterLogBrowser
commonName = (FQDN of your vCenter Server)

Orchestrator.cfg

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:(Your vCenter Server), DNS:(FQDN of your vCenter Server)

[ req_distinguished_name ]
countryName = (Country Code)
stateOrProvinceName = (State)
localityName = (City)
0.organizationName = (Organization)
organizationalUnitName =VMwareOrchestrator
commonName = (FQDN of your vCenter Server)

3.  You should now have one configuration file in each of the certificate folders you created earlier.  Next, we need to pull down the root certificate.  I am using Microsoft CA, so that is the only example I can give.  Open a browser and go to https://yourcaserver/certsrv/.  Make sure you fill in your CA server.  Click on Download a CA certificate, certificate chain or CRL. Change the encoding method to Base 64 and click Download CA certificate chain. Change the file name to cachain.p7b.

4.  Double click on the downloaded certificate, then locate the certificate in the console. If you have more than one certificate in the console, skip to step 3 below. If you have just one certificate, right click on the certificate and select All Tasks -> Export. Select Base-64 encoded and save the certificate with a filename of Root64.cer in the root of the Certs directory.

Notice that I have a root CA and a Subordinate CA
CACHAIN

5.  If you have a root and intermediate CAs (two or more certs in the console), you have some extra work. Export each certificate from the console as Base-64 and save into different files (e.g. Root64-1.cer and Root64-2.cer). You MUST save your Root CA as Root64-1.cer and the intermediary CA as Root64-2.cer.
RootCert

6.  We also need a concatenated file of the CAs (Root64.cer), in reverse order. Reverse order means the root is at the bottom of the file, and the subordinate CA is at the top.  From an administrative command prompt in the c:\certs folder run:
copy Root64-2.cer+Root64-1.cer Root64.cer
Combineroot

7.  Create a batch file in c:\certs called create_csr.bat.  Paste the following into this file:

Set OpenSSL_BIN=c:\OpenSSL\bin\openssl.exe

Set Cert_Path=C:\Certs

CD /d %Cert_Path%\vcenter\

%OpenSSL_BIN% genrsa 2048 > rui.key

%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config vcenter.cfg

CD /d %Cert_Path%\Inventory\

%OpenSSL_BIN% genrsa 2048 > rui.key

%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config inventory.cfg

CD /d %Cert_Path%\SSO\

%OpenSSL_BIN% genrsa 2048 > rui.key

%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config SSO.cfg

CD /d %Cert_Path%\UpdateManager\

%OpenSSL_BIN% genrsa 2048 > rui.key

%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config VUM.cfg

CD /d %Cert_Path%\webclient\

%OpenSSL_BIN% genrsa 2048 > rui.key

%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config webclient.cfg

CD /d %Cert_Path%\LogBrowser\

%OpenSSL_BIN% genrsa 2048 > rui.key

%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config LogBrowser.cfg

CD /d %Cert_Path%\Orchestrator\

%OpenSSL_BIN% genrsa 2048 > rui.key

%OpenSSL_BIN% req -out rui.csr -key rui.key -new -config Orchestrator.cfg

8.  Open an administrative command prompt and then browse to c:\certs.  Run the create_csr.bat.  This will create two files in each of the cert folders.  If the files don’t create, check the configuration files that you created earlier.

Example: Inventory Folder
inventoryexample

9.  Now it is time to mint the certificates using these two files.  Under c:\certs create another batch file.  This time call the batch file vCenter5.1_CertRequest.bat.  Paste in the following and make sure that you change the Certificate Authority Name.  This batch file can be found here http://www.derekseaman.com/2012/09/vmware-vcenter-51-installation-part-2.html .

:: Script to request vCenter 5.1 SSL certificates from a Microsoft CA
:: Modify these variables for your paths and CA information
:: Place your root64-1.cer and root64-1.cer (if using an intermediate CA)
:: in the Cert_Path directory. OpenSSL config files must already exist.
:: Also creates the chain.pem files for the VMware Certificate automation tool
::
:: Written by Derek Seaman, derekseaman.com
::

:: Certificate Authority Template name
Set Cert_Template=VMware-SSL

:: Certificate Authority Name
Set CA_Name=D001DC01\Contoso-D001DC01-CA
:: Path to OpenSSL
set OPENSSL_CONF=c:\OpenSSL\bin\openssl.cfg
Set OpenSSL_BIN=c:\OpenSSL\bin\openssl.exe

:: Path to your vcenter services directory with the config files
Set Cert_Path=C:\certs

:: Do not change anything below here
Set Root_CA_Cert=%Cert_Path%\Root64-1.cer
Set Sub_CA_Cert=%Cert_Path%\Root64-2.cer
Set CA_Chain=%Cert_Path%\Root.cer

if exist %Sub_CA_Cert% (
copy /B  %Sub_CA_Cert% + %Root_CA_Cert% %CA_Chain%
Set CA_Cert_Chain=%CA_Chain%
) Else (
Set CA_Cert_Chain=%Cert_Path%\root64.cer
)

CD /d %Cert_Path%\vcenter
%OpenSSL_BIN%  genrsa 2048 > rui.key
%OpenSSL_BIN%  req -out rui.csr -key rui.key -new -config vcenter.cfg
certreq -submit -q -f -config “%CA_NAME%” -attrib “CertificateTemplate:%Cert_Template%” rui.csr rui.crt
%OpenSSL_BIN%  pkcs12 -export -in rui.crt -inkey rui.key -certfile %CA_Cert_Chain% -name rui -passout pass:testpassword -out rui.pfx

copy /B rui.crt + %CA_Cert_Chain% chain.pem
CD /d %Cert_Path%\Inventory
%OpenSSL_BIN%  genrsa 2048 > rui.key
%OpenSSL_BIN%  req -out rui.csr -key rui.key -new -config inventory.cfg
certreq -submit -f -q -config “%CA_NAME%” -attrib “CertificateTemplate:%Cert_Template%” rui.csr rui.crt
%OpenSSL_BIN%  pkcs12 -export -in rui.crt -inkey rui.key -certfile %CA_Cert_Chain% -name rui -passout pass:testpassword -out rui.pfx

copy /B rui.crt + %CA_Cert_Chain% chain.pem
CD /d %Cert_Path%\SSO
%OpenSSL_BIN%  genrsa 2048 > rui.key
%OpenSSL_BIN%  req -out rui.csr -key rui.key -new -config SSO.cfg
certreq -submit -f -q -config “%CA_NAME%” -attrib “CertificateTemplate:%Cert_Template%” rui.csr rui.crt
%OpenSSL_BIN%  pkcs12 -export -in rui.crt -inkey rui.key -certfile %CA_Cert_Chain% -name rui -passout pass:testpassword -out rui.pfx
copy /B rui.crt + %CA_Cert_Chain% chain.pem

CD /d %Cert_Path%\UpdateManager
%OpenSSL_BIN%  genrsa 2048 > rui.key
%OpenSSL_BIN%  req -out rui.csr -key rui.key -new -config VUM.cfg
certreq -submit -f -q -config “%CA_NAME%” -attrib “CertificateTemplate:%Cert_Template%” rui.csr rui.crt
%OpenSSL_BIN%  pkcs12 -export -in rui.crt -inkey rui.key -certfile %CA_Cert_Chain% -name rui -passout pass:testpassword -out rui.pfx

copy /B rui.crt + %CA_Cert_Chain% chain.pem
CD /d %Cert_Path%\webclient
%OpenSSL_BIN%  genrsa 2048 > rui.key
%OpenSSL_BIN%  req -out rui.csr -key rui.key -new -config webclient.cfg
certreq -submit -f -q -config “%CA_NAME%” -attrib “CertificateTemplate:%Cert_Template%” rui.csr rui.crt
%OpenSSL_BIN%  pkcs12 -export -in rui.crt -inkey rui.key -certfile %CA_Cert_Chain% -name rui -passout pass:testpassword -out rui.pfx

copy /B rui.crt + %CA_Cert_Chain% chain.pem
CD /d %Cert_Path%\LogBrowser
%OpenSSL_BIN%  genrsa 2048 > rui.key
%OpenSSL_BIN%  req -out rui.csr -key rui.key -new -config LogBrowser.cfg
certreq -submit -f -q -config “%CA_NAME%” -attrib “CertificateTemplate:%Cert_Template%” rui.csr rui.crt
%OpenSSL_BIN%  pkcs12 -export -in rui.crt -inkey rui.key -certfile %CA_Cert_Chain% -name rui -passout pass:testpassword -out rui.pfx
copy /B rui.crt + %CA_Cert_Chain% chain.pem

CD /d %Cert_Path%\Orchestrator
%OpenSSL_BIN%  genrsa 2048 > rui.key
%OpenSSL_BIN%  req -out rui.csr -key rui.key -new -config Orchestrator.cfg
certreq -submit -f -q -config “%CA_NAME%” -attrib “CertificateTemplate:%Cert_Template%” rui.csr rui.crt
%OpenSSL_BIN%  pkcs12 -export -in rui.crt -inkey rui.key -certfile %CA_Cert_Chain% -name rui -passout pass:testpassword -out rui.pfx
copy /B rui.crt + %CA_Cert_Chain% chain.pem

You should see a bunch of messages like this:
successfulmessage

All of the certs folders should now contain some new files:
certsnewfiles

10.  Copy the c:\certs and c:\vmwarecerttool folder from the SSO server to both the Inventory and vCenter Server.  Part 2 will fail if you forget to do this.

Continue on with part 2.  https://favoritevmguy.wordpress.com/2013/06/17/part-2-vcenter-5-1-u1-creating-and-installing-ssl-certs-for-sso

Can’t power off/on a VM — VM is stuck

We had an issue the other day with our filers and because of it, some our machines orphaned themselves and VMware didn’t know what to do with them.  If I tried to power off the machine I would get the error,

Image

Basically, I don’t even think the machine was on, but my vCenter server showed it as being powered on.  Luckily I found a KB article that deals with this.    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1014165.

SSH into the host that the VM is said to be running from.  Then run the command:

esxcli vm process list

This gives you a list of all the running VMs, and the important piece you want is the World ID.  Copy that, and then run the command:

esxcli vm process kill –type=[soft,hard,force] –world-id=WorldNumber

Note: Three power-off methods are available. Soft is the most graceful, hard performs an immediate shutdown, and force should be used as a last resort.

As you can see, I ran this command and it killed the VM.  After doing this, I was able to power on the VM without issue.  I used soft and it worked.

Image

That’s all I have for today.

Storage Vmotion Problem

We have two vCenter servers that are in linked mode.  After upgrading to Vsphere 5 I have been having problems deploying machines.  I get this error:

Image

Strange right…  Well, I then decided to do a vStorage migration of a machine and got the same error:

Image

Turns out that this is a bug with no fix right now when you link two vcenter servers together.  To get around this, make sure that your VIC client is logging into the vCenter server that you are going to do the vStorage migration or deplate deployment from.  If I have vCenter Server A and B and I want to deploy from a template on B; you would log into the B vCenter to deploy.

Here is the KB article http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2013516

Host Disconnected from Vcenter

Had a very strange issue last night with one of my ESXi 4.1 Update 2 hosts.  In Vcenter, suddenly one of my hosts showed disconnected.  I could ping the host and remote into all virtual machines on the disconnected host, but for some reason Vcenter couldn’t see it.  Here were the troubleshooting steps I took yesterday.

1.  I used putty to SSH into the disconnected host.  I then restarted the service using the command “services.sh restart “.
2.  I still could not connect the host in Vcenter.  I then tried removing the host and re-adding to Vcenter.  It threw an error and would not add.
3.  I remoted into my Vcenter VM and tried pinging the disconnected host.  The Vcenter server was unable to ping either by name or IP.  What was strange is that I could ping another server at that location.  If I remoted into any other machine, it could ping the disconnected host…it was only the Vcenter server that couldn’t see it.  This was not dns or IP.  I restarted the Vcenter server to no avail.
4.  Luckily this disconnected host was on site and I was able to connect a monitor and keyboard to it.  Normally I would remote in using the ILO port, but that is not configured right now (another story altogether).  I restarted the management agents from here.  I was now able to ping from the host and from the Vcenter server.  YAY!

I don’t know why restarting the management agents from the console was any different that when I SSHed into the machine, but it was.

Manually Assigning a MAC in VMware

During our migration from ESXi 4.1 Update 2 to ESXi 5 Update 1 we decided to change all nics on VMs to VMXNET3.  Everything was fine until my users came in the next morning and couldn’t do some of the things that they needed to do.  It turns out that there were some programs tied to the old MAC address of the nic that I had removed from the virtual machine.  We had to get licenses tied to the new MAC addresses, and then the decision was made for the remaining VMs that I still had to work on, to keep the current MAC address.  How to do this…

1.  Install the latest VMware tools. (If you haven’t done this, you might not see VMXNET3 as an option for network card.
2. RDP into the virtual machine and take not of the IP information on all nics.  Then, change the IP to “Obtain an IP address automatically” and “Obtain DNS server address automatically”.  (This keeps you from getting weird messages later about IP addresses already being assigned to a nic.Image
3.  Shut down the VM.
4.  Edit Settings on the VM using the vSphere Client.
5.  Click on each network adapter and write down the current MAC address.  Then click on remove to remove the nic from the machine.
Image
6.  Click “Add” and then select “Ethernet Adapter”.
7.  Change the adapter type to VMXNET3 and then choose your network for the VM.
8.  Change the MAC to “Manual” and then enter the MAC that you wrote down before from the old nic.
Image
9.  Click “OK” and the old nic will be removed and the new nic added.  If you go back in and look at the nic you will notice that it has the MAC that you manually added.
10.  After powering on your VM, open the command prompt in administrative mode and type without quotes, “set devmgr_show_nonpresent_devices=1”  Then type “devmgmt.msc”
11.  When device manager opens click “View –> Show Hidden Devices”.  Hidden devices, especially old nic need to be removed.  You will see unused devices showing greyed out.  Right click on them and choose “uninstall”.

Hopefully this saves someone from a licensing headache when upgrading your VMs.