Tag Archives: NICS

Proliant DL380 embedded NICs missing after firmware update

I ran into a very strange issue today when I went to redeploy an old Proliant DL380 G5. The first thing I did was use the most current service pack DVD to update the firmware. The most current is from 2/2014 and has the number 2013.02.0. After installing ESXi 5.1 U1 I noticed that I was only showing 4 NICs and not the 6 I started with.

The two embedded NICs were missing!!

After a quick google search or twelve I stumbled upon an HP discussion with exactly the same problem that I was having. I followed the instructions from the HP discussion and here is what it took to fix (Most of this is copied from user hase3d’s post).

1. Download all necessary tools
     – download FreeDOS
     – download XDIAG.exe 
     – download bc08c740.bin 
     – read all information in setup.txt

2. Prepare the FreeDOS.iso
     -After downloading open the iso with a tool like UltraISO. I used Magic ISO.
-Add the XDIAG.exe and the bc08c740.bin to the iso – I these files to the          root so that I wouldn’t need to add a path later.
-Save the iso with a new name.
-Burn it or mount it with ilo.

3. Boot from FreeDOS
     -Select Install to harddisk
     -Press 1
     -Select your language and press Enter
    -Press ESC
    -Select run FreeDOS from CD-ROM

4. Mine booted to f:\freedos. Do a cd\ to get back to the root of f:
5. Run xdiag in engineering mode by typing xdiag -b06eng
6. type device 1
7. nvm fill 0 0x600 0
8. nvm upgrade -bc bc08c740.bin
9. nvm cfg
     -Press q
     -Type default
     -Press q again
     -Type 16=10 wich sets the BAR size to 32
     -Press q for the third time
     -Type save and then exit out to the main menu

10. Type device 2  and repeat steps 7-9, run the command 1=00:00:18:xx:xx:xx <— change the last digit for different mac on device 2.

I did not do anything else from the setup.txt file.

I powered down the host and then when I rebooted I had 6 NICs again!

vSphere HA detected that host is in a different network partition than the master

Target: Host
Previous Status: Green
New Status: Red
Alarm Definition:
([Event alarm expression: vSphere HA agent on a host has an error; Status = Red] OR [Event alarm expression: vSphere HA detected a network isolated host; Status = Red] OR [Event alarm expression: vSphere HA detected a network-partitioned host; Status = Red] OR [Event alarm expression: vSphere HA detected a host failure; Status = Red] OR [Event alarm expression: Host has no port groups enabled for vSphere HA; Status = Red] OR [Event alarm expression: vSphere HA agent is healthy; Status = Green])
Event details:
vSphere HA detected that host (host) is in a different network partition than the master (Cluster) in Datacenter

I had been getting this message randomly over the last couple months on some of my datacenter hosts. These alerts didn’t seem to be causing any problems within the cluster, but I wanted to get to the bottom of this. I opened a ticket with VMware and uploaded the logs from both the host and vCenter, but they didn’t see anything out of the ordinary. On the second webex with VMware I noticed a couple strange things with the management network that might be the cause.

  1. The first thing I noticed was that the NICs were set for “Auto Negotiate”. I originally set up our environment on ESXi 4 before upgrading to ESXi 5.1. When I initially set this up I hard coded (KB1004089) these to 1000GB/Full. I am wondering if at some point during the upgraded that they defaulted back. On our switches it was set at 1000GB/Full so it is important that we set this on the host NICs to 1000GB/Full as well.
  2. The second thing that I noticed that in the Management network that I had the Load Balancing set to “Route based on IP hash”. The problem here is that for this to work correctly you need a port channel configured (I do not have this configured this way). This might be the cause of the HA problem if the traffic is going across these NICs is getting confused because of the Load Balancing configuration. I changed this to “Route based on the originating virtual port ID”, which makes the traffic go out on the port that it came in on. There is a good read found here…http://blogs.vmware.com/kb/2013/03/troubleshooting-network-teaming-problems-with-ip-hash.html.

This case is still ongoing with VMware and I should know in the next couple weeks if this solves my problem; my gut tells me it will.