StephenMoll
Expert
Expert

Clash of the MACs!

We deploy and wipe ESXi servers on an almost weekly basis, as they are used for short term projects and the machine is wiped and reinstalled in readiness for the next project. Although we have vCenter Server, this is used only for a small set of more permanently installed virtualised systems.

Over the last couple of weeks we have been trying to define a process for the installation and creation of a standard ESXi 4.1 host setup on our HP DL380G6 servers.

Now, because the HP DL380G6 runs like a dog without some specific bundles installed on top of the ESXi 4.1 base installation, the process requires running vihostupdate from a Windows machine with the CLI installed. To remove this step we were trying different methods to image a final installation and just write the image to the server to refresh it. The final technique settled on installing ESXi 4.1 to a USB memory stick, which would be plugged into the internal port inside the DL380s. We found a nice bit of software that would reliably duplicate USB memory sticks, that ensures the copies are bootable.

All was well and good, until someone decided they needed TWO ESXi boxes for a particular project.

Anyone spot the problem yet?

The engineer found that when the two servers were on the network, despite having different IP addresses they simply would stay on the network. They would randomly drop of the network and come back, repeatedly. BUT... at no time were both visible on the network at the same time. We had a client machine with two CMD windows running PING continuously, and the responses would only be received by one or the other, and it would seemingly switch over at random intervals as to which was receiving responses. Needless to say trying to connect vSphere Client to one of the servers was a strange experience at best, and nothing constructive could be achieved even if a connection was made.

We resorted to using Wireshark to try and work out what was happening with the ICMP ping packets. We discovered that we could see the ping packets going to the two different IP addresses, but deeper inspection showed that the destinations for both those ping packets had the same MAC address, despite having different IP addresses. So basically our poor little Netgear switch was getting somewhat confused.

After getting confused about why we had a duplicate MAC address, when the NICs clearly had different ones, it suddenly dawned on me what had happened.

When we created the master USB memory stick by installing ESXi to it, it was duplicated in the minutest detail. Right down to the MAC address that had presumably been randomly generated for the VMKernel Management Port.

The solution wasn't all that complicated, create a new VMKernel port on one of the servers, and delete the original one. This is not so onerous doing with the vSphere client, but I would like to reduce the lists of 'things' that the project engineers need to do when they are given an ESXi host to use.

Is there a way to trigger a VMKernel port to have a newly assigned MAC address, AFTER it has been created? Preferably through some sort of command or script that could be run from the Tech Support Mode console?

0 Kudos
7 Replies
AntonVZhbankov
Immortal
Immortal

When ESXi boots from USB first time it remembers MACs and then just use them forever, even if you install USB flash into another physical server.

1) Open ESXi / Configration / Software / Advanced

2) set Net.FollowHardwareMAC to 1

3) Reboot host


---

MCITP: SA, MCTS Hyper-V, VCP 3/4, VMware vExpert

http://blog.vadmin.ru

EMCCAe, MCITP: SA+VA, VCP 3/4/5, VMware vExpert http://blog.vadmin.ru
0 Kudos
PduPreez
VMware Employee
VMware Employee

Hi

I do allot off installs with just imiging the USB key and never had Mac Address issues.

There is a ESXi 4.1 ISO you can download from HPs site, which has the Bundle already installed.

The proccess to image the USB keys is on this site:

http://www.vladan.fr/how-to-install-esxi-40-on-usb-memory-key/

Only difrence is on ESXi 4.1 ISO, after extraction you extract imagedd (should end up with 900MB file) and not the image.tzg.temptar as in ESXi 4.0

Hope it helps

Regards

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct. Thank you.

0 Kudos
PduPreez
VMware Employee
VMware Employee

Now that I think of it, this might be the issue I have on a site whre VMotion takes an hour per VM!!

This was on IBM blades with the new virtual connect NICs.

I installed the driver bundles on one USB stick and Imaged them to the rest.

I gues what I should have done is,

- Install the Bundles on 1 USB stick

- 1) Open ESXi / Configration / Software / Advanced

- 2) set Net.FollowHardwareMAC to 1

- 3) Shut Down host

- And then make an image and flash all the other 26 USB sticks 😮

mm, If my theory is correct after reading your input, I have some work to be done Smiley Wink

Regards

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct. Thank you.

0 Kudos
StephenMoll
Expert
Expert

The installation method you describe looks cumbersome in comparison to simply sticking the USB stick in the server, booting from the ESXi 4.1 installation CD, and running the install process normally. The system is then temporarily setup to network with a Windows XP machine which has the rCLI software installed, and the ESXi 4.1 bundles from the HP website are installed.

I then clear out the network settings, and then used Active Disk Image to create a raw disk image of the USB stick, which can then be written to all remaining memory sticks, thus duplicating them.

As far as I know, HP haven't released an ESXi installation CD with the bundles already in place. I can only find 4.0.

0 Kudos
PduPreez
VMware Employee
VMware Employee

The installation method you describe looks cumbersome in comparison to simply sticking the USB stick in the server, booting from the ESXi 4.1 installation CD, and running the install process normally. It might sound cumbersome, but in real time it would take you about 30min to prep the USB image. From there on it takes about 3min-5m to prep the next USB stick.

Imagine you need to install 30 x ESXi hosts. This way you can do all the installs at the office, go to site, slot in the USB sticks, boot up and configure the management network (less than 1min per server). This should take less than 1 hour then you can do everything remotely Smiley Wink I find this better than to sit in a cold server room for 4hours installing 30 x ESXi hosts from CD.

As far as I know, HP haven't released an ESXi installation CD with the bundles already in place. I can only find 4.0.

Yes there is, it is available from the HP software depot: https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPVM06

Click on "Receive for free"and log in

Then you'll see the ESXi 4.1 package

Regards

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct. Thank you.

0 Kudos
PduPreez
VMware Employee
VMware Employee

Hi

Did you find a workable solution for your scenario?

Regards

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct. Thank you.

0 Kudos
StephenMoll
Expert
Expert

I am trying a different tack, and trying to create a bootable USB stick with the ESXi installer on it.

Sadly when the ESXi installer is placed on a USB stick, it behaviour changes such that it MUST use a ks.cfg kickstart script file. The problem here is that the commands in the kickstart file are complex (at least to me) and I have not been successful in getting it to install in the way I want, i.e. simply install the hypervisor to the SD-cards we are now using (To get around another problem with HP Proliant servers), and to leave the VMFS volume either untouched or undefined. The problem is that the kickstart tries to create a VMFS volume on the SD-card, which of course doesn't support VMFS and is too small in any case!

0 Kudos