Contributor
Contributor

Problems with MAC addresses after upgrading from 5.0 to 5.1

Hi all,

I've just experienced a very anoying problem after the upgrade from vSphere 5.0 to 5.1 succeeded: Some of my VMs cannot be stared due to the fact that their MAC address are NOT in the allowed range anymore. So one of the VMs is running an HP P4000 Lefthand VSA an was assigned the MAC address

00:0C:29:84:12:E4 by vSphere 5.0 and after the upgrade to 5.1 the MAC address is "marked as invalid" and the VM cannot be started anymore. Since the license key for the HP P4000 Lefthand VSA is issued according to the MAC address I cannot "change" it because this would lead to a non functioning product. Also, re-licensing the HP P4000 Lefthand VSA is not an option.

Can anyone help?

Thanks,

Holger

10 Replies
Enthusiast
Enthusiast

Hi  TempelH

Welcome to communites .

Please discussed with your team if the same lan card MAC address changed by third party tool in past .

Its happen in that condition only .

"Nature always wears the colors of the spirit."
0 Kudos
Contributor
Contributor

Hi,

I'm having the same problem with the HP P4000 VSA too.

In my case I installed ESXi 4.0 on two HP DL370 G6 Hosts in 2009 and installed a P4000 VSA (which was Version 8.1 by then) using the enclosed Documentation. As far as I remember the VSA's VMX and VMDK files had to be uploaded onto each hosts datastore and one created the virtual machine by importing it to the host.

After that you had to edit the VM's properties to add a disk as RDM for storing the data and if I remember well the next step was to change the MAC address from dynamic to static - and after clicking "OK" this left you with a VM containing a MAC address beginning with 00:0C:29:xx:xx:xx.

Unfortunately I - as well as maybe you - created the VSA cluster on top of that and licensed the VSAs in this way too.

In the meanwhile I have upgraded the HP VSAs to 9.5 and the hosts were reinstalled with ESXi 5.0, and recently I started upgrading the hosts to 5.1.

After trying to power up the VSA on the first upgraded host, I received the same message as you:

"Module DevicePowerOnFailed.

Could not set up "macAddress" for ethernet0.

Invalid MAC address specified.

00:0c:29:5f:a0:0c is not an allowed static Ethernet address. It conflicts with VMware reserved MACs."

The root cause:

I started googling and found some articles which said to add ethernet0.checkMACAddress = “FALSE” in the VMX file, but this did not work too.

I also came to http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=219

which says that generated MAC addresses should be generally in the 00:0c:29 range, and static MAC addresses should be in the 00:50:56 range.

Which means, that our configuration - static MAC in 00:0c:29 range is not supported and should not even work (but it did - there seemed to be no verification).

Now, when reading through the ESXi 5.1 release notes (https://www.vmware.com/support/vsphere5/doc/vsphere-esx-vcenter-server-51-release-notes.html) I came over the following point:
"Prefix- and range-based MAC address allocation is supported only in vCenter Server 5.1 and ESXi 5.1

...The prefix- and range-based MAC address allocation schemes are not  supported on pre-5.1 hosts because pre-5.1 hosts explicitly validate if  an assigned MAC address uses the VMware OUI 00:50:56 prefix. If the MAC  address is not prefixed with 00:50:56, the virtual machine pre-5.1 host  fails to power on."

If I read that correctly this means that up until now you could only have VMs with dynamic MACs beginning with 00:0c:29 and ones with static MACs beginning with 00:50:56. With ESXi 5.1 you seem to be able to create VMs with any MAC prefix, which is surely a bonus point when doing P2V-migrations.

Unfortunately, with this new feature, they seem to have added another check behind the scenes: throw an error when static MAC address is in the 00:0c:29 range, so they can avoid duplicate MAC addresses when doing dynamic allocation.

So, ESXi 5.1 effectively forces you to change your static MAC address, if it begins with 00:0c:29.

The problem:

Now to the trouble this creates to the HP P4000 VSA cluster (and to explain it to all others who say "just relicense your software with the new MAC address" - which really is not the problem):

I changed the MAC address of the VSA on my upgraded host and booted it up, as exprected I received a message that the trial mode was activated because the licence key did not match the "feature key" (=mac address).

OK, no problem, just go to the HP page and relicense it.

BUT:

the VSA did not join the cluster. There were messages like:

"There is a system with serial number <NewMACAddress> which believes to be in management group <ABC> but the management group does not agree. Please call HP support"

and the VSA was shown in the management group outside of the cluster with a question mark.

In the cluster the old VSA was shown with a red X mark, and if you clicked on it, it said:

"Could not find system with serial number <OldMACAddress>"

So, not only the licensing was broken, but the whole storage cluster wreaked havoc when MAC addresses were changed.

The workaround:

Since I did not want to spend a whole week explaining and sending HP support log files and trying this & that and updating all possible surrounding wholly unrelated and irrelevant firmare and software just to get finally somebody from 3rd level support who can really fix your problem I tried to do it my way and came up with the following solution:

Important: this process will delete all existing data on your VSA. All Raid0 volumes will be destroyed.

- Change the MAC address to something ESXi would accept.

- Reconfigure the VSA's VM network properties to start without network connection.

- Start the VM and wait until the VSA has fully booted.

- Login to your VSA on the console using "start" and then enter your admin username and password.

- Navigate to the menu option "Config Management"

- Select "Remove from Management Group"

- Confirm all prompts saying "this will destroy our data" and "are you really sure" with "Yes", since this will only delete the data on this VSA.

- When the process is done, shut down the VSA, reconfigure the network settings for auto connect on boot and restart the VSA.

Now, if you've done all these steps, the VSA will be discovered by the Management console as a fresh storage node which you can add then to the Management group. Still, in the VSA cluster there is the old VSA node with an ugly X. You can get rid of it in the following way:

After having added the VSA to the management group select "Edit Cluster" and then "Exchange storage systems" from the cluster context menu, which lets you exchange your "ghost" VSA object in the cluster with your fresh VSA.

This saves you a time-consuming restripe of all volumes comparing to the method of adding the new VSA and deleting the old one.

After that, your volumes start resynching, and when has finished you can deletete the old VSA object from the management group and it wil not reappear.

If you had Raid0 Volumes which you wanted to save, you would need to downgrade to ESXi 5.0, get the VSA running, raise the data protection level (if you have the space) and then do the whole process.

Surely, the whole process is time-consuming, it took ~48 hours to resync the volumes of an 8TB two-node cluster which is about 60% full. I just imagine what one would have to do with a four or more node cluster.

The other way would be to go the HP support way until you get someone on the phone who will SSH into your VSA and replace the MAC address in all config files and in the dbd_manager shared database.

A solution (=a wish):

It would be really nice of VMware, if they provided us with some override option or warning dialog, if we select a MAC address from the dynamic MAC pool, so we can select the "yes, we know what we do and we are aware that we might create MAC address conflicts if we create VMs with dynamic addresses (we were able to create such situations up until now with the other ESX versions), and yes, we want to shoot ourselves into the knee but we really want to do so"-option.

I understand that this is to prevent user errors and to become more user friendly, but in this case we really need an "admin override"-button.

HTH,

Marc

P.S.: I am now preparing the upgrade of the remaining ESX host, so after this weekend I will not be affected by this problem any more. I am posting this hoping to help others who encounter the same problem, and to send some feedback concerning the new mac address feature, hoping there will be some patch or information about some override option. Also, I hope such major changes will be planned less obscure and with more foresightfulness, so that we may never get into such situations any more. (And yes, I do think that new MAC address restrictions - even if they as a side effect - are such a major change.)

Contributor
Contributor

Is that vSphere is connected to vCenter, somtimes vCenter will not allowing poweron if the mac address is in the range "00:0C:29"

0 Kudos
Contributor
Contributor

Hey,

Glad I found I'm not the only one stuck with this problem with Hp LeftHand VSA. Anybody found a way to override this new restriction on MAC prefixes so I can boot my VSA?

0 Kudos
Contributor
Contributor

It took a little messing around, but here's what I did to get my system going. In my case i had 2 interfaces, but only 1 MAC needed to match.

ESXi 5.1.0 build 799733

Good luck Smiley Happy
Dan

  • edit the vmx file

/vmfs/volumes/50bf4d82-31b73571-5543-001e4f378eac/MAS # vi MAS.vmx 

  • ensure you are using generated MACs:

ethernet0.addressType = "generated"

ethernet1.addressType = "generated"

  • edit these 3 lines to reflect the MAC you want. this  assumes you want to use one of the "VMWARE automagic (00:0c:29)" ones, notice the last 6 chars of the first two lines match the last 3 octets of your MAC

uuid.location = "56 4d 74 53 f4 52 bf 03-02 fb 39 13 6b 2b 6c fc"

uuid.bios = "56 4d 74 53 f4 52 bf 03-02 fb 39 13 6b 2b 6c fc"

ethernet0.generatedAddress = "00:0c:29:2b:6c:fc"

  • if you want ethernet1 to match something specific instead, you need to subtract 10 (0x0A) from the last octet of the ethernet0 MAC because of this line:

ethernet1.generatedAddressOffset = "10"

This will create ethernet1's MAC with a value of 10 more than ethernet0. I didn't play around with different values here, but presumably you could calculate & edit this number to get both MACs to match your needs.

  • remove the VM from inventory, and re-import it (by browsing the datastore to the vmx file)
  • When starting the VM, answer "I moved it" to the dialog box asking about what happened to your machine

edit: went back to the vmx file and noticed the uuid.location value was changed back to the original value for some reason... the other 2 (and of course most importantly, the actual MAC!) stuck just fine

0 Kudos
Contributor
Contributor

A solution (=a wish):

It  would be really nice of VMware, if they provided us with some override  option or warning dialog, if we select a MAC address from the dynamic  MAC pool, so we can select the "yes, we know what we do and we are aware  that we might create MAC address conflicts if we create VMs with  dynamic addresses (we were able to create such situations up until now  with the other ESX versions), and yes, we want to shoot ourselves into  the knee but we really want to do so"-option.

I  understand that this is to prevent user errors and to become more user  friendly, but in this case we really need an "admin override"-button.

Hi All

Is anybody from product managers / engineers read this post?

Do you plan to solve problem witch custom MAC?

Now this is blocking point to upgrade by my company - we are using VMWare in lab as testing environment. We have test licenses per MAC and there is so many problems to regenerate it - until there is no possibility to manually change MAC to any prefix we can't upgrade.

0 Kudos
Immortal
Immortal

Some do read the forums, some dont.

The BEST way to get something like this seen is by opening an SR.

--Matt VCDX #52 blog.cowger.us
0 Kudos
Contributor
Contributor

Hey Guys,

I've open a ticket with VMware...basically got the "Your out of luck". Ask HP to help you, we can't help you.

Still waiting for an HP solution.

0 Kudos
Contributor
Contributor

Found this article tonight that saved my behind, http://www.elasticsky.de/en/2012/11/vsphere-5-1-static-mac/#comment-411

Basically the same as someone posted above. I booted the machine, and choose "I moved it" and received the invalid mac error. So I made a backup of the VMX file, changed the mac to automatic, booted the machine with the new automatic mac and upgraded the tools and hardware etc. Then I shut down the VM, edited the VMX file (made backup again) changed the mac type to "generated" and put in my org mac address at the bottom of the file for ethernet0.address, and then booted the machine, leaving the network card type on automatic in the edit vm settings (while the machine was off)

I then booted the VM and it's working fine. The network card setting, when you edit the running vm, shows my old mac address, and is set to automatic. I did not have to modify the uuid or any other settings. Hope this helps.

And for those who say "just get a new license key" That may not always be possible with out major expense. For example in my case, this is old software no longer under maint. It's scheduled to be upgraded/migrated next year, the money is just not in the budget this year to do so.

0 Kudos
Contributor
Contributor

Hello again,

Finally got my long awaited answer from HP

This issue is a VMware issue and is outside of our control, so our recommendation is to change the MAC address on the StoreVirtual VSA VM
and obtain a new licesne from Webware.  The VSA should still be found by the CMC and be part of it's current Management Group, but it will have
an invalid license, so you will have 30 days to get it licensed correctly (or your data will go offline).  This scenario is basically the same as a system board replacement on a physical server, where a new license is required when MAC address #1 changes.

pretty much what I expected.....

0 Kudos