VM Freeze after successful VMotion

[Sven_Burger]

I have two identical ESX 3.01 hosts -

I have both the hosts added to a VirtualCenter 2003 Server.

Environment is not live - but will soon be. I have 4 VMs created on one host - after conducting a successful migration to the other host - the VM Freezes (I can tell because I have the console open).

From here I can do one of two things to get it back: Remove it from inventory and re-register it to the original host; or migrate it back to the original host; in which it unfreezes itself.

If I stop it and try and power on the VM while on the second host; it says 'Operating System not found' when trying to load Windows 2k3.

As I said the hardware resources on the host are the same - and what is really screwing with me is that the migration checker says I am good to go..

Any troubleshooting ideas?

[vmmeup]

More information would be helpfullhere.

How is your networking configure? (physical nics, physical switches, vswitches, vmkernel)

How are your datastores configured/ (i.e. shared storage, NFS, iSCSI, NFS)?

[rubensluque]

Is the datastore visible for both host's ? It seems that the data store disapear when the VM is vmtioned.

[sportjeeptj99]

Are you keeping the console open during the entire vmotion process? I always close an open console window if I am doing a vmotion.

[Sven_Burger]

In both boxes I have 6 network adapters:

vSwitch0

vmnic0 Service Console

vmnic1 Service Console

vSwitch1

vmnic2 ToLAN (VMNetwork)

vmnic3 ToLAN (VMNetwork)

vSwitch2

vmnic4 vmkernel (iSCSI) / Service Console 2 (this is needed for networking)

vmnic5 vmkernel (iSCSI) / Service Console 2 (this is needed for networking)

Both are attached via iSCSI to an EMC CX3-20 LUN (VMFS1) which is where the VMX and Virtual Disks are stored

[Sven_Burger]

That was my assumption as well - but vmkernel on the second host is talking to the VMFS LUN just fine via iSCSI.

The only thing I can think of here (which I will try right now) is that on the second host I don't have both NICs per vSwitch actually physically plugged in. Perhaps despite the fact that this box is seeing the SAN - VMotion may get confused and try to use one of the Virtual Adapters that is not actually connected? Sounds remote but it's the only difference I can find.

[Sven_Burger]

Yep - I keep it open to the 2003 console and the VM freezes. Happens pretty much right after the VM completes the VMotion process.

[moneill · Accepted solution]

I agree with sportjeep, it's best to keep the console closed, on earlier releases

an open console was documented as unsupported for vmotion.

If you want to keep a console open during the migration you can use an rdp connection with /console specified - this works.

Mike

[grasshopper]

anything in the vmkernel logs?

[Sven_Burger]

Having the Console open was just for a test to see what might be going on - if I do a migration - after it finishes the VM shows as powered off.

[Sven_Burger]

Remind me how to check those?

[grasshopper]

less /var/log/vmkernel

then press the 'page down' & 'page up' keys on your keyboard to scroll through. Do this on each offending host (source and target).

[grasshopper]

BTW, this implies that you have an ssh client such as putty.exe[/url] on your laptop/desktop.

On a side note, you can also watch the logs in real-time as you attempt the vmotion.

tail -f -s 2 /var/log/vmkernel

[Sven_Burger]

Mar 30 14:37:54 IS-TG-ESX02 vmkernel: iSCSI: Sense 70000600 00000006

Mar 30 14:37:54 IS-TG-ESX02 vmkernel: 3:03:20:11.664 cpu2:1038)SCSI: 8052: vmhba40:4:1:1 Retry (unit attn)

Mar 30 14:37:54 IS-TG-ESX02 vmkernel: 3:03:20:11.665 cpu2:1038)WARNING: SCSI: 1820: Manual switchover to vmhba40:4:1 completed successfully.

Mar 30 14:37:56 IS-TG-ESX02 vmkernel: 3:03:20:13.666 cpu2:1038)WARNING: SCSI: 1785: Manual switchover to path vmhba40:4:0 begins.

Mar 30 14:37:56 IS-TG-ESX02 vmkernel: 3:03:20:13.666 cpu2:1038)SCSI: 1789: Changing active path to vmhba40:4:0

Mar 30 14:37:56 IS-TG-ESX02 vmkernel: 3:03:20:13.691 cpu3:1065)iSCSI: session 0x7d73eb8 recv_cmd 0x3d009e28, cdb 0x0, status 0x2, response 0x0, senselen 14,

Here is part of it. Seems like it is working ok? I've just conducted two migrations with no problems and no freezing. (grr)

[Sven_Burger]

I'm not sure if I was just testing it all day yesterday with the Console open - or something has fixed itself (HA!) but it appears to be working normally now.

Thanks for everyone taking the time to post.

[grasshopper]

'Manual switchover' is not a good thing. Without seeing all the logs, it's difficult to tell, but the LUN numbers should be the same on both hosts.

[Sven_Burger]

Hmm..

It shows 40:0:0:1 as the VMFS Volume for both hosts....