VMware Cloud Community
alvinswim
Hot Shot
Hot Shot
Jump to solution

Storage VMotion - A general system error occurred: Failed waiting for data. Error bad0007.

Folks,

I've searched through out this forum and haven't found an answer that works in the many posts out there, I figure I'd ask it again.

Here's the background:

We have 12 Identical Dell M600 Blades in 2 chassis with 16GB Ram, 2 x Xeon E5430, they are all connected to an Equallogic PS 5000XV iSCSI SAN on a separate iSCSI network (vswitch1) with 2 dedicated Nics and dedicated switch, dedicated iscsi svc console, dedicated VMkernel port for iscsi access. The Access net (vswitch0) contains the VM port groups for our different networks as well as a svc console and vmkernel port for VMotion with 2 separate NICS as well.

We are Running ESX 3.5 U3 and VCenter 2.5 U3 on Win2k3 R2

Vmotion works between all the servers, Storage Vmotion works for most of the machines, HA is working, and set to 2 host failurs with no vm monitoring, DRS is set to manual for now as I have some machines on local stores as I complete my LUN rebuilds, there are no rules set for DRS and VMware EVC is enabled for Intel hosts. However, I'm just gonna describe the one machine that needs to do svmotion below.

Here's the problem:

I am trying to Svmotion via svmotion.pl --interactive, a Windows 2000 Machine with one virtual disk and one virtual RDM. I am aware of the requirements for the RDM and the settings required for svmotion, independent is unselected for the RDM, and I have also svmotioned several machines both linux and win2k3 with the same configuration without issue. In the interactive session I choose to individually place disks, and I choose only the VM's virtual disk to be moved, basically as I've seen it will move the vm's vdisk and then copy over the RDM pointer.

The CPU utilization of this machine is around 25% average. but I try to run migrations during the lowest times. and The Host it self shows only about 5.5GB of the 16GB RAM used. so I think we are good on RAM. the volume/datastore I'm migrating from has 485GB free, and the volume/datastore I'm migrating to has 145GB free. the VM virtual disk is only about 33GB.

I run the svmotion script from the windows version of the RCLI. and when begin the process I get the following error at around 2% progress:

"Received an error from the server: A general system error occurred: Failed waiting for data. Error bad0007. Bad parameter."

After searching around I found the following fixes from the release notes of U2

  • Migrate.PageInTimeoutResetOnProgress: Set the value to 1.

  • Migrate.PageInProgress: Set the value to 30, if you get an error even after configuring the Migrate.PageInTimeoutResetOnProgress variable.

I made those changes and I still get the same error.

When I dig in to the log, I see these entries in the vmkwarning log:

Feb 24 00:17:32 iq-virt-c2-b6 vmkernel: 82:04:13:56.794 cpu4:1394)WARNING: Heap: 1397: Heap migHeap0 already at its maximumSize. Cannot expand.

Feb 24 00:17:32 iq-virt-c2-b6 vmkernel: 82:04:13:56.794 cpu4:1394)WARNING: Heap: 1522: Heap_Align(migHeap0, 1030120338/1030120338 bytes, 4 align) failed. caller: 0x988f61

Feb 24 00:17:32 iq-virt-c2-b6 vmkernel: 82:04:13:56.794 cpu4:1394)WARNING: Migrate: 1243: 1235452646235015: Failed: Out of memory (0xbad0014) @0x98da8b

Feb 24 00:17:32 iq-virt-c2-b6 vmkernel: 82:04:13:56.794 cpu2:1395)WARNING: MigrateNet: 309: 1235452646235015: 5-0xa023818:Sent only 0 of 4096 bytes of message data: Broken pipe

Feb 24 00:17:32 iq-virt-c2-b6 vmkernel: 82:04:13:56.794 cpu6:1396)WARNING: Migrate: 1243: 1235452646235015: Failed: Migration protocol error (0xbad003e) @0x98da8b

Feb 24 00:17:32 iq-virt-c2-b6 vmkernel: 82:04:13:56.794 cpu2:1395)WARNING: Migrate: 6776: 1235452646235015: Couldn't send data for 56486: Broken pipe

At this point I'm stuck.. Could it be the windows RCLI? the vcenter server? or the service console not having enough RAM? We've already increased all our service consoles to 512MB...

Any help would be greatly appreciated..

Thanks in advance.

Alvin

Tags (1)
Reply
0 Kudos
1 Solution

Accepted Solutions
mikemast
Enthusiast
Enthusiast
Jump to solution

The vmkernel error about out of memory, I have had that before. And vmware support recommended setting the service console memory to max of 800M. And I did that and have not have issues after that.

See if that helps the issue.

Mike

View solution in original post

Reply
0 Kudos
9 Replies
RParker
Immortal
Immortal
Jump to solution

This looks like a problem with the local machine where you are trying to run svmotion.pl. Have you tried rebooting it?

Then use this plugin instead of the RCLI: http://sourceforge.net/project/showfiles.php?group_id=228535 (vip svmotion)

alvinswim
Hot Shot
Hot Shot
Jump to solution

Thanks RDParker,

I'll give it a shot, we've rebooted the VCenter server before and it shows 2GB free ram. who knows. I'll give the plugin a try if not, you think I should try the RCLI vm appliance too?

thanks I'll update you

Reply
0 Kudos
mikemast
Enthusiast
Enthusiast
Jump to solution

What is your service console memory set to?

Reply
0 Kudos
alvinswim
Hot Shot
Hot Shot
Jump to solution

The Service console memory is set to 512MB

Reply
0 Kudos
mikemast
Enthusiast
Enthusiast
Jump to solution

The vmkernel error about out of memory, I have had that before. And vmware support recommended setting the service console memory to max of 800M. And I did that and have not have issues after that.

See if that helps the issue.

Mike

Reply
0 Kudos
alvinswim
Hot Shot
Hot Shot
Jump to solution

hmm. I think I may have to give that a shot.

I tried RDParker's link to the plugin I still get the same issue but with a different VM and host. so I'm gonna try the linux RCLI then if that doesn't work I'll try increasing the memory to 800 on the svc console. the question is, because it requires a reboot, what happens if a reboot cures all this head ache or do you think its more like the svc console memory?

Reply
0 Kudos
mikemast
Enthusiast
Enthusiast
Jump to solution

I have made it a standard to increase my service console memory to 800M on all my hosts. It has seemed to help out.

But you can reboot with the setrings you have now and see if that fixes the issue. The up the memory if it doesnt.

Mike

Reply
0 Kudos
alvinswim
Hot Shot
Hot Shot
Jump to solution

thanks to all for the help, but what really helped was increasing the svc console ram to 800MB. so I guess we can say that svmotion requires more ram than vmotion. Vmotion worked just fine with 512MB..

I also think that running the open manage in stuff in the service console took a good chunk of memory.

The svmotion plugin is very nice so I recommend that, but I'm a command-line junkie so i'll stick with that.

thanks again

alvin

Reply
0 Kudos
GeorgeZhou
Contributor
Contributor
Jump to solution

great, seems my svmotion continues after free some host memory

Reply
0 Kudos