helper server resets and reports 'No Operating System Found' when trying to boot

[ifaxsoln]

When trying to convert a running physical redhat machine into vCenter using converter standalone, the helper server gets booted, gets an IP address, but then within a minute or two (just after converter reports that it is partitioning and formatting drives) the console of the helper server goes black, then comes back moments later with 'No Operating System Found'. It almost seems to me that the newly created helper server reboots just after the disks get created and formatted, but of course since there's nothing on them yet the machine cannot boot.

I have attached screen shots which illustrate what I see.

-Darren

[ifaxsoln]

I have followed the instructions at:

http://kb.vmware.com/kb/1008209

http://kb.vmware.com/kb/1008210

to stop the helper machine from rebooting and to allow login access. I am now actually able to login to it while it's partitioning and formatting.

However, the helper machine's console still goes blank about 2 mins into the process, and when it comes back it have the 'No Operating System' error message again.

Is it possible there is something within our ESXi setup that is causing this to happen and that the problem is unrelated to VMWare Converter? I'm not seeing any Task or Event being logged either at the VM level or at the ESXi server level that says the VM is being powered down, ... it just seems the console goes dark and resets for no reason?

-Darren

[ifaxsoln]

I'm thinking this section of the converter server's log file might be of interest?

[snip]
[2009-03-15 18:39:38.689 'App' 4080102304 verbose] [taskInfo,1278] [TaskInfoPublisher] Received task updates at version 13
[2009-03-15 18:39:38.689 'App' 4080102304 verbose] [taskInfo,1297] [TaskInfoPublisher] processing object update 0 for filter 0
[2009-03-15 18:39:38.689 'App' 4080102304 verbose] [taskInfo,1310] [TaskInfoPublisher] update for prop info
[2009-03-15 18:39:38.689 'App' 4080102304 verbose] [taskInfo,1328] [TaskInfoPublisher] SetTaskInfo for channel task-5
[2009-03-15 18:39:38.689 'App' 3587705760 verbose] [task,661] [LRO] Keep alive notification sent
[2009-03-15 18:39:38.689 'App' 3587705760 verbose] [taskInfo,816] [TaskInfoListener] Waiting on channel task-5
[2009-03-15 18:39:38.696 'SOAP' 4033870752 error] Exception in DeserializeObj: vmodl.MethodFault
[2009-03-15 18:39:38.697 'SOAP' 4033870752 error] [00] eip 0xf7ef04fe  
[01] eip 0xf7e4d5d0  
[02] eip 0xf7da75c0  
[03] eip 0xf79f1775  
[04] eip 0xf7ae26f0  
[05] eip 0xf7ae4541  
[06] eip 0xf7ae35bf  
[07] eip 0xf7ae48f0  
[08] eip 0xf7ae4bf7  
[09] eip 0xf7ae35bf  
[10] eip 0xf7ae48f0  
[11] eip 0xf7ae59e4  
[12] eip 0xf7ae645a  
[13] eip 0xf7ae49de  
[14] eip 0xf7ae35bf  
[15] eip 0xf7ae48f0  
[16] eip 0xf7ae59e4  
[17] eip 0xf7ae645a  
[18] eip 0xf7ae49de  
[19] eip 0xf7ae35bf  
[20] eip 0xf7ae48f0  
[21] eip 0xf7ae59e4  
[22] eip 0xf7ae645a  
[23] eip 0xf7ae49de  
[24] eip 0xf7ae35bf  
[25] eip 0xf7ae48f0  
[26] eip 0xf7ae5034  
[27] eip 0xf7ae5092  
[28] eip 0xf7af3084  
[29] eip 0xf7a5b79b  
[30] eip 0xf7a5c8ce  
[31] eip 0xf7a5d1e2  
[32] eip 0xf7a5d3ed  
[33] eip 0xf7a5c696  
[34] eip 0xf7f1acfb  
[35] eip 0xf7f2b8c1  
[36] eip 0xf7f1fd1c  
[37] eip 0xf7f25f32  
[38] eip 0xf7f1acfb  
[39] eip 0xf7f1a95d  
[40] eip 0xc013cc  
[41] eip 0xb5a1ae  

[2009-03-15 18:39:38.698 'App' 4033870752 verbose] [taskInfo,1176] [TaskInfoPublisher] WaitForUpdates failed: vmodl.fault.SystemError
[2009-03-15 18:39:38.698 'App' 4033870752 verbose] [taskInfo,1177] [00] eip 0xf7ef04fe  
[01] eip 0xf7e4d5d0  
[02] eip 0xf7da75c0  
[03] eip 0xf7af399f  
[04] eip 0xf7a5b79b  
[05] eip 0xf7a5c8ce  
[06] eip 0xf7a5d1e2  
[07] eip 0xf7a5d3ed  
[08] eip 0xf7a5c696  
[09] eip 0xf7f1acfb  
[10] eip 0xf7f2b8c1  
[11] eip 0xf7f1fd1c  
[12] eip 0xf7f25f32  
[13] eip 0xf7f1acfb  
[14] eip 0xf7f1a95d  
[15] eip 0xc013cc  
[16] eip 0xb5a1ae  

[2009-03-15 18:39:38.698 'App' 4033870752 verbose] [taskInfo,1095] [TaskInfoPublisher] Disconnected
[2009-03-15 18:39:38.698 'App' 3587705760 info] [taskSpec,357] [Task] Caught an exception while waiting on the agent task to complete. Gathering agent logs before propogating the exception further. Exception message: vmodl.fault.SystemError
[2009-03-15 18:39:38.699 'App' 3587705760 info] [diagnosticManager,223] The task with id = "task-15" was found to be a "recent" task.
[2009-03-15 18:39:38.699 'App' 3587705760 info] [diagnosticManager,285] No existing log bundle found for task with id = "task-15". The task is still "recent" so a log bundle will now be generated for it.
[2009-03-15 18:39:38.699 'App' 3587705760 info] [diagnosticManager,544] Retrieving task related diagnostics for server task with id = "task-15".
[2009-03-15 18:39:38.699 'App' 3587705760 verbose] [diagnosticManager,567] Generating agent log bundle for server task with id = "task-15" and agent task with id = "task-5".
[2009-03-15 18:39:39.865 'App' 3587705760 verbose] [diagnosticManager,594] Retrieving agent log bundle for server task with id = "task-15" and agent task with id = "task-5".  Storing at "/tmp/vmware-temp/vmware-root/agent-task-5.zip"
[2009-03-15 18:39:40.362 'App' 3587705760 warning] [diagnosticManager,608] CRC mismatch in agent task logs. Agent logs will not be added to server task log bundle.
[2009-03-15 18:39:40.362 'App' 3587705760 warning] [diagnosticManager,85] Error deleting temporary agent log bundle at path"/tmp/vmware-temp/vmware-root/agent-task-5.zip" from server machine.
[2009-03-15 18:39:41.550 'App' 3587705760 error] [task,339] [LRO] Unexpected Exception: vmodl.fault.SystemError
[2009-03-15 18:39:41.550 'App' 3587705760 verbose] [task,340] [LRO] Backtrace:
[00] eip 0xf7ef04fe  
[01] eip 0xf7e4d5d0  
[02] eip 0xf7da75c0  
[03] eip 0xf7b2b6b7  
[04] eip 0xf26855d6  
[05] eip 0xf268596b  
[06] eip 0xf26784f3  
[07] eip 0xf2698a81  
[08] eip 0xf26765e2  
[09] eip 0xf267780b  
[10] eip 0xf267e1a7  
[11] eip 0xf267cdd7  
[12] eip 0xf7f1acfb  
[13] eip 0xf7f2b944  
[14] eip 0xf7f25ff2  
[15] eip 0xf7f1acfb  
[16] eip 0xf7f1a95d  
[17] eip 0xc013cc  
[18] eip 0xb5a1ae  

[2009-03-15 18:39:41.551 'App' 3587705760 info] [task,368] [task-15] -- ERROR -- Convert: vmodl.fault.SystemError
(vmodl.fault.SystemError) {
   dynamicType = <unset>, 
   reason = "vmodl.MethodFault", 
   msg = ""
}
[#43] [2009-03-15 18:43:51.226 'App' 4080102304 info] [diagnosticManager,301] Generating Converter.Server log bundle with no agent logs.

Please let me know what other information I can provide? I've dug in pretty deep here but am still relatively dead in the water.

-Darren

[vmweathers]

That server log's set of tracebacks is probably a red herring: they could be a result of the failure, but are not the cause and almost certainly not even a hint towards the cause. The helper VM itself is crashing, which is not related to the Converter server at all.

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[vmweathers]

to stop the helper machine from rebooting and to allow login access. I am now actually able to login to it while it's partitioning and formatting.

Ok, that's great, this should allow us to proceed further with debugging.

However, the helper machine's console still goes blank about 2 mins into the process, and when it comes back it have the 'No Operating System' error message again.

Yeah, the helperVM itself is crashing... I'm not sure if this is a failure of the vmx process in ESX, or if the kernel has crashed.

Is it possible there is something within our ESXi setup that is causing this to happen and that the problem is unrelated to VMWare Converter? I'm not seeing any Task or Event being logged either at the VM level or at the ESXi server level that says the VM is being powered down, ... it just seems the console goes dark and resets for no reason?

It could indeed be related to the ESX for some reason. That's why I asked for these items in my reply on the other thread:

" After this, please obtain the vmx log file from the target datastore's VM folder, as well as the hostd (or vpxa) logs. ... Could be somehow specific to the hardware, please include in the new thread any details you can provide on the target ESX's hardware and software version."

So can you please obtain these things (logs, ESX h/w s/w details) and post them? If you're not comfortable posting them in this forum then private mail me and I'll give you an email address to send them to.

FYI, the vmx log file is in the actual VM's folder in it's datastore. The hostd/vpxa (when using VC it's vpxa, when going directly to ESX it's hostd) logs are under /var/log/. BTW, I'll also need some information to help interpret the logs... i.e., ~ time at which the VM was started and then crashed, and the name of the VM.

(If your question has been resolved please mark the answers as "Helpful" or "Correct".)

[ifaxsoln]

So it turns out that after some very careful analysis of our logs, representatives from VMWare have been able to determine that the helper server is being reset by the HA/Heartbeat stuff we had enabled at the cluster level. It's not yet clear whether this was a misconfig on our side, or something wrong with the vmware tools in the helper server (we think it should be doing whatever heartbeating it needs to do since it's clearly alive), but by disabling heartbeat we have been able to complete at least one P2V migration.

The next one we're attempting is our mail server, and it seems we need to quiesce the filesystem on that one a bit before the tar procedure executing under the covers will be happy (on our first attempt tax exited with nonzero status). We had hoped we could simply convert it red-hot, then rsync the mail spools from a quiesced physical server to just 'top things up' as it were, but we may need to really turn down the services on the box before we're able to import/convert it.

-Darren