Re: operation timed out... or did it?

john_johansson · ‎01-30-2007

My problem is the following:

I recently started upgrading to VI3, i'm currently running VC 2 and have two esx 3.01 hosts and two esx 2.x host.

Until this morning everything was working fine and i was about to continue upgrading the esx hosts.

But then strange things started to happen. I could no longer edit VM settings, use VMotion, power on VM's etc, anything and i everything i try to do, more than often result in Operation timed out, after waiting for 10-15 minutes and watching it being "in progress".

Funny thing though, if i connect directly to my esx host through the VI client, it all seems fine.

Some examples:

1. i want to do some hot migrations. I choose a machine, try to migrate it in VC, it goes to 10% then stalls, at the same time when looking on the hosts through the VI client connected directly to it, it all shows up with no error msg's and completes after less than a minute, but in VC it just sits on 10% for 15 min, then times out and becomes orphaned in VC.

2. i want to increase memory on a powered off VM in vc, i choose edit settings, increase memory, click ok, then it says "in progress" for fifteen min, then times out, but watching it through VI client connected directly to host, it works like a charm.

i'm quite a novice on VMware as you might have already guessed, and this is my first post, so please be gentle anywho, any help at all on resolving this problem would be highly appreciated.

Many thanks,

John

Rob_Bohmann1 · ‎01-30-2007

you could try either restarting the virtual center agents on the hosts affected or removing and readding the hosts in virtual center as a first step.

Is this happening to all of your hosts or just the 2.5 or just the 3.x hosts?

john_johansson · ‎01-31-2007

Ok, here's what i did, i ran

service mgmt-vmware restart

service vmware-vpxa restart

Those commands did bring back the servers listed as orphaned after vmotion operations, but did not change the fact.

I will try removing the host from VC soon, it seems that it needs to be in maintainence mode to be removed, and that's going to be a be a pain, to get it into maintainence mode i need to migrate all running VM's from it, and i can only do 2 at the time, and they take 15 min to time out before a can start new vmotions.

To answer your question:

The problems i'm having only seem to be on one of the upgraded hosts, lets call my hosts ESX1-4, 1 & 2 are running 3.01, 3 & 4 are 2.x.

Host 1 is the one with the problems. 2 seems to be working fine, except on VMotions, when the following scenario happen:

A migration from 1 -> 2 progress will stall after 10% and stay on 10 'til it times out.

VMotion from 2 -> 1 progress bar will go to 90% and stay there 'till time out.

VMotion from 3 & 4 works fine to 2 (and probably 1 as well, can't remember, don't want to do any more till it's solved, since migration from old hosts are irreversible)

Edit hardware settings, power on hosts etc. all works good on 2,3 & ,4, so as stated above, it seems to be a problem with host 1.

I'll let you know how the removal/readding of the host in VC went once it's done.

Gazzaski · ‎01-31-2007

I have seen symptoms like this on our systems, what fixed the issues for me was to stop and start the virtual centre service on your virtual centre server, so that the whole of VC is refreshed.

john_johansson · ‎01-31-2007

Starting and stopping virtual center in my case does nothing. I have done that so many times now i lost count

Anywho, as suggested i tried to remove the host and then readd it.

Removed it sucessfully, but now i'm not able to readd it !! it just shows up as disvonnected, and when trying to connect it, the add host wizard comes up, and when done, it says: Not enough licenses to add host or something along thoose lines.

Funny thing is thouh, i have 4 available cpu licenses for esx standard and vmotion etc. i've restarted the license server and virtual center server, nothing works.

I tried to change license server configuration in VC to connect to IP instead of localhost, can't do it! not enough license error message pops up when trying to do that also.

Something seems to be seriously wrong with virtual center / license server ...

Any help at all on how to proceed is highly appreciated!!

Gazzaski · ‎01-31-2007

Hi John

I did not mention it it my last post as it seem so drastic, but the one time when restarting the VC service did not work and we also had the license error, as a last resort we removed all hosts from the database and uninstalled VC from the server re-installed and added them all back in again, as I say it was a last resort so if someone here as a solution before doing this great, but if you run out of ideas it will be worth a go, we have had no problems since, and it was about a month ago now.

john_johansson · ‎01-31-2007

Thats it probably what i'll end up doing eventually, as i last resort. First i wanna give this thread a day or two and see if it can be solved, but i'm kinda running out of hope :~

Rob_Bohmann1 · ‎01-31-2007

Hi John,

I looked in my VC server as I added a host earlier today. Here is what I see in my VC logs and despite the errors, it was added and licensed successfully. Can you look in your logs and post the entries when you try to add your host back into VC?

... does not like SSL cert when I add it with just the host name vs FQDN.

then...

"failed to connect to host xxxx, check that authd is running correctly (lin/connect error 11). then about 30-45 seconds later it states

..synchronizing host xxxx

..updating vpxa for host xxxx

and then a few more synchronizing host xxxx.

john_johansson · ‎01-31-2007

hi,

My logs are full of errrors, when trying to add the failing host:

Virtual Center log:

\[2007-02-01 06:39:32.764 'App' 4844 info] \[VpxLRO] -- BEGIN task-813 -- group-h4 -- vim.Folder.addStandaloneHost

\[2007-02-01 06:39:32.936 'BaseLibs' 4844 warning] SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems: