VMware Cloud Community
jmatos
Contributor
Contributor

VC 2.5 Stops Responding After a While

Hi,

my VC upgrade went well but now, after some days, it stops responding until i restart the service (which takes too long to stop). After this restart, VC works fine for a few minutes only.

The error message is "the request failed because the remote server took too long to respond".

Already restarted mgmt-vmware on the hosts and shrinked db as well (260 mb w/ SQL 2K5 Express)

Anyone went through this issue?

Reply
0 Kudos
49 Replies
jprior
Enthusiast
Enthusiast

I got that error after upgrading to 2.5 as well, I think its an increase in resources required by SQL 2005 vs earlier versions.

How much ram does your VC box/VM have? I was running VC in a VM w/512mb of RAM allocated to it, did an upgrade to 2.5. I had problems with the SQL server using up all my CPU and running out of RAM, virtual center service would fail and need restarting, which would take forever and only help for a few minutes. Increasing the RAM helped a lot, but I still ended up making a new VM on a more powerful host and doing a fresh install. new VM has a single vCPU of a 2.0ghz cloverton xeon, 1Gb vRAM, two scsi virtual disks residing on a SAN lun. first disk is OS (server 2003 r2 sp2 x86), second is virtual center & SQL (SQL express 2005 + management studio express). I'm RDP on to it right now, running VIC and have ~10% cpu and 110mb free RAM, of the 1024Mb. SQL is taking up 230Mb, VPxClient is 190Mb, vpxd is 87Mb, update manager is 83Mb, convertor is 35mb, tomcat is 33mb.

Reply
0 Kudos
jmatos
Contributor
Contributor

Hi,

my VC works on a Physical Dual Core w/ 3GB RAM. Windows 2K3 SP2 Standard.

It's not a lack of performance issue. I'm almost thinking in a complete reinstallation of VC Server...

Reply
0 Kudos
paulkbeyer
Contributor
Contributor

Hi, I've got the exact same problem since I upgraded my VC installation.

I went from the bvery latest versions of both VC and ESX hosts. It's only a 5 host farm without any of the DRS/HA features enabled currently so very uncomplicated. I kept historical performance data but if I do a report on performance back to a month which goes past the upgrade date - only date from pre Dec 14th (date I upgraded) appears. Current performance data (real time) is available from the VIClient and daily and weekly data simply isn't there. The VC server is a physcial dual 3.6Ghz box with 4Gb on a 32bit platform and bugger all else running on it , yet the SQLexpress process is consuming 50% (1 CPU) almost constantly.

I've got a support request open which is SR#1100245091 and i've not had anything useful back from the support rep as yet. So if I get progress shortly on this case number, refer to that with your VMware support rep once I've posted more info.

I'm seriously thinking about a full install from scratch too as it seeming like a quicker fix and I'm not THAT bothered about the performance data even though i'd much prefer to keep it.

Could you post the upgrade steps you took, perhaps there's something totally screwy with the way we upgraded.

I'll do the same shortly.

Reply
0 Kudos
jprior
Enthusiast
Enthusiast

I followed the steps outline in Virtual Gabrie's blog, here - http://virtualgabe.wordpress.com/2007/12/08/installing-virtual-center-25-esx-35/

Reply
0 Kudos
paulkbeyer
Contributor
Contributor

Hi, yes, you've saved me some fingerwork, I used his blog to guide me through the forest of questions that were presented to me too. Though I did do something slightly different around the OBDC section where I believe I deleted the old DSN and created a new one with the same name using the SQL native client driver which made more sense to me than doing what Gabrie suggested. Other than that it was verbatim as there was little else to worry about. That aside I still have the same symptoms as you so I don't think my actions made any difference. I'm still waiting for something ledgible from support other than 'what's your hardware' so I'll update when I can.

Here's hoping this gets sorted soon.

Regards

Paul.

Reply
0 Kudos
jprior
Enthusiast
Enthusiast

Yes, I created a new system DSN using the native driver pointing to the dB and used that instead of editing the old one or deleting it. I planned to go back and remove the dsn later, but ended up deleting the whole VM once I found that a fresh install had no issues.

Reply
0 Kudos
paulkbeyer
Contributor
Contributor

Oh, i don't believe you've mentioned that already? You've installed VC from scratch and it's presented you with no problems?

I'm not really surprised, I was just hoping to get a resolution rather than smashing my old VC install to bits and rebuilding it.. I'm still waiting to hear from support but if that's cured it for you for definate that'd be good to hear.

Regards

Paul.

Reply
0 Kudos
kenrblan
Contributor
Contributor

I just started looking at the forums when I encountered the same issue. I also did an upgrade install with the migration from MSDE to MS SQL Express 2005. I would be curious to find out what support tells you.

My VirtualCenter server is a physical server that has a Xeon 2.4Ghz processor with 2GB memory running on Windows 2003 R2 SP2. The upgrade was from VC 2.02 to 2.5. It is managing 5 physical hosts (all running ESX 3.5) with 44 virtual machines. Within the farm there are two DRS/HA clusters. It appears that my historical performance data has been retained, but no performance data since the upgrade is available (only realtime).

Ken

Reply
0 Kudos
jprior
Enthusiast
Enthusiast

Yes, sorry - it did indeed cure the problems, no large CPU usage, no failing virtual center service, no timeouts etc.

My original configuration was running in a Server 2003 SP1 EE vm, with single vCPU and 512mb ram on a Dell Poweredge 2850 (dual Xeon 2.8 HT, 3Gb RAM, Server 2003 EE SP1). VM had a single Virtual disk, 12gb scsi. I performed the upgrade per Gabrie's blog, and immediately had problems with the VM becoming unresponsive, and the client unable to connect. I upped the VM to 1gb ram, but still the virtual center service would fail on starting, and the SQL service would take 100% cpu for 15-20 minutes, then idle for a while, the ramp up again.

I moved the whole VM to a new host, Dell Poweredge 2950 with dual 2ghz xeon quad cores, 8gb ram, Server 2003 X64 SP2 EE) and still the same problem. So I created a new VM, with 1 vCpu, 1Gb RAM, 2 SCSI virtual disks - 12Gb OS, 8Gb Data. Installed Server 2003 x32 SP2 EE and patched, with .NET 2.0. I then installed MS SQL Express 2005 w/SP's to the data disk, and Management Studio Express to the same data disk. Created two database's, vmware and vmupdate (first for VC, second for patch manager) then created two system DSN's (native SQL driver). Installed Virtual Center, using custom install to select the data drive as the install target, and to pick my DSN's. Had to redownload my license file as the format has changed from VC 2.0, but was able to add all my ESX 3.5 servers back and recreate my HA/DRS clusters without issues.

Reply
0 Kudos
cryptonym
Enthusiast
Enthusiast

I am having a problem, not with lockups, but with missing performance data. Basicly, I have real-time and past day data, but nothing for last week or month. I filed ticket 1100712751 with tech support. Feel free to refer to it if you also file. So far they had me remove one of my hosts, then add it back in, thinking it may be a problem with the host id in the database. 15 hours later, no performance data exists in the week or month view. Tasks and events are logging correctly, but not the performance data. We'll see if they come up with anything more today. We confirmed that the time is in sync on the hosts and vc.

My setup is a little more complex then it seems most are reported here. I have 6 esx hosts and about 20 vms, but many of the hosts are in DRS and/or HA clusters. Additionally, I don't want to lose my security settings and other settings. I don't care if I lose the performance data, it hasn't collected it the last 2 weeks (I also upgraded on the 14th) while I was off anyway. I had to upgrade the database from MSDE to SQLExpress before doing the VC upgrade.

I hope they can provide a work around. I can roll it all back to 2.02 easy enough, as I pulled one of the mirrored drives prior to the upgrade.

If any news comes along on this I'll post here.

Warren

Reply
0 Kudos
cryptonym
Enthusiast
Enthusiast

I just completed a call with the vmware tech. They have recreated the problem in the lab by upgrading the db from msde to sqlexpress, then doing the VC upgrade. They had me export my VC logs and upload them and are reviewing them. From here it is being escalated to the development team to determine if there is a work around or hotfix coming.

While poking around myself, I went into the vmware infrastructure dir on the VC server and was looking at the .sql files it uses. Near the top in each of the "job_schedule?_mssql.sql" files is an if statement that detects whether it is running on "Express Edition" and skips the process if it is. I suspect in the end, the problem may lie in these files. Could someone who has done a fresh install of VC (not an upgrade) please look inside of these files on their working setup and see if this if clause is or is not present in their files?

I also went looking in the sql system tables, and a select * from msdb.dbo.sysjobschedules shows there are links to 3 jobs (ie. 5 min, 1hr, 1week jobs), but in my case the next run time and next run date are set to 0, which probably means "never". I sent this info up to vmware tech to see if they think we can seed it to trigger the initial run. Could someone who has done a fresh install of VC (not an upgrade) please run the above query in sql express and see if these values are also 0 on their working system?

I plan to do a full rollback to my 2.02 setup on MSDE and run on an upgrade without telling it to keep the existing performance data. The tech seemed hopeful that this may fix the problem for me, and still keep all of my security and other setting intact.

More later.

Reply
0 Kudos
jprior
Enthusiast
Enthusiast

I just completed a call with the vmware tech. They have recreated the problem in the lab by upgrading the db from msde to sqlexpress, then doing the VC upgrade. They had me export my VC logs and upload them and are reviewing them. From here it is being escalated to the development team to determine if there is a work around or hotfix coming.

While poking around myself, I went into the vmware infrastructure dir on the VC server and was looking at the .sql files it uses. Near the top in each of *_the "job_schedule?_mssql.sql" files is an if statement that detects whether it is running on "Express Edition" and skips the process if it is. I suspect in the end, the problem may lie in these files. Could someone who has done a fresh install of VC (not an upgrade) please look inside of these files on their working setup and see if this if clause is or is not present in their files?_</strong</p>

I also went looking in the sql system tables, and a select * from msdb.dbo.sysjobschedules shows there are links to 3 jobs (ie. 5 min, 1hr, 1week jobs), but in my case the next run time and next run date are set to 0, which probably means "never". I sent this info up to vmware tech to see if they think we can seed it to trigger the initial run. Could someone who has done a fresh install of VC (not an upgrade) please run the above query in sql express and see if these values are also 0 on their working system?

I plan to do a full rollback to my 2.02 setup on MSDE and run on an upgrade without telling it to keep the existing performance data. The tech seemed hopeful that this may fix the problem for me, and still keep all of my security and other setting intact.

More later.

My VC 2.5 system is a fresh install, SQL Express 2005. Looking in the job_schedule1_mssql.sql file I find

"IF ((select serverproperty('edition')) = 'Express Edition')
GOTO EndSave "

My install appears to be working without issues. Let me know if you need additional info etc.

Reply
0 Kudos
Dave_Mishchenko
Immortal
Immortal

A new install on SQL Express doesn't create any jobs and that is because Express doesn't include SQL Agent which would take care of running the jobs.

Reply
0 Kudos
paulkbeyer
Contributor
Contributor

Chaps,

I will be progressing my case along with all your situations but not till Tuesday. I'll update the thread as soon as I have progress.

Regards

Paul

Reply
0 Kudos
cryptonym
Enthusiast
Enthusiast

FYI

VM tech asked me to roll back to the 2.02 and upgrade again, but not to bring in the existing performance data. At least restoration was simple, I just had to pop in the old mirrored drive and remirror it onto the server over the existing (broken) setup. Here is what I've done:

1. Installed MS SQLExpress

2. Detached the old database from MSDE

3. Attached old database to SQLExpress

4. Setup the new ODBC Native connection

5. Created a new ODBC connection for the update manager

6. Ran the VC2.5 install

7. Chose to NOT preserve the existing performance data

Unfortunately, I still have no data in the past week or month views. Only last day and realtime is presenting. Looking more and more like the solution may be to deploy the uncrippled version of SQL2005.

I forwarded this info back to the tech this morning and will have to wait and see what they come back with next. Due to other scheduled work, it will probably be at least Monday before I can put more time in on this issue.

Warren

Reply
0 Kudos
cryptonym
Enthusiast
Enthusiast

Bad news from vmware...

According to the development staff via my tech contact, it is expected behavior to not log (rollup actually) performance data when using SQL Express, due to not having SQLAgent in SQLExpress. This makes me unhappy, since this is a loss of functionality from the MSDE implementation. I'm not sure why VMware would choose to force people to switch to an inferior solution, and give up functionality that they've enjoyed for years.

You will be happy to know that a documentation "fix" is in the pipeline that will document this small detail that they overlooked. According to him it was just an accidental omission from the documentation, which I can understand. But I don't consider it truly "fixed", would you???

The solution is to roll my system back to the VC 2.0.2 and MSDE again, and then go through the upgrade again, this time deploying against SQL2005. Fortunately, we have plenty of those SQL2005 servers in production, I just didn't think it was needed. Guess I was wrong, and I pity those who don't have them just laying around.

Basicly, SQLExpress is crippleware, laming VirtualCenter horribly. I have logged 2 "feature requests". The first is for a Linux version of VirtualCenter, to get off this Micro$oft junk. The second was for VirtualCenter to support MySQL, to replace the Micro$oft SQLExpress junk.

I encourage everyone to submit the same two requests. They've heard both before, and haven't delivered yet, but this latest twist is just too much to ignore.

Warren

Reply
0 Kudos
Dave_Mishchenko
Immortal
Immortal

&gt; This makes me unhappy, since this is a loss of functionality from the MSDE implementation. I'm not sure why VMware would choose to force people to switch to an inferior solution, and give up functionality that they've enjoyed for years.

Just to be fair, MSDE has not been supported for production on VC 2.0.x.

Reply
0 Kudos
cryptonym
Enthusiast
Enthusiast

Valid point, but MSDE worked without err, and all the functionality was enabled at least. And vm support never turned down any questions I had back then anyway.

Reply
0 Kudos
Hairyman
Enthusiast
Enthusiast

The company that i work for setup the VC server with a full version of SQL 2005 Standard on Windows 2003 R2 SP2, we had no issues when we upgraded from 2.0.1 to 2.0.2 and hopefully we won't when we go to 2.5. It's currently running on a Dell PE1850 as the original server blue screened (Dell PE 1950)and this one was sitting "spare"

Cheers

Aaron

Reply
0 Kudos