No logging of performance data when VI Client is offline

[pfuhli]

Hey there,

I'm very impressed of the new capabilities VC 2.0 provides. But I have a problem with logging of performance data. It seems that VC is only logging data when my client is running. All the other time the lines of the charts are going to zero. Do I have to configure this at the server?

What do I have to change?

Thanks for your help!

daniel

[AndyN]

I am having the same problem. I have upgraded to VC 2.0.1. I am using my VC server to manage 2 ESX 3.0.1 hosts and 2 ESX 2.5.3 hosts.

On the ESX 3.0.1 hosts I am only getting Performance data for any counter back about 2 days, specifically Sunday at 8:30 AM. Historical data is non-existent before that time.

On my ESX 2.5.3 hosts, I am seeing the same cutoff of historical data, AND the Memory counter is flat lined at zero for the hosts and all the VMs. RealTime data looks fine for Memory, but once I select Weekly or any other historical option....I see nothing but a flat line at zero.

Help!

[stuten]

I upgrade to 2.0.1 and decided to let my upgraded db go since the performance data seemed lost anyway. For the last week all data looked great across the board. I came in this morning ot find that all the charts have major drop outs now, across all the hosts

I really thought 2.0.1 had fixed the problem. The only oddity I am aware of is that yesterday our VC server lost connection to the SQL server for about 10 mins -- other than that nothing has changed.

Prior to the 2.0.1 upgrade, my VC install would crash frequently (which is what I attributed the single point drops to, doesn't explain the huge drop outs in the historic views). Since the upgrade the only time that the VC server hasn't been running was yesterday when the SQL connection was lost.

I was waited until 2.0.1 hoping it would fix things, but no luck. I'm going to out a case and add to the pile.

[stuten]

As info, I have opened a case with support concerning this issue.

I wanted to share a couple of observations to see if anyone else sees the same. Yesterday I had data for the past couple of weeks with no holes -- life was good. I lost connectivity between my sql server and VC and as such the VC service shutdown. I cranked it back up and went about my business. I noticed this morning that I now have many zero chunks of data -- data that I had yesterday (not new data that I didn't get). For instance, the Past Week graph looks like swiss chesse when yesterday all those holes had data.

While I'm waiting on support I've noticed that my previous night's database dump was about 500 MB. Last night, presumably after the loss, the database was only about 250 MB. I restored the dump from the previous night and checked out the row count of vpx_hist_stat ... interstingly, I found that instead of having about 4.9 Million rows I now have 800,000 rows.

We dump transaction logs hourly so I went and looked at them. Leading up to when I had to restart the VC server the logs were all around 9 MB in size (very small). On the next hour after I restarted the service, suddenly the transaction log dump was 1.7 GB (wow, could that have been my data?) and then there after was back to around 9 MB in size.

Any one else that is seeing data loss seen anything similar?

Message was edited by:

stuten

I strongly urge anyone seeing this to open a case with VMWare so that they realize this is a real problem and not a config problem with a few people. So far I'm not getting the feeling that they are taking this very seriously.

[nwallaceml]

Hmmmm... you may be onto something here. Yesterday morning we had to reboot our SQL server and as a result, the VC server service shut down. I restarted the service a couple hours later, and went about my business. After reading your post I decided to take a look at our performance data and saw that most of our history is completely wiped out. We now only have data from the 16th @ 5:30pm to present. The only hole is the period of time the VC service was down.

Like you, when I upgraded from VC 2 to VC 2.01, I decided to start with a new database, so I should have nearly 2 weeks of data. We only do nightly database backups and transaction log dumps, but I can see on the 16th our db was 117mb and transaction log was 77mb (which is avg). On the 17th our db was 46mb and the transaction log was 431mb. As of right now, our db is 130mb and our transaction log is just over 1gb.

How does this thing go from a steady db growth of about 25mb per day with transaction logs hovering between 70mb and 80mb for 2 weeks, then bam, all of a sudden we have a transaction log over 1gb and I don't know what to think of the db. 130mb is about where it would be if nothing happened, but last night it was only 46mb and 12 hours later has grown by over 80mb.... strange

[nwallaceml]

I just remembered that I should look at the space allocated to the data files... space allocated in the db is 55mb out of 130mb. Space allocated for the transaction log is 98mb out of 1gb. So, whatever happened, wiped out my historical data and caused SQL server to grow the database and transaction log files pretty significantly. I have these files set to automatically grow by 10%, which I believe is the default....

[stuten]

same here, my db was an effective size of 1/2 (so no white space), the transaction log was full of gas, of course. Nonetheless, VC certainly pushed a lot of transactions through it to make it that large (oh like maybe my missing 4.1 Million rows of performance data)

Based on the response I get back I'm going to ask them to do a simple repro of taking an esx server, a vc server and a sql server. Let it run long enough to get some data and then yank the NIC out of the SQL server and see what happens. That isn't hard to do.

[AndyN]

Are you guys are using VC 2.0.1 to manage ESX 2.5.x hosts? If so, how does your historical performance data look for those hosts? I still see no data for Memory at all in my VC 2.0.1 console. Just a flat line at zero. My 3.0.1 hosts show normal looking historical Memory data.

[nwallaceml]

I only have ESX 2.5.x hosts right now. I am still waiting for VMWare to give the official blessing for the specific NetApp filer we have deployed here before upgrading to ESX 3.

I have performance data for cpu, memory, disk, network and system. The history for all counters only date back to the 16th when whatever happened, happened.

BTW, for what it is worth: SQL 2000 SP4, running 10 concurrent db connections, 8 collection threads and statistics collection level 2.

[AndyN]

It looks like setting the Statistics setting to "Level 2" may have done the trick....

[stuten]

I am currently running level 2 and had the problem. I have also seen it at level 1. Regardless of the settings, for me at least, if virtual center losses connection to the database and thus terminates upon restart many records go bye bye.

[eliot]

Well ive upgraded to vc2.01 and some servers to 3.01 and have huge gaps in my logs over the last week (several days ..)

Picture:

http://homepages.nildram.co.uk/~eliotmez/someone_eat_my_data.jpg

[MattOhio]

stuten - Did you get any response back from VMware on your SR regarding this?

[stuten]

Not yet, last response was that it was being "pursued". I have asked for an update, but nothing yet.

I'll be sure to update the forum when I hear something.

[wcrahen]

Just to go along with the rest I have seen the same thing happen, these graphs look just like mine. I was going to blow away the VC database and start fresh (rather than the upgrade I did) and see if that helps, but from all your comments it sounds like that might not be worthwhile.

[Mork]

I just stumbled across this thread and I am also experiencing this issue.

I updated to VC 2.0.1 as soon as I could after it was released, and my DB isn't upgraded from 1.x as I put a new server in for my VI3 infrastructure.

I'd love to see this issue resolved...

[Paul_N]

Similar issue here: ticket created in August was closed citing the fix would be in VC 2.0.1. However I'm still having the problem, even with ESX 3.0.1 servers. In my new ticket they suggested the problem was due to time clock irregularities, sure enough some of my servers were in the wrong time zones. But a week later the problem is there, massive dropouts.

[northwest]

I have had an open SR on this issue since August. Went to 2.0.1 mid october, and re-installed a host from scratch for them, and the issue persists. There have been short gaps like it has missed a collection periods data, but historical data just disappears in chunks. Support has not been as quick to respond lately. I hope they are testing with someone else right now.

[stuten]

I've had trouble getting even a status update once a week. I am trying now to see about any sort of escalation path. I hope the level of support on this problem isn't a trend.

[eliot]

I'm seeing chunks of stats dissappearing too. Last week I had a full weeks worth of nice stats, great problem solved i thought. Yesterday all stats before last thursday have gone.

[Noyzyboy]

I also have this problem on at least 3 clients sites. All ESX Servers are 3.0.1 VirtualCenter is 2.0.1. Would be great to see an answer since I needed some performance data to resolve a separate issue the other day and couldn't get it.