Performance Chart Issue in Virtual Center

[northwest]

Has anyone ever had and fixed the following issue? (or still having it?).

The Past Day chart option shows missing statistics on an hourly basis (off and on). All other charts are fine, have never seen missing data in the realtime charts. See attached image.

I have had this problem since 3.0.0, and have had open SRs since, to no avail. The last SR was closed in Jan, saying it was not economically feasible to pursue the issue, as with a DB restructure in VC 2.5, it was "impossible" to happen any longer.

Yet here I am at VC 2.5/ ESX 3.5, all clean installs, same issue.

Here's hoping....

Chris

[weinstein5]

There might be an issue with your network connectivity - usually drops in the real time statistics is caused by VC losing communication to the ESX hosts briefly so it misses a sampling point - the other perfromance charts do not show it particularly since in you case it is so briefly because the data is sampled over a longer period of time -

[northwest]

That has been investigated, and does not seem to be the issue. The perf data comes via UDP and is not guaranteed, but in the last SR, we ruled out missing data.

[mimitte]

I have exactly the same issue with ESX3.0.1 and VC 2.0.2 with SQL Server 2000

Missing data are only in "Past day" chart. Nothing is missing in other charts (realtime, past week, ...)

The missing statistics are on an hourly basis, but not the same from an ESX server to another :

for example on an ESX server, the miss data is at 15:25 - 16:25 - 17:25 - 19:25 (not all hours are missed) - on another ESX, the miss data is at 15:40 - 16:40 - 19:40

I have 7 ESX servers 3.0.1 : they all have the problem & 1 ESX server 2.5.2 : it doesn't have.

I have looked everywhere in logs ==> nothing

No more ideas ...

[northwest]

Thanks mimitte. At least I'm not alone.

Have you ever opened an SR on the issue?

[mimitte]

No, northwest, I don't.

I'm going to migrate ESX and I thought it might resolve the problem. But when I read your 1st post, I think I will do

[Arnaud_D]

I have the same issue (ESX 3.5, VC 2.5) : missing data for about 1 hour at different periods on differents ESX in the same cluster.

[donikatz]

Sorry to revive such an old thread, but I can't find anything more recent on this issue. We are experiencing the exact same problem with VC 2.5 U4 in a small 10 host, 80 VM environment. Performance data for each host drops for 10 minutes once per hour (or two), always at the same interval but not the exact same time for each machine. Happening with ESX 3.5 and 3.x hosts. As implied by the OP, VC does a statistics collection rollup on an hourly basis, so it would seem logical this could be a related DB issue. Has there been any progress or resolution on this? Thanks

[JWeier]

Hey,

Not sure if you are still having this problem but I was having this problem and it had to do with the VMware VirtualCenter Management Webservices service being stopped on the vCenter server. As soon as I started it the charts/graphs appeared.

Hopefully this helps.

-Jay

[Dshen]

Hi I'm actually still having this problem on vSphere 4.0U1. Similar issues that mimitte saw but the errors are on all historical data but NOT on real-time data. Historical show 30min blocks of N/A data reported. Checked and the VMware VirtualCenter Management Webservices service is started. Any other suggestions would be appreciated. Thanks!

[danm66]

Is your statistics collection level still at 1? check the vpxd logs on the VC for sql errors and check that the rollup jobs on the db are running properly.

[Dshen]

Yes. Statistic collection level is set at 1 and the logs don't show any errors. Rollup Jobs looks to be running smoothly. Can't think of any other reason why this would be missing a collection time. I think it might have skipped a job for rollup schedule appears to coincide with the amount of data period that are lost. Only other thing I can think of would be our Drive that holds the DB is down to just under 6GB left of free space. Would a DB job require that much space to run a job?

UPDATE: HD Space is not the issue with this error with the Database. Cleared up the hard drive but still seeing error.

[danm66]

how regular are the gaps? can you post the latest vpxd-###.log?

[Dshen]

They are random occurances. Can be 2x a day or 1x a day or 1x every 2 days. Here's part of the log where I believe the error occurred (deleting some repeated lines):

Unable to recover from 01000:3621

SQL execution failed: { call load_stats_proc(?, ?, ?, ?, ?,

?) }

Execution elapsed time: 187 ms

Diagnostic data from driver is 23000:1:2601:[Microsoft][SQL

Native Client][SQL Server]Cannot insert duplicate key row in object 'dbo.VPX_HIST_STAT1' with unique index

'PK_VPX_HIST_STAT1'.

Diagnostic data from driver is 01000:2:3621:[Microsoft][SQL

Native Client][SQL Server]The statement has been terminated.

Diagnostic data from driver is 23000:3:2601:[Microsoft][SQL

Native Client][SQL Server]Cannot insert duplicate key row in object 'dbo.VPX_HIST_STAT1' with unique index

Bind parameters:

datatype: 2, size: 4,arraySize: 539

value = 327

datatype: 1, size: 4,arraySize: 539

value = 5299

datatype: 11, size: 0,arraySize: 539

datatype: 1, size: 4,arraySize: 539

value = 327

datatype: 3, size: 21,arraySize: 539

value = 30681189073294515

datatype: 11, size: 0,arraySize: 539

Waiting on channel

session[52daef0f-5c87-0358-d30d-105ebb275691]52aa1f92-54ff-f4b5-1dea-11449ffd13f5

processing object update 0 for filter 0

update for prop info

SetTaskInfo for channel

session[52daef0f-5c87-0358-d30d-105ebb275691]52aa1f92-54ff-f4b5-1dea-11449ffd13f5

VpxdMoHost::CollectRemote database error while flushing stats data: "ODBC

error: () - " is returned when executing SQL statement "{ call load_stats_proc(?, ?, ?, ?, ?, ?) }"

[danm66]

open up the database and see how many rows are in the vpx_hist_stat1 table. Also, how many hosts/vm's?

[Dshen]

hi,

There are 377172 Rows. 4 Datacenters with 1 having a cluster. Total of 7 ESXi 4.0U1 Hosts and 72 Total VMs.

Thanks.

[Dshen]

UPDATE: This issue had disappeared for a couple of weeks but re-appeared. It appears the following issue is the cause:

Diagnostic data from driver is 23000:9:2601:[Microsoft][SQL Native Client][SQL Server]Cannot insert duplicate key row in object 'dbo.VPX_HIST_STAT1' with unique index 'PK_VPX_HIST_STAT1'.

Diagnostic data from driver is 01000:10:3621:[Microsoft][SQL Native Client][SQL Server]The statement has been terminated.

Been instructed by vmsupport to truncate the SQL Table. Hopefully, this is the resolution but may not be the one that everyone is willing to accept. I still do not know what caused the problem to disappear for a period of time only to reappear now.