cesprov's Accepted Solutions

A separate PSC isn't "required" but it would be desirable.  It sounds like you currently have an embedded PSC in your main site.  Since both are in the same SSO domain, then it's doubtful you currently have two embedded PSCs as two embedded PSCs can't sync with one another.  Which means your DR site may be using the main site's PSC.  If that's the case, if you lose the main site, you may lose your ability to log in to your DR vCenter which kind of defeats the purpose.  While you don't have to, you should look into migrating your embedded PSC at the main site to an external PSC and sync it with the embedded one.  Then change the main site's vCenter to use that new external PSC.  Then install a new PSC at the DR site and sync it with the main site's PSC.  Lastly set the DR vCenter to use the DR site's new PSC.  That way if you lose either site, both will still be able to log in to the local vCenter using the local PSC. I should add that you may not need to migrate the embedded PSC at the main site to an external one first.  As long as you don't install the one at the DR site as an embedded one also, they should be able to sync.  But ideally your environment should have an external PSC on both sides.

I have now pinpointed the exact bug in vCenter that causes this issue and it appears to affect every install of 6.0b through 6.0U2 (I didn't test 6.0 RTM or 6.0a).  On v5.5U2 vCenter databases, the VPX_STAT_COUNTER table has the following indexes: IX_VPX_SC_ENTITY_ID VPX_STAT_COUNTER_M1 VPX_STAT_COUNTER_M2 VPX_STAT_COUNTER_P1 VPX_STAT_COUNTER_U1 After upgrading to 6.0b through 6.0U2, or even on a brand-new 6.0U2 install (i.e. no upgrade), the VPX_STAT_COUNTER table has only the following two indexes: PK_VPX_STAT_COUNTER VPX_STAT_COUNTER_P1 The problem is that the stats_rollup2_proc stored procedure (the SQL Agent task "Past Week stats rollup" calls l_stats_rollup2_proc which in turn calls stats_rollup2_proc) contains a reference to one of these missing indexes: SET @sqlCommand_rt3 = 'INSERT INTO ' + ... + ' VPX_STAT_COUNTER SC  WITH(INDEX(VPX_STAT_COUNTER_M1) ... This causes the execution of the SELECT statement within that INSERT to return: Msg 308, Level 16, State 1, Line 1 Index 'VPX_STAT_COUNTER_M1' on table 'VPX_STAT_COUNTER' (specified in the FROM clause) does not exist. The effect of this is that the SELECT statement within @sqlCommand_rt3's INSERT statement, which is supposed to read VPX_STAT_DEF.ROLLUP_TYPE = 3 counters from HIST2 tables, returns 0 rows due to the above error, meaning VPX_STAT_DEF.ROLLUP_TYPE = 3" counters from HIST2 tables are never rolled up into HIST3 tables because @sqlCommand_rt3 isn't SELECTing any data.  To make matters worse, the error handling in this stored procedure is not catching this error, or at least not reporting it up to the SQL Agent so history shows this task as running successfully when it clearly is not. The answer that VMware support seems to be implying is that the VPX_STAT_COUNTER indexes are erroneously missing from 6.0 installs and should be there.  Either that or those indexes were removed on purpose and someone forgot to update the stored procedures to use one of the remaining indexes (my testing indicates that changing the stored procedure to use VPX_STAT_COUNTER_P1 instead may also fix the issue).  Either way, based on my testing, this affects every 6.0 vCenter install, upgrade or new install doesn't matter.  Most people probably don't notice the issue as most .latest counters don't seem to be exposed through the GUIs.  Unless you are specifically looking for .latest counters in interval 7200 using PowerCLI, or have a third party program that looks for them as was my case, you probably don't even notice you're missing this performance data from the 7200 interval and beyond. The fix is simple: recreate the missing indexes on the VPX_STAT_COUNTER table and the .latest counters then roll up to 7200/HIST3 properly again the next time the task executes.  By default the Weekly and Monthly tasks run at 2:15a so you need to wait until day 2 to see the .latest counters appear for the 86400/HIST4 interval.

Despite being told that there is no workaround and the only solution was to downgrade to 5.5U2 when I opened an SR with VMware, I found out through other channels that there is a workaround script that appears to change the handling of CPU interrupts from automatic to manual, which is supposedly the cause of this issue.  Why VMware is handing that script out to some people and not others I don't know, hopefully it was just the tech that worked my case not having knowledge of said workaround script at the time. EDIT:  I should add that since applying that script to our hosts, we haven't seen the issue reoccur yet when I had three crashes in the first week.  /knockonwood