VMware Cloud Community
7POD
Contributor
Contributor

Problem with VSAN Health Check windows server (Unexpected status code: 503)

Hi Upgrade the vCenter from 6.0.0 - 6.0.1 u1 omn my windows server and now we cant see the health for the VSAN.

The Vsan-health logs are showing.

2015-09-15T08:17:07.845Z CRITICAL vsan-health[MainThread] [VsanHealthServer::UncaughtExcpetionHandler] Traceback (most recent call last):

   File "C:\Program Files\VMware\vCenter Server\vsan-health\VsanHealthServer.py", line 354, in <module>

    Initialize(options=gCmdOptions, remainingOptions=gCmdRemainingOptions)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\VsanHealthServer.py", line 305, in Initialize

    ImportTypesAndManagedObjects()

   File "C:\Program Files\VMware\vCenter Server\vsan-health\VsanHealthServer.py", line 316, in ImportTypesAndManagedObjects

    import pyMoVsan

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\__init__.py", line 38, in <module>

    __import__(name, globals(), locals(), [])

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanVcClusterHealthSystemImpl.py", line 4134, in <module>

    VsanVcClusterHealthSystemImpl("vsan-cluster-health-system")

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanVcClusterHealthSystemImpl.py", line 847, in __init__

    VsanEventUtil.registerHealthAlarms(self.conn.si.content)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanEventUtil.py", line 621, in registerHealthAlarms

    cls.registerAlarm(content, content.rootFolder, cls.eventIds, enable=True)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanEventUtil.py", line 569, in registerAlarm

    unregisterEventIds = cls._getUnRegisterTestId(content, mo, eventIds)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanEventUtil.py", line 542, in _getUnRegisterTestId

    if alarm.info.expression is None or \

   File "C:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\VmomiSupport.py", line 537, in __call__

    return self.f(*args, **kwargs)

   File "C:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\VmomiSupport.py", line 360, in _InvokeAccessor

    return self._stub.InvokeAccessor(self, info)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanPyVmomiProfiler.py", line 95, in InvokeAccessor

    out = self._InvokeAccessor(mo, info)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanPyVmomiProfiler.py", line 103, in _InvokeAccessor

    return self._InvokeMethod(mo, info, (prop,))

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanPyVmomiProfiler.py", line 73, in _InvokeMethod

    obj = ds.Deserialize(resp, info.result)

   File "C:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\SoapAdapter.py", line 749, in Deserialize

    self.parser.ParseFile(response)

   File "C:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\SoapAdapter.py", line 648, in EndElementHandler

    raise TypeError(data)

TypeError: vCenter

2015-09-15T08:18:06.369Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:19:06.378Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:20:06.395Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:21:06.405Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:22:06.453Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:23:06.473Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:24:06.482Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:25:06.489Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:26:06.494Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:27:06.502Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:28:06.512Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:29:06.516Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T08:30:06.522Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

Tags (2)
12 Replies
depping
Leadership
Leadership

Did you install vCenter on different ports?

Reply
0 Kudos
7POD
Contributor
Contributor

No its on the default ports.

Reply
0 Kudos
CHogan
VMware Employee
VMware Employee

This smells like a permissions issue. When you did the upgrade, did you run the MSI from a Window that had admin permission (Run As...) If not, just unistall the MSI, open a Windows as administrator (Run As...), then run the MSI from that same window. Let us know if it works...

More details in the health check 6.0 guide

http://cormachogan.com
Reply
0 Kudos
7POD
Contributor
Contributor

I have now uninstalled the VSAN-health with MSI (Run As) Admin from CMD

After the uninstall renamed the (C:\Program Files\VMware\vCenter Server\vsan-Health) folder.

Restarted the vCenter server.

Installed the MSI with (Run As) admin form CMD

Restarted the vcenter services

and still the same problem. Smiley Sad

when i look at the log time the time on the log is not correct the logs time is T13:08:27 and in Sweden the time is 15:09 ? STRANGE...

.311Z WARNING vsan-health[MainThread] [VsanPyVmomiProfiler::InvokeAccessor] Invoke: mo=alarm-410, info=info

2015-09-15T12:58:29.313Z WARNING vsan-health[MainThread] [VsanPyVmomiProfiler::InvokeAccessor] Invoke: mo=alarm-411, info=info

2015-09-15T12:58:29.523Z CRITICAL vsan-health[MainThread] [VsanHealthServer::UncaughtExcpetionHandler] Traceback (most recent call last):

   File "C:\Program Files\VMware\vCenter Server\vsan-health\VsanHealthServer.py", line 354, in <module>

    Initialize(options=gCmdOptions, remainingOptions=gCmdRemainingOptions)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\VsanHealthServer.py", line 305, in Initialize

    ImportTypesAndManagedObjects()

   File "C:\Program Files\VMware\vCenter Server\vsan-health\VsanHealthServer.py", line 316, in ImportTypesAndManagedObjects

    import pyMoVsan

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\__init__.py", line 38, in <module>

    __import__(name, globals(), locals(), [])

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanVcClusterHealthSystemImpl.py", line 4134, in <module>

    VsanVcClusterHealthSystemImpl("vsan-cluster-health-system")

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanVcClusterHealthSystemImpl.py", line 847, in __init__

    VsanEventUtil.registerHealthAlarms(self.conn.si.content)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanEventUtil.py", line 621, in registerHealthAlarms

    cls.registerAlarm(content, content.rootFolder, cls.eventIds, enable=True)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanEventUtil.py", line 569, in registerAlarm

    unregisterEventIds = cls._getUnRegisterTestId(content, mo, eventIds)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanEventUtil.py", line 542, in _getUnRegisterTestId

    if alarm.info.expression is None or \

   File "C:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\VmomiSupport.py", line 537, in __call__

    return self.f(*args, **kwargs)

   File "C:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\VmomiSupport.py", line 360, in _InvokeAccessor

    return self._stub.InvokeAccessor(self, info)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanPyVmomiProfiler.py", line 95, in InvokeAccessor

    out = self._InvokeAccessor(mo, info)

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanPyVmomiProfiler.py", line 103, in _InvokeAccessor

    return self._InvokeMethod(mo, info, (prop,))

   File "C:\Program Files\VMware\vCenter Server\vsan-health\pyMoVsan\VsanPyVmomiProfiler.py", line 73, in _InvokeMethod

    obj = ds.Deserialize(resp, info.result)

   File "C:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\SoapAdapter.py", line 749, in Deserialize

    self.parser.ParseFile(response)

   File "C:\Program Files\VMware\vCenter Server\python-modules\pyVmomi\SoapAdapter.py", line 648, in EndElementHandler

    raise TypeError(data)

TypeError: vCenter

2015-09-15T12:59:27.510Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T13:00:27.515Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T13:01:27.517Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T13:02:27.520Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T13:03:27.525Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T13:04:27.528Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T13:05:27.532Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T13:06:27.539Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T13:07:27.546Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

2015-09-15T13:08:27.552Z WARNING vsan-health[Thread-1] [VsanPyVmomiProfiler::InvokeMethod] Invoke: mo=ServiceInstance, info=CurrentTime

Reply
0 Kudos
Alimnette
Contributor
Contributor

Hello,


i have the same problem after update vcenter 6.0 to 6.0 U1 with PSC External.

i have uninstall and install with runas but a same issue

Best Regards,

Matthieu

Reply
0 Kudos
niljersweden
Contributor
Contributor

Hi,

I have the same issue, is there any solution on this.?

Unexpected status code: 503

Cannot get the health service instance.

pastedImage_0.png

Br,

Jerry

Reply
0 Kudos
niljersweden
Contributor
Contributor

Reply
0 Kudos
jjewett
Enthusiast
Enthusiast

Same issue here.  VCSA 6.0u1 with VSAN and the 503 message on the Cluster object Monitor tab Virtual SAN > Health. The kb2128353 did not apply, the permissions on the certificates in /etc/vmware-vpx/ssl were already correct.

Reply
0 Kudos
mcannella
Contributor
Contributor

Do you guys have DELL OME Vcenter Plugin Installed?

That is what was causing our issue.  We had to remove the Dell OME Plugin and it started to work.  Vmware is aware of issue and there is an open Bug with Dell and will be fixed in next release.

Reply
0 Kudos
aaronwsmith
Enthusiast
Enthusiast

Thanks much for this tip!  We had this problem with 2 of our VSAN 6.1 enabled vCenters.  Initially the problem was the CA Certificate Replacement issues documented in the following 3 KBs:

Embedded PSC: https://kb.vmware.com/kb/2121689

VSAN: https://kb.vmware.com/kb/2128353

ESX Agent Manager: https://kb.vmware.com/kb/2112577

But we also had Dell OpenManager integrated into both vCenters and were continuing to see the HTTP 503 error.  After unregistering the vCenters from Dell OpenManager Admin portal and restarting the VSAN Health Check service, it began to work again!

Reply
0 Kudos
FirstByte
Enthusiast
Enthusiast

Is a known fault with ESXi 6U1 (ie. VSAN 6.1). Now resolved in ESXi 6 U2 (ie. VSAN 6.2)

Refer to:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=21384...

Hope that helps

Darren

aaronwsmith
Enthusiast
Enthusiast

Thanks for the update Darren!  I'll confirm this resolves the issue for us once we upgrade to 6.0U2 in our VSAN environment.

Reply
0 Kudos