I have just rolled out vCloud 5.5 and vSphere 5.5 and have struck an issue that re-occurs every 13 days after a physical host reboot.
When trying to connect to a VM console using either [vCD/Web Client/c# Client] I get the following errors:
1. Blank Screen
2. Occasionally MKS malformed response from server
3. Unable to connect to the MKS: Connection terminated by server
I have tried Migrating the VM to another host made no difference, restarting the management agents on the host made no difference, Rebooting the host
and migrating the VM back to host fixed the issue. Suspect a bug in vSphere 5.5 as all [3] host had identical problems. So far I have struck this issue twice and
my servers have been up for 7 days and expect the 3rd occurrence to occur 13/11/2013 ;(
Issue started 16/10 08:30am [first noticed]
Server uptime all [3] hosts 13 days
ESXi build 5.5.0 1331820
Issue re-occurred 29/10 15:00 [second time]
Server uptime all [3] hosts 13 days
ESXi build 5.5.0 1331820
Below is Syslog Server Capture when attempting a VM console connection.
<166>2013-10-15T23:08:39.722Z fcsesx01.fred.local Hostd: -->
<166>2013-10-15T23:08:39.722Z fcsesx01.fred.local Hostd: [3A160B70 verbose 'Hostsvc.ResourcePool pool18'] Added child 26 to pool
<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [3A8D2B70 info 'Libs'] CnxAuthdConnect: Returning false because CnxAuthdProtoReadResponse2 failed
<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [3A8D2B70 info 'Libs'] CnxConnectAuthd: Returning false because CnxAuthdConnect failed
<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [3A8D2B70 info 'Libs'] Cnx_Connect: Returning false because CnxConnectAuthd failed
<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [3A8D2B70 info 'Libs'] Cnx_Connect: Error message: Connection terminated by server
<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] Foundry_[Create|Open]Ex failed: Error: (3008) Cannot connect to the virtual machine
<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] Failed to load virtual machine.
<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] Failed to load VM N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)
<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] Marking VirtualMachine invalid
<166>2013-10-15T23:08:39.724Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] State Transition (VM_STATE_INITIALIZING -> VM_STATE_INVALID_CONFIG)
<166>2013-10-15T23:08:39.724Z fcsesx01.fred.local Hostd: [FFE40B70 verbose 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] Time to load virtual machine: 51 (msecs)
<166>2013-10-15T23:08:39.724Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc'] Loaded virtual machine: /vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx
<166>2013-10-15T23:08:39.724Z fcsesx01.fred.local Hostd: -->
Any help would be appreciated.
I have also logged the call with HP as we have software support with them and they mentioned that a number of sites have similar issues and a call has been place with VMware for further investigation
Rgds
Mark Askham
Senior Infrastructure Architect
Same issue on all G8 bl460c.
We've already updated the lates firmware but same problem:
Locked files on LUN (logs can't be downloaded while vm is running), backup is blocked, KVM doesn't work.
I also upgraded my OA, Flex-10 and blade firmware before installing ESX 5.5 fresh on all the blades.
Same...running 2013.09.08
Indeed the ams service is the culprit. A quick restart of the service temporarily relieves the problem. We are running on BL460 Gen8 as well.
We are facing the same issue with BL460c Gen8
Any new word from VMware or HP?
It appears there's a patch for the issue VMware KB: VMware ESXi 5.5, Patch ESXi550-201312401-BG: Updates esx-base .
Can anyone confirm that this patch resolves the issue? The issue listed in the patch summary seems similar, but not exact:
PR 1119481: Attempts to access the virtual machine console from vCenter Server might fail with error message similar to the following:
Unable to connect to the MKS: The operation is not allowed in the current state
This issue occurs after you perform any one of the following actions:
When you attempt to access the virtual machine console in the vSphere Web Client, a blank screen might be displayed and as a result you are unable to access the console. Attempts to attach a remote device to the virtual machine might also fail.
Hi! Stephenbarry,
I have ESXi 5.5 with build 5.5.0 1474528, installed with Patch "ESXi550-201312401-BG". The HP firmware is "HP_Service_Pack_for_Proliant_2013.09.0-0_744345-001_spp_2013.09.0-SPP2013090.2013_0830.30.
Today, I encountered the same issue reported in this thread. Stopping or restarting the HP ams service is the only workaround I see as of now.
HariVen
I patched all my servers yesterday, won't know whether it has worked until the 13 days are up - I will update on here when I know if this has been successful
VMware released a patch for this issue. See VMware KB: After reloading a virtual machine console access fails with the error: Unable to connect ...
I don't know is this is the solution. I will test the patch today.
HP did include the Agentless Management within their Gen8
Series so that would be unique alright as this is a HP image rather than a
VMware image. It has been known to be a problematic agent.
The hp-ams package could be removed for a permanent fix.
The method and issues associated with doing this are outlined in this blog:
HP Proliant Gen8 Agentless Management floods ESXi and vCenter Logs | Virten.net
Patched all my servers today. Will report back in a few weeks.
HP did include the Agentless Management within their Gen8
Series so that would be unique alright as this is a HP image rather than a
VMware image. It has been known to be a problematic agent.
The hp-ams package could be removed for a permanent fix.
The method and issues associated with doing this are outlined in this blog:
HP Proliant Gen8 Agentless Management floods ESXi and vCenter Logs | Virten.net
Just got the connection errors still even after patching the server with the latest round of patches that someone posted earlier.
Hi,
Anyone tried and see if the updated HP Agentless Management Offline Bundle for 5.5 solves the problem? This bundle was released 2 oct 2013 and the latest HP ESXi iso was released Sept 2013, so it should contain updated agents.
Fixes:
Hi.
I tried update hp-ams-550.9.4.0-2 file. 3 weeks ago.
But same problem.
So now. i restart every 2 weeks hp-ams.
Yeah, I installed the same package now along with the VMware patches.
All hp-ams services are reporting 550.9.4.0-29.1198611, including the upgraded gen8 server.
I guess restart or disable the service is the way to go at this stage.
Problem was back for me this morning as well. Have referred it back to HP and they have asked for new sets of log files etc, back to rebooting every 12 days.
Hi, we're using the HP image as well. Running into the same issues, can't find a hp-ams.sh though.
/opt/hp/hpssacli/bin # find / -iname hp-ams.sh
nothing found
/opt/hp/hpssacli/bin # find / -iname memscrubd
/etc/init.d/memscrubd
/usr/lib/vmware/memscrub/bin/memscrubd
find does work
/opt/hp/hpssacli/bin # esxcli software vib list
Name Version Vendor Acceptance Level Install Date
----------------------------- ------------------------------------- --------------- ---------------- ------------
elxnet 10.0.575.7-1OEM.550.0.0.1198611 Emulex VMwareCertified 2013-10-23
ima-be2iscsi 4.6.261.0-1OEM.550.0.0.1198611 Emulex VMwareCertified 2013-10-23
lpfc 10.0.575.8-1OEM.550.0.0.1198611 Emulex VMwareCertified 2013-10-23
scsi-be2iscsi 4.6.261.0-1OEM.550.0.0.1198611 Emulex VMwareCertified 2013-10-23
char-hpilo 550.9.0.0.9-1OEM.550.0.0.1198611 Hewlett-Packard PartnerSupported 2013-10-23
hp-build 5.70.38-1198611 Hewlett-Packard PartnerSupported 2013-10-23
hpbootcfg 5.5.0.01-01.02.10.1198611 Hewlett-Packard PartnerSupported 2013-10-23
hpnmi 550.2.2.26-1198610 Hewlett-Packard PartnerSupported 2013-10-23
hponcfg 5.5.0.04-00.11.13.1198611 Hewlett-Packard PartnerSupported 2013-10-23
hpssacli 1.50.4.0-5.5.0.1198611 Hewlett-Packard PartnerSupported 2013-10-23
hptestevent 5.5.0.01-00.00.6.1198611 Hewlett-Packard PartnerSupported 2013-10-23
scsi-hpsa 5.5.0.50-1OEM.550.0.0.1198611 Hewlett-Packard VMwareCertified 2013-10-23
scsi-hpvsa 5.5.0-68OEM.550.0.0.1198611 Hewlett-Packard PartnerSupported 2013-10-23
net-igb 4.2.16.8-1OEM.550.0.0.1198611 Intel VMwareCertified 2013-10-23
net-ixgbe 3.15.1.8-1OEM.550.0.0.1198611 Intel VMwareCertified 2013-10-23
scsi-mpt2sas 15.10.05.00.1vmw-1OEM.550.0.0.1198610 LSI VMwareCertified 2013-10-23
ima-qla4xxx 500.2.01.31-1vmw.0.3.040121 QLogic VMwareCertified 2013-10-23
net-qlcnic 5.5.164-1OEM.550.0.0.1198611 QLogic VMwareCertified 2013-10-23
qlnativefc 1.1.7.0-1OEM.550.0.0.1198610 QLogic VMwareCertified 2013-10-23
scsi-qla4xxx 634.55.20.0-1OEM.550.0.0.1198610 QLogic VMwareCertified 2013-10-23
ata-pata-amd 0.3.10-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ata-pata-atiixp 0.4.6-4vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ata-pata-cmd64x 0.2.5-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ata-pata-hpt3x2n 0.3.4-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ata-pata-pdc2027x 1.0-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ata-pata-serverworks 0.4.3-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ata-pata-sil680 0.4.8-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ata-pata-via 0.3.3-2vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
block-cciss 3.6.14-10vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ehci-ehci-hcd 1.0-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
esx-base 5.5.0-0.0.1331820 VMware VMwareCertified 2013-10-23
esx-dvfilter-generic-fastpath 5.5.0-0.0.1331820 VMware VMwareCertified 2013-10-23
esx-tboot 5.5.0-0.0.1331820 VMware VMwareCertified 2013-10-23
esx-xlibs 5.5.0-0.0.1331820 VMware VMwareCertified 2013-10-23
esx-xserver 5.5.0-0.0.1331820 VMware VMwareCertified 2013-10-23
ipmi-ipmi-devintf 39.1-4vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ipmi-ipmi-msghandler 39.1-4vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ipmi-ipmi-si-drv 39.1-4vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
lsi-mr3 0.255.03.01-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
lsi-msgpt3 00.255.03.03-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
misc-cnic-register 1.72.1.v50.1i-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
misc-drivers 5.5.0-0.0.1331820 VMware VMwareCertified 2013-10-23
mtip32xx-native 3.3.4-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-bnx2 2.2.3d.v55.2-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-bnx2x 1.72.56.v55.2-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-cnic 1.72.52.v55.1-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-e1000 8.0.3.1-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-e1000e 1.1.2-4vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-enic 1.4.2.15a-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-forcedeth 0.61-2vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-mlx4-core 1.9.7.0-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-mlx4-en 1.9.7.0-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-nx-nic 5.5.634-3vmw.550.0.0.1198611 VMware VMwareCertified 2013-10-23
net-tg3 3.123c.v55.5-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
net-vmxnet3 1.1.3.0-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
ohci-usb-ohci 1.0-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
rste 2.0.2.0088-4vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
sata-ahci 3.0-17vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
sata-ata-piix 2.12-9vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
sata-sata-nv 3.5-4vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
sata-sata-promise 2.12-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
sata-sata-sil24 1.1-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
sata-sata-sil 2.3-4vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
sata-sata-svw 2.3-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-aacraid 1.1.5.1-9vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-adp94xx 1.0.8.12-6vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-aic79xx 3.1-5vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-bnx2fc 1.72.53.v55.1-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-bnx2i 2.72.11.v55.4-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-fnic 1.5.0.4-1vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-ips 7.12.05-4vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-megaraid-mbox 2.20.5.1-6vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-megaraid-sas 5.34-9vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-megaraid2 2.00.4-9vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-mptsas 4.23.01.00-9vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-mptspi 4.23.01.00-9vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
scsi-qla2xxx 902.k1.1-9vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
uhci-usb-uhci 1.0-3vmw.550.0.0.1331820 VMware VMwareCertified 2013-10-23
tools-light 5.5.0-0.0.1331820 VMware VMwareCertified 2013-10-23
/opt/hp/hpssacli/bin # ls /etc/init.d/
DCUI clomd hp-oem.init lbtd netlogond rhttpproxy sfcbd-watchdog storageRM vmfstraced vsantraced
ESXShell cmmdsd hpilo.init lsassd nscd sdrsInjector slpd swapobjd vobd vsanvpd
SSH dcbd iked lwiod ntpd sensord smartd usbarbitrator vprobed wsman
cdp hostd lacp memscrubd osfsd sfcbd snmpd vmamqpd vpxa xorg
Any ideas? This is quite an annoying issue.