VMware Cloud Community
maskham
Contributor
Contributor

vSphere 5.5 VM console access errors / MKS connection terminated by server & MKS malformed repsonse from server

I have just rolled out vCloud 5.5 and vSphere 5.5 and have struck an issue that re-occurs every 13 days after a physical host reboot.

When trying to connect to a VM console using either [vCD/Web Client/c# Client] I get the following errors:

1. Blank Screen

2. Occasionally MKS malformed response from server

3. Unable to connect to the MKS: Connection terminated by server

I have tried Migrating the VM to another host made no difference, restarting the management agents on the host made no difference, Rebooting the host

and migrating the VM back to host fixed the issue. Suspect a bug in vSphere 5.5 as all [3] host had identical problems. So far I have struck this issue twice and

my servers have been up for 7 days and expect the 3rd occurrence to occur 13/11/2013 ;(

Issue started 16/10 08:30am [first noticed]

Server uptime all [3] hosts 13 days

ESXi build 5.5.0 1331820


Issue re-occurred 29/10 15:00 [second time]

Server uptime all [3] hosts 13 days

ESXi build 5.5.0 1331820

 

Below is Syslog Server Capture when attempting a VM console connection.


<166>2013-10-15T23:08:39.722Z fcsesx01.fred.local Hostd: -->

<166>2013-10-15T23:08:39.722Z fcsesx01.fred.local Hostd: [3A160B70 verbose 'Hostsvc.ResourcePool pool18'] Added child 26 to pool

<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [3A8D2B70 info 'Libs'] CnxAuthdConnect: Returning false because CnxAuthdProtoReadResponse2 failed

<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [3A8D2B70 info 'Libs'] CnxConnectAuthd: Returning false because CnxAuthdConnect failed

<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [3A8D2B70 info 'Libs'] Cnx_Connect: Returning false because CnxConnectAuthd failed

<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [3A8D2B70 info 'Libs'] Cnx_Connect: Error message: Connection terminated by server

<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] Foundry_[Create|Open]Ex failed: Error: (3008) Cannot connect to the virtual machine

<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] Failed to load virtual machine.

<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] Failed to load VM N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)

<166>2013-10-15T23:08:39.723Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] Marking VirtualMachine invalid

<166>2013-10-15T23:08:39.724Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] State Transition (VM_STATE_INITIALIZING -> VM_STATE_INVALID_CONFIG)

<166>2013-10-15T23:08:39.724Z fcsesx01.fred.local Hostd: [FFE40B70 verbose 'Vmsvc.vm:/vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx'] Time to load virtual machine: 51 (msecs)

<166>2013-10-15T23:08:39.724Z fcsesx01.fred.local Hostd: [FFE40B70 info 'Vmsvc'] Loaded virtual machine: /vmfs/volumes/5248c6f0-6a442ffe-c35f-0017a47724d8/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980)/diskload (a33d96fa-47c9-4135-b05e-fd39ce56d980).vmx

<166>2013-10-15T23:08:39.724Z fcsesx01.fred.local Hostd: -->


Any help would be appreciated.

I have also logged the call with HP as we have software support with them and they mentioned that a number of sites have similar issues and a call has been place with VMware for further investigation

Rgds

Mark Askham

Senior Infrastructure Architect

76 Replies
MoeSzyslak
Contributor
Contributor

Same issue on all G8 bl460c.

We've already updated the lates firmware but same problem:

Locked files on LUN (logs can't be downloaded while vm is running), backup is blocked, KVM doesn't work.

Reply
0 Kudos
abaack
Contributor
Contributor

I also upgraded my OA, Flex-10 and blade firmware before installing ESX 5.5 fresh on all the blades.

Reply
0 Kudos
joeshmo
Contributor
Contributor

Same...running 2013.09.08

Reply
0 Kudos
MR-Z
VMware Employee
VMware Employee

Indeed the ams service is the culprit. A quick restart of the service temporarily relieves the problem. We are running on BL460 Gen8 as well.

Reply
0 Kudos
raj459
Contributor
Contributor

We are facing the same issue with BL460c Gen8

Reply
0 Kudos
abaack
Contributor
Contributor

Any new word from VMware or HP?

Reply
0 Kudos
joshswain
Contributor
Contributor

Reply
0 Kudos
stephenrbarry
Enthusiast
Enthusiast

Can anyone confirm that this patch resolves the issue?  The issue listed in the patch summary seems similar, but not exact:

PR 1119481: Attempts to access the virtual machine console from vCenter Server might fail with error message similar to the following:

Unable to connect to the MKS: The operation is not allowed in the current state

This issue occurs after you perform any one of the following actions:

  • You reload .vmx file without removing the virtual machine from the inventory.
  • You restart the hostd service when the virtual machine is running.

When you attempt to access the virtual machine console in the vSphere Web Client, a blank screen might be displayed and as a result you are unable to access the console. Attempts to attach a remote device to the virtual machine might also fail.

Reply
0 Kudos
Hulk007
Contributor
Contributor

Hi! Stephenbarry,

I have ESXi 5.5 with build 5.5.0 1474528, installed with Patch "ESXi550-201312401-BG". The HP firmware is "HP_Service_Pack_for_Proliant_2013.09.0-0_744345-001_spp_2013.09.0-SPP2013090.2013_0830.30.
Today, I encountered the same issue reported in this thread. Stopping or restarting the HP ams service is the only workaround I see as of now.

HariVen

Reply
0 Kudos
AndyRawcs
Contributor
Contributor

I patched all my servers yesterday, won't know whether it has worked until the 13 days are up - I will update on here when I know if this has been successful

Reply
0 Kudos
McKimmi
Contributor
Contributor

VMware released a patch for this issue. See VMware KB: After reloading a virtual machine console access fails with the error: Unable to connect ...

I don't know is this is the solution. I will test the patch today.

Reply
0 Kudos
McKimmi
Contributor
Contributor

HP did include the Agentless Management within their Gen8
Series so that would be unique alright as this is a HP image rather than a
VMware image. It has been known to be a problematic agent.

    

The hp-ams package could be removed for a permanent fix.
The method and issues associated with doing this are outlined in this blog:

HP Proliant Gen8 Agentless Management floods ESXi and vCenter Logs | Virten.net

Reply
0 Kudos
themac201110141
Contributor
Contributor

Patched all my servers today. Will report back in a few weeks.

Reply
0 Kudos
McKimmi
Contributor
Contributor

HP did include the Agentless Management within their Gen8
Series so that would be unique alright as this is a HP image rather than a
VMware image. It has been known to be a problematic agent.

    

The hp-ams package could be removed for a permanent fix.
The method and issues associated with doing this are outlined in this blog:

HP Proliant Gen8 Agentless Management floods ESXi and vCenter Logs | Virten.net

Reply
0 Kudos
abaack
Contributor
Contributor

Just got the connection errors still even after patching the server with the latest round of patches that someone posted earlier.

Reply
0 Kudos
BigBjorn
Enthusiast
Enthusiast

Hi,

Anyone tried and see if the updated HP Agentless Management Offline Bundle for 5.5 solves the problem? This bundle was released 2 oct 2013 and the latest HP ESXi iso was released Sept 2013, so it should contain updated agents.

See HP Support site

Fixes:

  • Fixed AMS excessive vCenter Logging issue
  • Fixed AMS empty and malformed MAC address in cpqNicIfLogMapMACAddress mib
Reply
0 Kudos
Minc
Contributor
Contributor

Hi.

I tried update hp-ams-550.9.4.0-2 file. 3 weeks ago.

But same problem.

So now. i restart every 2 weeks hp-ams.

Reply
0 Kudos
BigBjorn
Enthusiast
Enthusiast

Yeah, I installed the same package now along with the VMware patches.


All hp-ams services are reporting 550.9.4.0-29.1198611, including the upgraded gen8 server.

I guess restart or disable the service is the way to go at this stage.

Reply
0 Kudos
AndyRawcs
Contributor
Contributor

Problem was back for me this morning as well. Have referred it back to HP and they have asked for new sets of log files etc, back to rebooting every 12 days.

Reply
0 Kudos
freaky2000
Contributor
Contributor

Hi, we're using the HP image as well. Running into the same issues, can't find a hp-ams.sh though.

/opt/hp/hpssacli/bin # find / -iname hp-ams.sh

nothing found

/opt/hp/hpssacli/bin # find / -iname memscrubd

/etc/init.d/memscrubd

/usr/lib/vmware/memscrub/bin/memscrubd

find does work

/opt/hp/hpssacli/bin # esxcli software vib list

Name                           Version                                Vendor           Acceptance Level  Install Date

-----------------------------  -------------------------------------  ---------------  ----------------  ------------

elxnet                         10.0.575.7-1OEM.550.0.0.1198611        Emulex           VMwareCertified   2013-10-23 

ima-be2iscsi                   4.6.261.0-1OEM.550.0.0.1198611         Emulex           VMwareCertified   2013-10-23 

lpfc                           10.0.575.8-1OEM.550.0.0.1198611        Emulex           VMwareCertified   2013-10-23 

scsi-be2iscsi                  4.6.261.0-1OEM.550.0.0.1198611         Emulex           VMwareCertified   2013-10-23 

char-hpilo                     550.9.0.0.9-1OEM.550.0.0.1198611       Hewlett-Packard  PartnerSupported  2013-10-23 

hp-build                       5.70.38-1198611                        Hewlett-Packard  PartnerSupported  2013-10-23 

hpbootcfg                      5.5.0.01-01.02.10.1198611              Hewlett-Packard  PartnerSupported  2013-10-23 

hpnmi                          550.2.2.26-1198610                     Hewlett-Packard  PartnerSupported  2013-10-23 

hponcfg                        5.5.0.04-00.11.13.1198611              Hewlett-Packard  PartnerSupported  2013-10-23 

hpssacli                       1.50.4.0-5.5.0.1198611                 Hewlett-Packard  PartnerSupported  2013-10-23 

hptestevent                    5.5.0.01-00.00.6.1198611               Hewlett-Packard  PartnerSupported  2013-10-23 

scsi-hpsa                      5.5.0.50-1OEM.550.0.0.1198611          Hewlett-Packard  VMwareCertified   2013-10-23 

scsi-hpvsa                     5.5.0-68OEM.550.0.0.1198611            Hewlett-Packard  PartnerSupported  2013-10-23 

net-igb                        4.2.16.8-1OEM.550.0.0.1198611          Intel            VMwareCertified   2013-10-23 

net-ixgbe                      3.15.1.8-1OEM.550.0.0.1198611          Intel            VMwareCertified   2013-10-23 

scsi-mpt2sas                   15.10.05.00.1vmw-1OEM.550.0.0.1198610  LSI              VMwareCertified   2013-10-23 

ima-qla4xxx                    500.2.01.31-1vmw.0.3.040121            QLogic           VMwareCertified   2013-10-23 

net-qlcnic                     5.5.164-1OEM.550.0.0.1198611           QLogic           VMwareCertified   2013-10-23 

qlnativefc                     1.1.7.0-1OEM.550.0.0.1198610           QLogic           VMwareCertified   2013-10-23 

scsi-qla4xxx                   634.55.20.0-1OEM.550.0.0.1198610       QLogic           VMwareCertified   2013-10-23 

ata-pata-amd                   0.3.10-3vmw.550.0.0.1331820            VMware           VMwareCertified   2013-10-23 

ata-pata-atiixp                0.4.6-4vmw.550.0.0.1331820             VMware           VMwareCertified   2013-10-23 

ata-pata-cmd64x                0.2.5-3vmw.550.0.0.1331820             VMware           VMwareCertified   2013-10-23 

ata-pata-hpt3x2n               0.3.4-3vmw.550.0.0.1331820             VMware           VMwareCertified   2013-10-23 

ata-pata-pdc2027x              1.0-3vmw.550.0.0.1331820               VMware           VMwareCertified   2013-10-23 

ata-pata-serverworks           0.4.3-3vmw.550.0.0.1331820             VMware           VMwareCertified   2013-10-23 

ata-pata-sil680                0.4.8-3vmw.550.0.0.1331820             VMware           VMwareCertified   2013-10-23 

ata-pata-via                   0.3.3-2vmw.550.0.0.1331820             VMware           VMwareCertified   2013-10-23 

block-cciss                    3.6.14-10vmw.550.0.0.1331820           VMware           VMwareCertified   2013-10-23 

ehci-ehci-hcd                  1.0-3vmw.550.0.0.1331820               VMware           VMwareCertified   2013-10-23 

esx-base                       5.5.0-0.0.1331820                      VMware           VMwareCertified   2013-10-23 

esx-dvfilter-generic-fastpath  5.5.0-0.0.1331820                      VMware           VMwareCertified   2013-10-23 

esx-tboot                      5.5.0-0.0.1331820                      VMware           VMwareCertified   2013-10-23 

esx-xlibs                      5.5.0-0.0.1331820                      VMware           VMwareCertified   2013-10-23 

esx-xserver                    5.5.0-0.0.1331820                      VMware           VMwareCertified   2013-10-23 

ipmi-ipmi-devintf              39.1-4vmw.550.0.0.1331820              VMware           VMwareCertified   2013-10-23 

ipmi-ipmi-msghandler           39.1-4vmw.550.0.0.1331820              VMware           VMwareCertified   2013-10-23 

ipmi-ipmi-si-drv               39.1-4vmw.550.0.0.1331820              VMware           VMwareCertified   2013-10-23 

lsi-mr3                        0.255.03.01-1vmw.550.0.0.1331820       VMware           VMwareCertified   2013-10-23 

lsi-msgpt3                     00.255.03.03-1vmw.550.0.0.1331820      VMware           VMwareCertified   2013-10-23 

misc-cnic-register             1.72.1.v50.1i-1vmw.550.0.0.1331820     VMware           VMwareCertified   2013-10-23 

misc-drivers                   5.5.0-0.0.1331820                      VMware           VMwareCertified   2013-10-23 

mtip32xx-native                3.3.4-1vmw.550.0.0.1331820             VMware           VMwareCertified   2013-10-23 

net-bnx2                       2.2.3d.v55.2-1vmw.550.0.0.1331820      VMware           VMwareCertified   2013-10-23 

net-bnx2x                      1.72.56.v55.2-1vmw.550.0.0.1331820     VMware           VMwareCertified   2013-10-23 

net-cnic                       1.72.52.v55.1-1vmw.550.0.0.1331820     VMware           VMwareCertified   2013-10-23 

net-e1000                      8.0.3.1-3vmw.550.0.0.1331820           VMware           VMwareCertified   2013-10-23 

net-e1000e                     1.1.2-4vmw.550.0.0.1331820             VMware           VMwareCertified   2013-10-23 

net-enic                       1.4.2.15a-1vmw.550.0.0.1331820         VMware           VMwareCertified   2013-10-23 

net-forcedeth                  0.61-2vmw.550.0.0.1331820              VMware           VMwareCertified   2013-10-23 

net-mlx4-core                  1.9.7.0-1vmw.550.0.0.1331820           VMware           VMwareCertified   2013-10-23 

net-mlx4-en                    1.9.7.0-1vmw.550.0.0.1331820           VMware           VMwareCertified   2013-10-23 

net-nx-nic                     5.5.634-3vmw.550.0.0.1198611           VMware           VMwareCertified   2013-10-23 

net-tg3                        3.123c.v55.5-1vmw.550.0.0.1331820      VMware           VMwareCertified   2013-10-23 

net-vmxnet3                    1.1.3.0-3vmw.550.0.0.1331820           VMware           VMwareCertified   2013-10-23 

ohci-usb-ohci                  1.0-3vmw.550.0.0.1331820               VMware           VMwareCertified   2013-10-23 

rste                           2.0.2.0088-4vmw.550.0.0.1331820        VMware           VMwareCertified   2013-10-23 

sata-ahci                      3.0-17vmw.550.0.0.1331820              VMware           VMwareCertified   2013-10-23 

sata-ata-piix                  2.12-9vmw.550.0.0.1331820              VMware           VMwareCertified   2013-10-23 

sata-sata-nv                   3.5-4vmw.550.0.0.1331820               VMware           VMwareCertified   2013-10-23 

sata-sata-promise              2.12-3vmw.550.0.0.1331820              VMware           VMwareCertified   2013-10-23 

sata-sata-sil24                1.1-1vmw.550.0.0.1331820               VMware           VMwareCertified   2013-10-23 

sata-sata-sil                  2.3-4vmw.550.0.0.1331820               VMware           VMwareCertified   2013-10-23 

sata-sata-svw                  2.3-3vmw.550.0.0.1331820               VMware           VMwareCertified   2013-10-23 

scsi-aacraid                   1.1.5.1-9vmw.550.0.0.1331820           VMware           VMwareCertified   2013-10-23 

scsi-adp94xx                   1.0.8.12-6vmw.550.0.0.1331820          VMware           VMwareCertified   2013-10-23 

scsi-aic79xx                   3.1-5vmw.550.0.0.1331820               VMware           VMwareCertified   2013-10-23 

scsi-bnx2fc                    1.72.53.v55.1-1vmw.550.0.0.1331820     VMware           VMwareCertified   2013-10-23 

scsi-bnx2i                     2.72.11.v55.4-1vmw.550.0.0.1331820     VMware           VMwareCertified   2013-10-23 

scsi-fnic                      1.5.0.4-1vmw.550.0.0.1331820           VMware           VMwareCertified   2013-10-23 

scsi-ips                       7.12.05-4vmw.550.0.0.1331820           VMware           VMwareCertified   2013-10-23 

scsi-megaraid-mbox             2.20.5.1-6vmw.550.0.0.1331820          VMware           VMwareCertified   2013-10-23 

scsi-megaraid-sas              5.34-9vmw.550.0.0.1331820              VMware           VMwareCertified   2013-10-23 

scsi-megaraid2                 2.00.4-9vmw.550.0.0.1331820            VMware           VMwareCertified   2013-10-23 

scsi-mptsas                    4.23.01.00-9vmw.550.0.0.1331820        VMware           VMwareCertified   2013-10-23 

scsi-mptspi                    4.23.01.00-9vmw.550.0.0.1331820        VMware           VMwareCertified   2013-10-23 

scsi-qla2xxx                   902.k1.1-9vmw.550.0.0.1331820          VMware           VMwareCertified   2013-10-23 

uhci-usb-uhci                  1.0-3vmw.550.0.0.1331820               VMware           VMwareCertified   2013-10-23 

tools-light                    5.5.0-0.0.1331820                      VMware           VMwareCertified   2013-10-23 

/opt/hp/hpssacli/bin # ls /etc/init.d/

DCUI            clomd           hp-oem.init     lbtd            netlogond       rhttpproxy      sfcbd-watchdog  storageRM       vmfstraced      vsantraced

ESXShell        cmmdsd          hpilo.init      lsassd          nscd            sdrsInjector    slpd            swapobjd        vobd            vsanvpd

SSH             dcbd            iked            lwiod           ntpd            sensord         smartd          usbarbitrator   vprobed         wsman

cdp             hostd           lacp            memscrubd       osfsd           sfcbd           snmpd           vmamqpd         vpxa            xorg

Any ideas? This is quite an annoying issue.

Reply
0 Kudos