Re: vCenter 6.5 - some issues happening

EGS78 · ‎10-10-2018

Scenario:

- Last Tuesday, I updated vCenter Server Appliance 6.5 to last version available: 6.5.0.22000;

- Since then, my vCenter Server is very slow: Migrations are very slow, put the host into Maintenance Mode is very very slow, etc.

- So, I've rebooted the vCenter Server and...something wrong happened: vCenter Server doesn't work, vCenter Server Appliance doesn't work properly, and so on.

- I've taken some prints, and they're attached on this topic.

What can I do? Re-creating the vCenter Server?

Obs.: vSphere Client 6.5 is too much slow. Maybe it could be happening because of the Flash Plug-in?

That's it.

Thank you

GayathriS · ‎10-11-2018

Functionality Caveats

The vSphere Client is written in an HTML5-based language and frameworks supported by all browsers. However, different browsers have different performance characteristics regarding the HTML5 standard. In particular, performance with Internet Explorer 11 can be slower than with other browsers, because of the rendering engine that Internet Explorer 11 uses. If you experience such issues, try using another supported browser.

VMware vCenter Server 6.5 Update 1 Release Notes

VCSA 6.5 web client very slow

Please mark this as correct if this answers and helps you .

regards

Gayathri

EGS78 · ‎10-11-2018

Hi GayathriS,

Thank you for your reply. So, I am using chrome, last version (I tried using Firefox, but the navigation and usability were awful...it doesn't work properly...).

- Well, I have installed vCenter Appliance 6.5 and I've updated again, but I haven't added hosts yet. I'm testing some things before add hosts...

Best regards.

EGS78 · ‎10-15-2018

Could somebody help me please?

My vCenter still has some errors...I've taken new prints, and they're attached on this topic.

It's very, very, very slow...It doesn't working properly...

Any idea?

Regards.

msripada · ‎10-15-2018

Is there any storage latency on the datastore on which the vcenter 6.5 is residing? Did you try validating the storage latency?

Thanks,

MS

EGS78 · ‎10-15-2018

Hi msripada, how are you doing today?

Thanks for your prompt reply. No, I didn't try it. Actually, I don't know how to do this. Is there any tutorial explaining how can I verify the storage latency? BTW, I feel the vCenter Server is very slow by the time I try to migrate a VM between storages...It doesn't happen when I try to migrate between hosts...

best regards.

msripada · ‎10-15-2018

You can login to the ESXi host using ssh

Use esxtop command and identify the naa id for the datastore and check for DAVG values for the datastore. The below KB can help you.

VMware Knowledge Base

Multiple VM migrations between storages can also put additional load on the underlying storage. How is the Shared storage configured, is it NFS, iscsi or FC?

Thanks,

MS

EGS78 · ‎10-16-2018

msripada,

Thanks for answering. Sorry for my late reply. My internet connection was off this morning. So, the Shared Storage is configured as Fiber Channel.

Well, I've started installing a new vCenter Server for about an hour ago. The print below shows status at the moment:

It's happening on the host 2. When I installed another vCenter on the host 1, this process took a maximum of 15 minutes...I think host 2 is the problem...but I don't know what it is happening with this host. Is there anything that I can see on this host?

Thanks in advance.

msripada · ‎10-16-2018

can you check on host2 vmkernel.log under var/log if there is any storage disconnects or issues identified on ESXi host 2?

Thanks,

MS

EGS78 · ‎10-17-2018

msripada,

So, I just took a look at the vmkernel.log on the host2. I didn't see anything wrong, but, well, I'm not a specialist

This morning, I did a test: I installed a VM (Windows Server 2012 R2) in both Hosts (host 1 and host 2). On the host 1, the installation took about 15 minutes. On the host 2, the installation took almost 2 hours!! I don't understand...

Well, we had an power outage a few days ago. Perhaps these events have occurred after that (I wasn't here in the company when it happened).

Anyway, I attached the vmkernel.log of host2 in this post. If you could please take a look...I really appreciate it.

Thank you so much. You are really helping me a lot.

Bye.

msripada · ‎10-17-2018

All the FCP commands are failing to datastore DSVNX04

2018-10-17T15:26:55.708Z cpu20:33435)ScsiDeviceIO: 2595: Cmd(0x439dc093cb80) 0x2a, CmdSN 0x800e001b from world 373124 to dev "naa.600601604e203400c15ffc51f84ce611" failed H:0x8 D:0x0 P:0x0

2018-10-17T15:26:56.056Z cpu16:35407)HBX: 2802: 'DSVNX04': HB at offset 3346432 - Waiting for timed out HB:

2018-10-17T15:26:56.056Z cpu16:35407) [HB state abcdef02 offset 3346432 gen 111 stampUS 507785538063 uuid 5bbf955c-5631d5c6-8caa-90b11c14b838 jrnl <FB 223200> drv 14.61 lockImpl 4]

2018-10-17T15:27:14.710Z cpu21:412306)ScsiDeviceIO: 2651: Cmd(0x439dcda1e4c0) 0xfe, CmdSN 0x3caed from world 32798 to dev "naa.600601604e203400c15ffc51f84ce611" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2018-10-17T15:27:14.758Z cpu12:32841)HBX: 276: 'DSVNX04': HB at offset 3346432 - Reclaimed heartbeat [Timeout]:

2018-10-17T15:27:14.758Z cpu12:32841) [HB state abcdef02 offset 3346432 gen 111 stampUS 507813589326 uuid 5bbf955c-5631d5c6-8caa-90b11c14b838 jrnl <FB 223200> drv 14.61 lockImpl 4]

2018-10-17T15:27:16.852Z cpu20:33013)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <0/0> sid x010300, did x010600, oxid xffff iotag x908 Time Out Returning Host Busy

2018-10-17T15:27:16.913Z cpu20:33435)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x28 failed <0/0> sid x010300, did x010600, oxid xffff iotag x8dd Local Reason x1d Returning Host Busy

2018-10-17T15:27:33.709Z cpu20:33172)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd xf1 failed <0/0> sid x010300, did x010600, oxid xffff iotag x901 Abort Requested Host Abort Req

2018-10-17T15:27:34.707Z cpu20:33435)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <0/0> sid x010300, did x010600, oxid xffff iotag x930 Abort Requested Host Abort Req

2018-10-17T15:27:34.708Z cpu20:33172)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <0/0> sid x010300, did x010600, oxid xffff iotag x931 Abort Requested Host Abort Req

2018-10-17T15:27:34.708Z cpu14:33435)ScsiDeviceIO: 2595: Cmd(0x43a5cc47b640) 0x2a, CmdSN 0x8000004a from world 404791 to dev "naa.600601604e2034005907beecdc49e611" failed H:0x8 D:0x0 P:0x0

2018-10-17T15:27:34.708Z cpu20:33173)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <0/0> sid x010300, did x010600, oxid xffff iotag x97d Abort Requested Host Abort Req

2018-10-17T15:27:34.709Z cpu20:33173)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <0/0> sid x010300, did x010600, oxid xffff iotag x986 Abort Requested Host Abort Req

2018-10-17T15:27:34.710Z cpu20:33173)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <0/0> sid x010300, did x010600, oxid xffff iotag x9aa Abort Requested Host Abort Req

2018-10-17T15:27:34.710Z cpu14:32821)ScsiDeviceIO: 2595: Cmd(0x43a5ccb037c0) 0x2a, CmdSN 0x8000001d from world 404791 to dev "naa.600601604e2034005907beecdc49e611" failed H:0x8 D:0x0 P:0x0

2018-10-17T15:27:35.134Z cpu22:35411)HBX: 2802: 'DSVNX01': HB at offset 3346432 - Waiting for timed out HB:

2018-10-17T15:27:35.134Z cpu22:35411) [HB state abcdef02 offset 3346432 gen 133 stampUS 507824538058 uuid 5bbf955c-5631d5c6-8caa-90b11c14b838 jrnl <FB 775200> drv 14.61 lockImpl 4]

2018-10-17T15:27:38.851Z cpu20:280261)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <1/1> sid x010300, did x010800, oxid xffff iotag x912 Time Out Returning Host Busy

2018-10-17T15:27:53.710Z cpu21:412306)ScsiDeviceIO: 2651: Cmd(0x439dcc332bc0) 0xfe, CmdSN 0x47543 from world 32798 to dev "naa.600601604e2034005907beecdc49e611" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x1.

2018-10-17T15:27:53.758Z cpu4:32843)HBX: 276: 'DSVNX01': HB at offset 3346432 - Reclaimed heartbeat [Timeout]:

2018-10-17T15:27:53.758Z cpu4:32843) [HB state abcdef02 offset 3346432 gen 133 stampUS 507852589358 uuid 5bbf955c-5631d5c6-8caa-90b11c14b838 jrnl <FB 775200> drv 14.61 lockImpl 4]

2018-10-17T15:28:03.710Z cpu20:35700)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd xf1 failed <0/0> sid x010300, did x010600, oxid xffff iotag x914 Abort Requested Host Abort Req

2018-10-17T15:28:05.760Z cpu20:280260)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <0/0> sid x010300, did x010600, oxid xffff iotag x906 Abort Requested Host Abort Req

2018-10-17T15:28:05.761Z cpu20:280260)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <0/0> sid x010300, did x010600, oxid xffff iotag x94e Abort Requested Host Abort Req

2018-10-17T15:28:05.761Z cpu14:32821)ScsiDeviceIO: 2595: Cmd(0x439dc093a480) 0x2a, CmdSN 0x80000069 from world 404791 to dev "naa.600601604e2034005907beecdc49e611" failed H:0x8 D:0x0 P:0x0

2018-10-17T15:28:06.196Z cpu13:35462)HBX: 2802: 'DSVNX01': HB at offset 3346432 - Waiting for timed out HB:

2018-10-17T15:28:06.196Z cpu13:35462) [HB state abcdef02 offset 3346432 gen 133 stampUS 507854538065 uuid 5bbf955c-5631d5c6-8caa-90b11c14b838 jrnl <FB 775200> drv 14.61 lockImpl 4]

2018-10-17T15:28:08.849Z cpu20:280261)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <1/1> sid x010300, did x010800, oxid xffff iotag x93a Time Out Returning Host Busy

2018-10-17T15:28:09.849Z cpu20:280261)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <1/1> sid x010300, did x010800, oxid xffff iotag x974 Time Out Returning Host Busy

2018-10-17T15:28:13.849Z cpu20:280268)lpfc: lpfc_scsi_cmd_iocb_cmpl:2185: 3:(0):3271: FCP cmd x2a failed <0/3> sid x010300, did x010600, oxid xffff iotag x964 Time Out Returning Host Busy

2018-10-17T15:28:23.710Z cpu21:412306)ScsiDeviceIO: 2651: Cmd(0x439dc0911400) 0xfe, CmdSN 0x4754a from world 32798 to dev "naa.600601604e2034005907beecdc49e611" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.

2018-10-17T15:28:23.714Z cpu13:32842)HBX: 276: 'DSVNX01': HB at offset 3346432 - Reclaimed heartbeat [Timeout]:

2018-10-17T15:28:23.714Z cpu13:32842) [HB state abcdef02 offset 3346432 gen 133 stampUS 507882545286 uuid 5bbf955c-5631d5c6-8caa-90b11c14b838 jrnl <FB 775200> drv 14.61 lockImpl 4]

2018-10-17T15:28:40.440Z cpu5:34554)NMP: nmp_ThrottleLogForDevice:3286: Cmd 0x1a (0x439dc0941b40, 0) to dev "naa.690b11c00d786000185bb8ec1c8db349" on path "vmhba1:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE

2018-10-17T15:28:40.440Z cpu5:34554)ScsiDeviceIO: 2651: Cmd(0x439dc0941b40) 0x1a, CmdSN 0xa4c0 from world 0 to dev "naa.690b11c00d786000185bb8ec1c8db349" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-10-17T15:28:40.462Z cpu5:34554)ScsiDeviceIO: 2651: Cmd(0x439dcd7f3080) 0x1a, CmdSN 0xa4c5 from world 0 to dev "naa.690b11c00d786000185bb8ec1c8db349" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2018-10-17T15:28:40.465Z cpu5:34554)ScsiDeviceIO: 2651: Cmd(0x439dcc32e940) 0x1a, CmdSN 0xa4ca from world 0 to dev "naa.690b11c00d786000185bb8ec1c8db349" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

msripada · ‎10-17-2018

you can try rebooting the esxi and check to update the lpfc drivers

VMware Knowledge Base

EGS78 · ‎10-19-2018

msripada how are you doing today?

Thank you so much for your kindness in help me. I really appreciate that.

I see the version of both ESXi (host 1 and host 2) are different. Yesterday I downloaded a patch to update the host 2 so that it can be similar host 1. I'm gonna try install this patch today afternoon...

Go for it!

Thank you! You are helping me a lot!

Best regards