VMware Cloud Community
lucarobusto_arp
Enthusiast
Enthusiast

Slow boot time after upgrading to ESXI 7 from 6.7 - hang on "mw_satp_alua loaded successfully"

Hi all,

after moving from ESXI 6.7 to 7.0.2 I find very slw Server boot, hanging on "vmw_satp_alua loaded successfully" start message.

Following attached the vmkernel.log file.

I don't have any Microsoft Cluster in my VMs, I could read it's one of the slowing factors.

 

Do you have any idea what could be to reduce the boot performance?

2021-04-22T14:04:11.751Z cpu26:2098301)LVM: 17286: File system '[QNAP VM1 9TB SSD(sata) 1, 5f4e5ddc-6551a6ac-d144-38eaa7318da0]' (LV 5f4e5ddc-3585623c-ba1e-38eaa7318da0) mounted in 'rw' mode.
2021-04-22T14:04:11.777Z cpu26:2098301)NFS: 1342: Invalid volume UUID 5f733e7c-79ef862a-fb8f-1402ec77987c
2021-04-22T14:04:11.824Z cpu26:2098301)LVM: 17286: File system '[QNAP VM2 SSD 9TB 1, 5f733e7c-a02e549c-d824-1402ec77987c]' (LV 5f733e7c-79ef862a-fb8f-1402ec77987c) mounted in 'rw' mode.
2021-04-22T14:04:11.849Z cpu26:2098301)NFS: 1342: Invalid volume UUID 5f733f08-e31b0702-91d0-1402ec77987c
2021-04-22T14:04:11.897Z cpu26:2098301)LVM: 17286: File system '[QNAP VM2 16TB HDD 1, 5f733f08-0cf7805c-3f08-1402ec77987c]' (LV 5f733f08-e31b0702-91d0-1402ec77987c) mounted in 'rw' mode.
2021-04-22T14:04:11.927Z cpu26:2098301)NFS: 1342: Invalid volume UUID 602d44c8-c111e63e-f14c-1402ec779b60
2021-04-22T14:04:11.982Z cpu26:2098301)LVM: 17286: File system '[QNAP VM3 18TB HDD1, 602d44c8-ce9f4e92-b84c-1402ec779b60]' (LV 602d44c8-c111e63e-f14c-1402ec779b60) mounted in 'rw' mode.
2021-04-22T14:04:12.011Z cpu26:2098301)NFS: 1342: Invalid volume UUID 602d4507-f5746cd6-c0c0-1402ec779b60
2021-04-22T14:04:12.064Z cpu26:2098301)CpuSched: 817: user latency of 2098311 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:04:12.065Z cpu26:2098301)LVM: 17286: File system '[QNAP VM3 10TB SSD 1, 602d4507-013b61fe-3c74-1402ec779b60]' (LV 602d4507-f5746cd6-c0c0-1402ec779b60) mounted in 'rw' mode.
2021-04-22T14:04:12.079Z cpu26:2098301)NFS: 1342: Invalid volume UUID 606db675-f3b1d5b8-50f9-38eaa79007d8
2021-04-22T14:04:12.111Z cpu26:2098301)LVM: 17286: File system '[OSDATA-606db676-0aa7fb72-c417-38eaa79007d8, 606db676-0aa7fb72-c417-38eaa79007d8]' (LV 606db675-f3b1d5b8-50f9-38eaa79007d8) mounted in 'rw' mode.
2021-04-22T14:04:53.399Z cpu6:2097802)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device naa.6e843b640b4b3b4de9fed4402da808da repeated 3 times
2021-04-22T14:04:53.399Z cpu6:2097802)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device naa.6e843b6e311bd35dbb1fd428edaefddc repeated 3 times
2021-04-22T14:04:53.399Z cpu6:2097802)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device naa.6e843b61523a9a0dc05dd4b8bd8b92d7 repeated 3 times
2021-04-22T14:04:53.399Z cpu6:2097802)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device naa.6e843b62aa61ca4dcad9d4c40da5e2d7 repeated 2 times
2021-04-22T14:07:42.090Z cpu19:2098301)CpuSched: 817: user latency of 2098412 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:11:14.117Z cpu17:2098301)CpuSched: 817: user latency of 2098512 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:14:40.794Z cpu8:2098301)CpuSched: 817: user latency of 2098612 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:18:12.234Z cpu23:2098301)CpuSched: 817: user latency of 2098712 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:21:44.143Z cpu8:2098301)CpuSched: 817: user latency of 2098812 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:25:16.100Z cpu21:2098301)CpuSched: 817: user latency of 2098912 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:28:48.537Z cpu13:2098301)CpuSched: 817: user latency of 2099012 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:32:14.531Z cpu13:2098301)CpuSched: 817: user latency of 2099113 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:35:41.503Z cpu0:2098301)CpuSched: 817: user latency of 2099213 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:39:09.812Z cpu18:2098301)CpuSched: 817: user latency of 2099313 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:42:33.417Z cpu16:2098301)CpuSched: 817: user latency of 2099413 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:46:08.145Z cpu16:2098301)CpuSched: 817: user latency of 2099513 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:49:43.126Z cpu10:2098301)CpuSched: 817: user latency of 2099613 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:53:08.373Z cpu4:2098301)CpuSched: 817: user latency of 2099713 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T14:56:37.968Z cpu0:2098301)CpuSched: 817: user latency of 2099813 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T15:00:07.643Z cpu20:2098301)CpuSched: 817: user latency of 2099913 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T15:03:36.670Z cpu20:2098301)CpuSched: 817: user latency of 2100013 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T15:07:09.722Z cpu7:2098301)CpuSched: 817: user latency of 2100113 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T15:10:36.066Z cpu8:2098301)CpuSched: 817: user latency of 2100213 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T15:14:08.379Z cpu7:2098301)CpuSched: 817: user latency of 2100313 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T15:17:40.849Z cpu5:2098301)CpuSched: 817: user latency of 2100413 J6AsyncReplayManager 0 changed by 2098301 nativeExecutor -6
2021-04-22T15:21:04.834Z cpu42:2100511)VC: 4710: Device rescan time 35 msec (total number of devices 13)
2021-04-22T15:21:04.834Z cpu42:2100511)VC: 4713: Filesystem probe time 69 msec (devices probed 13 of 13)
2021-04-22T15:21:04.834Z cpu42:2100511)VC: 4715: Refresh open volume time 0 msec
2021-04-22T15:21:04.879Z cpu42:2100511)WARNING: VisorFS: 1091: Attempt to setattr non sticky dir/file from tar mount
2021-04-22T15:21:05.566Z cpu47:2097684)Jumpstart plugin system-storage activated.
2021-04-22T15:21:05.574Z cpu20:2097673)Activating Jumpstart plugin vsan.
2021-04-22T15:21:05.574Z cpu18:2097658)Activating Jumpstart plugin backup-configurations.
2021-04-22T15:21:05.574Z cpu33:2097707)Activating Jumpstart plugin tag-boot-bank.
2021-04-22T15:21:05.575Z cpu32:2097701)Activating Jumpstart plugin crond.
2021-04-22T15:21:05.575Z cpu25:2097698)Activating Jumpstart plugin restore-datafiles.
2021-04-22T15:21:05.579Z cpu1:2097599)Loading module swapobj ...
2021-04-22T15:21:05.612Z cpu32:2097701)Jumpstart plugin crond activated.
2021-04-22T15:21:05.617Z cpu18:2097658)Jumpstart plugin backup-configurations activated.
2021-04-22T15:21:05.623Z cpu25:2097698)Jumpstart plugin restore-datafiles activated.
2021-04-22T15:21:05.684Z cpu1:2097599)Elf: 2060: module swapobj has license VMware
2021-04-22T15:21:05.686Z cpu1:2097599)SwapObj: 63: SwapObj backend registration successful.
2021-04-22T15:21:05.686Z cpu1:2097599)Mod: 4789: Initialization of swapobj succeeded with module ID 86.
2021-04-22T15:21:05.686Z cpu1:2097599)swapobj loaded successfully.
2021-04-22T15:21:05.701Z cpu33:2097707)Jumpstart plugin tag-boot-bank activated.
2021-04-22T15:21:05.817Z cpu20:2097673)Jumpstart plugin vsan activated.
2021-04-22T15:21:05.824Z cpu39:2097693)Activating Jumpstart plugin vmkapps.
2021-04-22T15:21:05.824Z cpu6:2097641)Activating Jumpstart plugin mount-disk-fs.
2021-04-22T15:21:05.927Z cpu1:2097599)VC: 4710: Device rescan time 30 msec (total number of devices 13)
2021-04-22T15:21:05.927Z cpu1:2097599)VC: 4713: Filesystem probe time 68 msec (devices probed 12 of 13)
2021-04-22T15:21:05.927Z cpu1:2097599)VC: 4715: Refresh open volume time 2 msec
2021-04-22T15:21:05.930Z cpu6:2097641)Jumpstart plugin mount-disk-fs activated.

 

-------------

Running sxcli storage core device list I get the following:

[root@localhost:~] esxcli storage core device list
naa.6e843b635343d58d27d2d478dd9a1bd3
Display Name: QNAP iSCSI Disk (naa.6e843b635343d58d27d2d478dd9a1bd3)
Has Settable Display Name: true
Size: 10104448
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.6e843b635343d58d27d2d478dd9a1bd3
Vendor: QNAP
Model: iSCSI Storage
Revision: 4.0
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: supported
Other UIDs: vml.02000000006e843b635343d58d27d2d478dd9a1bd3695343534920
Is Shared Clusterwide: true
Is SAS: false
Is USB: false
Is Boot Device: false
Device Max Queue Depth: 128
No of outstanding IOs with competing worlds: 32
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

naa.6e843b61523a9a0dc05dd4b8bd8b92d7
Display Name: QNAP iSCSI Disk (naa.6e843b61523a9a0dc05dd4b8bd8b92d7)
Has Settable Display Name: true
Size: 10485760
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.6e843b61523a9a0dc05dd4b8bd8b92d7
Vendor: QNAP
Model: iSCSI Storage
Revision: 4.0
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: supported
Other UIDs: vml.02000100006e843b61523a9a0dc05dd4b8bd8b92d7695343534920
Is Shared Clusterwide: true
Is SAS: false
Is USB: false
Is Boot Device: false
Device Max Queue Depth: 128
No of outstanding IOs with competing worlds: 32
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

t10.ATA_____ORTIAL_SSD______________________________0000060629__________
Display Name: Local ATA Disk (t10.ATA_____ORTIAL_SSD______________________________0000060629__________)
Has Settable Display Name: false
Size: 488386
Device Type: Direct-Access
Multipath Plugin: HPP
Devfs Path: /vmfs/devices/disks/t10.ATA_____ORTIAL_SSD______________________________0000060629__________
Vendor: ATA
Model: ORTIAL SSD
Revision: 117K
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: false
Is SSD: true
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: yes
Attached Filters:
VAAI Status: unsupported
Other UIDs: vml.010000000030303030303630363239202020202020202020204f525449414c
Is Shared Clusterwide: false
Is SAS: false
Is USB: false
Is Boot Device: false
Device Max Queue Depth: 31
No of outstanding IOs with competing worlds: 31
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

naa.6e843b62b0eba9dd7b05d4843d92a3de
Display Name: QNAP iSCSI Disk (naa.6e843b62b0eba9dd7b05d4843d92a3de)
Has Settable Display Name: true
Size: 10485760
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.6e843b62b0eba9dd7b05d4843d92a3de
Vendor: QNAP
Model: iSCSI Storage
Revision: 4.0
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: supported
Other UIDs: vml.02000000006e843b62b0eba9dd7b05d4843d92a3de695343534920
Is Shared Clusterwide: true
Is SAS: false
Is USB: false
Is Boot Device: false
Device Max Queue Depth: 128
No of outstanding IOs with competing worlds: 32
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

naa.6e843b6a8bf8ab4dc621d4324db3f4d9
Display Name: QNAP iSCSI Disk (naa.6e843b6a8bf8ab4dc621d4324db3f4d9)
Has Settable Display Name: true
Size: 9961472
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.6e843b6a8bf8ab4dc621d4324db3f4d9
Vendor: QNAP
Model: iSCSI Storage
Revision: 4.0
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: supported
Other UIDs: vml.02000000006e843b6a8bf8ab4dc621d4324db3f4d9695343534920
Is Shared Clusterwide: true
Is SAS: false
Is USB: false
Is Boot Device: false
Device Max Queue Depth: 128
No of outstanding IOs with competing worlds: 32
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

naa.6e843b6e311bd35dbb1fd428edaefddc
Display Name: QNAP iSCSI Disk (naa.6e843b6e311bd35dbb1fd428edaefddc)
Has Settable Display Name: true
Size: 9437184
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.6e843b6e311bd35dbb1fd428edaefddc
Vendor: QNAP
Model: iSCSI Storage
Revision: 4.0
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: supported
Other UIDs: vml.02000200006e843b6e311bd35dbb1fd428edaefddc695343534920
Is Shared Clusterwide: true
Is SAS: false
Is USB: false
Is Boot Device: false
Device Max Queue Depth: 128
No of outstanding IOs with competing worlds: 32
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

naa.6e843b62aa61ca4dcad9d4c40da5e2d7
Display Name: QNAP iSCSI Disk (naa.6e843b62aa61ca4dcad9d4c40da5e2d7)
Has Settable Display Name: true
Size: 16637440
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.6e843b62aa61ca4dcad9d4c40da5e2d7
Vendor: QNAP
Model: iSCSI Storage
Revision: 4.0
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: supported
Other UIDs: vml.02000100006e843b62aa61ca4dcad9d4c40da5e2d7695343534920
Is Shared Clusterwide: true
Is SAS: false
Is USB: false
Is Boot Device: false
Device Max Queue Depth: 128
No of outstanding IOs with competing worlds: 32
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

mpx.vmhba0:C0:T5:L0
Display Name: Local hp CD-ROM (mpx.vmhba0:C0:T5:L0)
Has Settable Display Name: false
Size: 0
Device Type: CD-ROM
Multipath Plugin: HPP
Devfs Path:
Vendor: hp
Model: DVDRAM GUB0N
Revision: UK00
SCSI Level: 5
Is Pseudo: false
Status: off
Is RDM Capable: false
Is Local: true
Is Removable: true
Is SSD: false
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: true
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: unknown
Other UIDs: vml.0005000000766d686261303a353a30
Is Shared Clusterwide: false
Is SAS: false
Is USB: false
Is Boot Device: false
Device Max Queue Depth: 16
No of outstanding IOs with competing worlds: 16
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

naa.6e843b66f1b4efbd2f4cd4c8bdadf5d3
Display Name: QNAP iSCSI Disk (naa.6e843b66f1b4efbd2f4cd4c8bdadf5d3)
Has Settable Display Name: true
Size: 18874368
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.6e843b66f1b4efbd2f4cd4c8bdadf5d3
Vendor: QNAP
Model: iSCSI Storage
Revision: 4.0
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: supported
Other UIDs: vml.02000000006e843b66f1b4efbd2f4cd4c8bdadf5d3695343534920
Is Shared Clusterwide: true
Is SAS: false
Is USB: false
Is Boot Device: false
Device Max Queue Depth: 128
No of outstanding IOs with competing worlds: 32
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

t10.ATA_____ORTIAL_SSD______________________________0000060618__________
Display Name: Local ATA Disk (t10.ATA_____ORTIAL_SSD______________________________0000060618__________)
Has Settable Display Name: false
Size: 488386
Device Type: Direct-Access
Multipath Plugin: HPP
Devfs Path: /vmfs/devices/disks/t10.ATA_____ORTIAL_SSD______________________________0000060618__________
Vendor: ATA
Model: ORTIAL SSD
Revision: 117K
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: false
Is SSD: true
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: yes
Attached Filters:
VAAI Status: unsupported
Other UIDs: vml.010000000030303030303630363138202020202020202020204f525449414c
Is Shared Clusterwide: false
Is SAS: false
Is USB: false
Is Boot Device: true
Device Max Queue Depth: 31
No of outstanding IOs with competing worlds: 31
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

naa.6e843b640b4b3b4de9fed4402da808da
Display Name: QNAP iSCSI Disk (naa.6e843b640b4b3b4de9fed4402da808da)
Has Settable Display Name: true
Size: 10485760
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.6e843b640b4b3b4de9fed4402da808da
Vendor: QNAP
Model: iSCSI Storage
Revision: 4.0
SCSI Level: 5
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: false
Is Removable: false
Is SSD: false
Is VVOL PE: false
Is Offline: false
Is Perennially Reserved: false
Queue Full Sample Size: 0
Queue Full Threshold: 0
Thin Provisioning Status: unknown
Attached Filters:
VAAI Status: supported
Other UIDs: vml.02000100006e843b640b4b3b4de9fed4402da808da695343534920
Is Shared Clusterwide: true
Is SAS: false
Is USB: false
Is Boot Device: false
Device Max Queue Depth: 128
No of outstanding IOs with competing worlds: 32
Drive Type: unknown
RAID Level: unknown
Number of Physical Drives: unknown
Protection Enabled: false
PI Activated: false
PI Type: 0
PI Protection Mask: NO PROTECTION
Supported Guard Types: NO GUARD SUPPORT
DIX Enabled: false
DIX Guard Type: NO GUARD SUPPORT
Emulated DIX/DIF Enabled: false

 

 

 

17 Replies
lucarobusto_arp
Enthusiast
Enthusiast

If I try to apply this:

 

https://blogs.vmware.com/virtualblocks/2020/07/31/using-perennially-reserved-flag-wsfc-rdms/

boot times do not slow down, and I also get this warning i big number of time in the vmkernel.log file.

cpu6:2098298)WARNING: Partition: 1437: Device "naa.6e843b66f1b4efbd2f4cd4c8bdadf5d3" with a VMFS partition is marked perennially reserved. This is not supported and may lead to data loss.

 

 

Any idea how can I investigate?

someta
Contributor
Contributor

I have the exact same problem. It takes about an hour to boot on 7.0.2, while 6.7 boots immediately.

EDIT: Just tried 7.0.1 and that booted just fine. I am using the custom Lenovo images.

Reply
0 Kudos
lucarobusto_arp
Enthusiast
Enthusiast

I solved fresh installing 7.0.2, and not upgrading from 6.7

Reply
0 Kudos
lukaslang
Enthusiast
Enthusiast

We have the exact same problem using a HPE customized 7.0 U2 image. Strangely enough, the first Host updated without a problem.

Reply
0 Kudos
lucarobusto_arp
Enthusiast
Enthusiast

This issue happened on all my 5 HPE hosts. 

Reply
0 Kudos
lukaslang
Enthusiast
Enthusiast

Which kind of storage are you using? Maybe it is related to 3PAR Volumes?

Reply
0 Kudos
someta
Contributor
Contributor

I am using Unity so I don't think it's related to storage.

Reply
0 Kudos
lukaslang
Enthusiast
Enthusiast

In my 10 year vsphere career I never had such issues. Also vcenter has many strange problems. Maybe it is time to invest in quality than feature quantity.

Fresh installing the host did resolve the problem for me also, but it is PITA if you have many hosts.

Reply
0 Kudos
lucarobusto_arp
Enthusiast
Enthusiast

I am sure it's related to storage, and possible to iSCSI ( I have 6 iSCSI targets for each host).

If I disconnect iSCSI targets, boot is fast.

Reply
0 Kudos
lukaslang
Enthusiast
Enthusiast

I am not using iSCSI or the above mentioned RDM Disks. Everything is attached over FC.

In addition I found out, that VMs beeing backed up residing on an upgraded Host need consolidation after a snapshot backup. The manual consolidation finishes without an error, BUT the warning remains and the disks are not consolidated properly.

Only way to consolidate the disk is to shut down the VM or vmotion it to a not upgraded 6.7 Host. A newly installed 7.0 U2 Hosts does not have this behavior.

It seems that there are some major underlying issues in the upgrade process. At the moment I cannot recommend an Upgrade from 6.7 to 7 U2a (at least for our environment).

Reply
0 Kudos
lucarobusto_arp
Enthusiast
Enthusiast

I did not face issues you mention. Consider that at the moment I still have a couple oh hosts not fresh installed, with the underlying issue.

I think I'll schedule a fresh install with major priority after your mentions.

 

Thanks.

Reply
0 Kudos
mbartle
Enthusiast
Enthusiast

HI everyone.  

This just happened to me this morning. I used Lifecycle Manager to upgrade one host from 6.7U3 to 7.0.2 Dell EMC and it now takes 45 minutes to boot.  It gets stuck at the exact same spot and also has similar events in the log files.

Was there ever a fix determined for this ? I would prefer to not have to rebuild each host from scratch,  I've opened a SR and will post back with any updates.

Reply
0 Kudos
taskino
Enthusiast
Enthusiast

Here is the KB Article describing the problem;

https://kb.vmware.com/s/article/84249

You need to copy boot.cfg from bootbank to /altbootbank and edit it. 

SSh to the ESXi host in question.
  - cp /bootbank/boot.cfg /altbootbank/boot.cfg
  - vi /altbootbank/boot.cfg

      Change the first line 'bootstate=0' > 'bootstate=3'
      Change last line 'updated=N' > 'update=N-1' (eg update=3 > updated=2)
    
   - Save file
Reboot and test it. 

GertJ4VM
Contributor
Contributor

>This issue is resolved in VMware vSphere ESXi 7.0 U2c. 

Nope had the same issue with 7.0 U3c, and same workaround fixed it, thx

Reply
0 Kudos
oueb
Contributor
Contributor

Hello! Had the same issue with a Lenovo SR650 server when trying to upgrade from 6.5 17167537 to 7.0 U2A when using Life Cylcle Manager. Tried to upgrade by mounting the ISO with XCC card and no issue at all. I've just done one host out of 6 so far. Will see if it is ok for the other ones.

Reply
0 Kudos
TomKalabis
Enthusiast
Enthusiast

fresh install of VMware-VMvisor-Installer-7.0.0.update03-19482537.x86_64-DellEMC_Customized-A04 to BOSS adapter in DELL R740. Storage is DELL SC3020 with SAS 12G direct connectivity. After first reboot the booting screen hangs on "vmw_satp_alua loaded successfully" and reboot - shutdown process was very slow - "reboot_run_handlersScsiEarly" , BUT i had connected 2 luns to the server visible was only 1. 

i think, that is problem with storage and alua. 

In my case i simply add these commands to ESXi host and after reboot boots in some seconds and i see both LUNs. 

  • esxcli system module parameters set -p issue_scsi_cmd_to_bringup_drive=0 -m lsi_msgpt3
  • esxcli storage nmp satp rule add -s VMW_SATP_ALUA -V COMPELNT -P VMW_PSP_RR -o disable_action_OnRetryErrors -e "Dell EMC SC Series Claim Rule" -O "policy=iops;iops=3"

 

 

 

 

Reply
0 Kudos
kjdfhaueiase
Enthusiast
Enthusiast

Everyone on this thread should read this KB.

I was seeing this problem but only on the server with a different brand USB drive.

Are you seeing "All Paths Down" in your syslog.log? This means you lost access to the boot drive.  The problem in this KB could be the culprit. The step for me that was slowing down boot was setting up the swap file. This now makes perfect sense.

What I find to be absolutely shocking: This has been a bug for 2 years. Not only is it not fixed, but they are carrying on as though it's no big deal because ESX still works. They are saying both "this is not a problem with the vmlinux driver", and "have a conversation with your USB/SD manufacturer about dropouts". 

And now we're hearing booting from USB profiles is getting deprecated. Gee, is it because you don't know how to write a driver?

https://kb.vmware.com/s/article/83450

Reply
0 Kudos