VMware Cloud Community
tcrosasso
Contributor
Contributor
Jump to solution

Datastore access problem after ESX 7.0 upgrade

Hello,

We upgraded an ESX from VMware ESX, 6.7.0, 15820472 to 7.0 Update 3g using the upgrade from ESXi ISO method.

Following the upgrade, the ESX only displayed 2 datastores available out of 4.

The ESXi are Dell R640 servers mounted in a vSphere HA cluster.
The SAN array is a Dell EMC SC 275232.
The SAN array is connected to the ESXi with 12 Gbps SAS (Dell PowerEdge HBA SAS 12 Gbit/s controller)

After having reinstalled the ESX in version 6.7 all the datatstores were again visible by the ESX.

Could you please help me to enable us to smoothly upgrade the ESXs in our cluster to 7.0 Update 3g.

I attach the ESX logs recovered just after the upgrade to version 7.0 Update 3g.

Many thanks to anyone who can help me.

Reply
0 Kudos
1 Solution

Accepted Solutions
tcrosasso
Contributor
Contributor
Jump to solution

Hello everyone and really sorry for my extremely late return...

I was finally able to solve my problem by applying this dell kb:

https://www.dell.com/support/kbdoc/en-us/000138029/sc-storage-customer-notification-driver-compatibi...

In summary :

  • Connect in SSH on the problematic ESX
  • Enter this command: esxcli system module parameters set -p issue_scsi_cmd_to_bringup_drive=0 -m lsi_msgpt3
  • Restart the ESX
  • The datastores are visible again !

I hope this will be useful for others, and again thank you all.

View solution in original post

9 Replies
Digory34
Enthusiast
Enthusiast
Jump to solution

Hi Tcrosasso,

Did you try this KB ?

Dell Storage: Preparing VMware ESXi Hosts To Attach to SCv20x0, SCv30x0, SC4020, SC5020 SAS Arrays |...

Please check after the upgrade if the driver lsi_msgpt3 is used. on 6.7 mpt3sas could be use but not in Esxi 7.0.

Also you have to enable : issue_scsi_cmd_to_bringup_drive to 1.

so in the nutshell, check the driver use after the upgrade (disable mpt3sas and enable lsi_msgpt3) and enabled the feature issue_scsi_cmd_to_bringup_drive.

Reply
0 Kudos
tcrosasso
Contributor
Contributor
Jump to solution

Thank you Digory, I will test that during the next upgrade attempt.

On the other hand, if that's right, I shouldn't see any datastore from the upgraded ESX, but here I see 2 out of 4 ESX.

The 4 datastores are on the same SAN area and on the same volume.

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

That's indeed strange.

What I could think of (only a guess) is that one of the SAS IDs on the host changed, so that the LUNs are only presented/visible to one of the host's SAS HBA Ports, which in turn can only detect the LUNs that are owned by the controller it is connected to.

André

Btw. According to the log files, you've upgraded to U3f. The Dell customized image for U3g has been released today.

Reply
0 Kudos
tcrosasso
Contributor
Contributor
Jump to solution

In my vmkernel logs I see these warnings, which seem to correspond to my problem. device:naa.6000d310043320000000000000000000000000b and naa.6000d3100433200000000000000000009 corresponding to the IDs of the missing datastores. Despite my research, I did not find any information on these warnings...

 

2022-09-28T12:36:26.498Z cpu6:2097772)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1937: Could not select path for device "Unregistered Device".
2022-09-28T12:36:26.498Z cpu6:2097772)StorageApdHandler: 966: APD Handle Created with lock[StorageApd-0x430816d96f30]
2022-09-28T12:36:26.498Z cpu6:2097772)PsaStorEvents: 676: Event Subsystem: Device Events, Created!
2022-09-28T12:36:26.498Z cpu6:2097772)PsaStorEvents: 676: Event Subsystem: Device Events - Internal, Created!
2022-09-28T12:36:26.498Z cpu6:2097772)VMWARE SCSI Id: Id for vmhba2:C0:T1:L3
0x60 0x00 0xd3 0x10 0x04 0x33 0x20 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x0b 0x43 0x6f 0x6d 0x70 0x65 0x6c
2022-09-28T12:36:26.498Z cpu6:2097772)WARNING: vmw_psp_rr: psp_rrSelectPath:2270: Could not select path for device "naa.6000d31004332000000000000000000b".
2022-09-28T12:36:26.498Z cpu6:2097772)WARNING: NMP: nmp_IssueCommandToDevice:5950: I/O could not be issued to device "naa.6000d31004332000000000000000000b" due to Not found
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDeviceIO: 4176: Cmd(0x45b956089948) 0x12, CmdSN 0x989 from world 0 to dev "naa.6000d31004332000000000000000000b" failed H:0x1 D:0x0 P:0x0
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDeviceIO: 11073: Get VPD 86 Inquiry for device "naa.6000d31004332000000000000000000b" from Plugin "NMP" failed. Not supported
2022-09-28T12:36:26.498Z cpu6:2097772)WARNING: vmw_psp_rr: psp_rrSelectPath:2270: Could not select path for device "naa.6000d31004332000000000000000000b".
2022-09-28T12:36:26.498Z cpu6:2097772)WARNING: NMP: nmp_IssueCommandToDevice:5950: I/O could not be issued to device "naa.6000d31004332000000000000000000b" due to Not found
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDeviceIO: 4176: Cmd(0x45b956089948) 0x12, CmdSN 0x98a from world 0 to dev "naa.6000d31004332000000000000000000b" failed H:0x1 D:0x0 P:0x0
2022-09-28T12:36:26.498Z cpu6:2097772)WARNING: vmw_psp_rr: psp_rrSelectPath:2270: Could not select path for device "naa.6000d31004332000000000000000000b".
2022-09-28T12:36:26.498Z cpu6:2097772)WARNING: NMP: nmp_IssueCommandToDevice:5950: I/O could not be issued to device "naa.6000d31004332000000000000000000b" due to Not found
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDeviceIO: 4176: Cmd(0x45b956089948) 0x12, CmdSN 0x98b from world 0 to dev "naa.6000d31004332000000000000000000b" failed H:0x1 D:0x0 P:0x0
2022-09-28T12:36:26.498Z cpu6:2097772)WARNING: ScsiDevice: 2665: Failing registration of device naa.6000d31004332000000000000000000b as there are no active paths
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDevice: 5171: Failed to Unregister 'REPORTED LUNS DATA CHANGED' unit attention for device:naa.6000d31004332000000000000000000b. Reason: Not found
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDevice: 5181: Failed to Unregister 'MODE PARAMETERS CHANGED' unit attention for device:naa.6000d31004332000000000000000000b. Reason: Not found
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDevice: 5191: Failed to Unregister 'CAPACITY DATA HAS CHANGED' unit attention for device:naa.6000d31004332000000000000000000b. Reason: Not found
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDevice: 5201: Failed to Unregister 'ASYMMETRIC ACCESS STATE CHANGED' unit attention for device:naa.6000d31004332000000000000000000b. Reason: Not found
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDevice: 5211: Failed to Unregister 'INQUIRY PARAMETERS CHANGED' unit attention for device:naa.6000d31004332000000000000000000b. Reason: Not found
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDevice: 5221: Failed to register for 'TARGET MICRO CODE CHANGED' unit attention for device:naa.6000d31004332000000000000000000b. Reason: Not found
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDevice: 5231: Failed to Unregister 'POWER ON RESET' unit attention for device:naa.6000d31004332000000000000000000b. Reason: Not found
2022-09-28T12:36:26.498Z cpu6:2097772)ScsiDevice: 5299: Unregistered device 'naa.6000d31004332000000000000000000b' [0x430816dcd4c0]
2022-09-28T12:36:26.498Z cpu6:2097772)PsaStorEvents: 720: Event Subsystem: Device Events, Destroyed!
2022-09-28T12:36:26.498Z cpu6:2097772)PsaStorEvents: 720: Event Subsystem: Device Events - Internal, Destroyed!
2022-09-28T12:36:26.498Z cpu6:2097772)WARNING: NMP: nmp_RegisterDevice:876: Registration of NMP device with primary uid 'naa.6000d31004332000000000000000000b' failed. Not supported
2022-09-28T12:36:26.498Z cpu6:2097772)StorageApdHandler: 1051: Freeing APD handle 0x430816d96f30 []
2022-09-28T12:36:26.498Z cpu6:2097772)StorageApdHandler: 1135: APD Handle freed!
2022-09-28T12:36:26.499Z cpu6:2097772)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1937: Could not select path for device "Unregistered Device".
2022-09-28T12:36:26.499Z cpu6:2097772)StorageApdHandler: 966: APD Handle Created with lock[StorageApd-0x430816d96f30]
2022-09-28T12:36:26.499Z cpu6:2097772)PsaStorEvents: 676: Event Subsystem: Device Events, Created!
2022-09-28T12:36:26.499Z cpu6:2097772)PsaStorEvents: 676: Event Subsystem: Device Events - Internal, Created!
2022-09-28T12:36:26.499Z cpu6:2097772)VMWARE SCSI Id: Id for vmhba2:C0:T1:L2
0x60 0x00 0xd3 0x10 0x04 0x33 0x20 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x09 0x43 0x6f 0x6d 0x70 0x65 0x6c
2022-09-28T12:36:26.499Z cpu6:2097772)WARNING: vmw_psp_rr: psp_rrSelectPath:2270: Could not select path for device "naa.6000d310043320000000000000000009".
2022-09-28T12:36:26.499Z cpu6:2097772)WARNING: NMP: nmp_IssueCommandToDevice:5950: I/O could not be issued to device "naa.6000d310043320000000000000000009" due to Not found
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDeviceIO: 4176: Cmd(0x45b956089948) 0x12, CmdSN 0x98c from world 0 to dev "naa.6000d310043320000000000000000009" failed H:0x1 D:0x0 P:0x0
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDeviceIO: 11073: Get VPD 86 Inquiry for device "naa.6000d310043320000000000000000009" from Plugin "NMP" failed. Not supported
2022-09-28T12:36:26.499Z cpu6:2097772)WARNING: vmw_psp_rr: psp_rrSelectPath:2270: Could not select path for device "naa.6000d310043320000000000000000009".
2022-09-28T12:36:26.499Z cpu6:2097772)WARNING: NMP: nmp_IssueCommandToDevice:5950: I/O could not be issued to device "naa.6000d310043320000000000000000009" due to Not found
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDeviceIO: 4176: Cmd(0x45b956089948) 0x12, CmdSN 0x98d from world 0 to dev "naa.6000d310043320000000000000000009" failed H:0x1 D:0x0 P:0x0
2022-09-28T12:36:26.499Z cpu6:2097772)WARNING: vmw_psp_rr: psp_rrSelectPath:2270: Could not select path for device "naa.6000d310043320000000000000000009".
2022-09-28T12:36:26.499Z cpu6:2097772)WARNING: NMP: nmp_IssueCommandToDevice:5950: I/O could not be issued to device "naa.6000d310043320000000000000000009" due to Not found
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDeviceIO: 4176: Cmd(0x45b956089948) 0x12, CmdSN 0x98e from world 0 to dev "naa.6000d310043320000000000000000009" failed H:0x1 D:0x0 P:0x0
2022-09-28T12:36:26.499Z cpu6:2097772)WARNING: ScsiDevice: 2665: Failing registration of device naa.6000d310043320000000000000000009 as there are no active paths
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDevice: 5171: Failed to Unregister 'REPORTED LUNS DATA CHANGED' unit attention for device:naa.6000d310043320000000000000000009. Reason: Not found
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDevice: 5181: Failed to Unregister 'MODE PARAMETERS CHANGED' unit attention for device:naa.6000d310043320000000000000000009. Reason: Not found
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDevice: 5191: Failed to Unregister 'CAPACITY DATA HAS CHANGED' unit attention for device:naa.6000d310043320000000000000000009. Reason: Not found
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDevice: 5201: Failed to Unregister 'ASYMMETRIC ACCESS STATE CHANGED' unit attention for device:naa.6000d310043320000000000000000009. Reason: Not found
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDevice: 5211: Failed to Unregister 'INQUIRY PARAMETERS CHANGED' unit attention for device:naa.6000d310043320000000000000000009. Reason: Not found
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDevice: 5221: Failed to register for 'TARGET MICRO CODE CHANGED' unit attention for device:naa.6000d310043320000000000000000009. Reason: Not found
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDevice: 5231: Failed to Unregister 'POWER ON RESET' unit attention for device:naa.6000d310043320000000000000000009. Reason: Not found
2022-09-28T12:36:26.499Z cpu6:2097772)ScsiDevice: 5299: Unregistered device 'naa.6000d310043320000000000000000009' [0x430816dcd4c0]
2022-09-28T12:36:26.499Z cpu6:2097772)PsaStorEvents: 720: Event Subsystem: Device Events, Destroyed!
2022-09-28T12:36:26.499Z cpu6:2097772)PsaStorEvents: 720: Event Subsystem: Device Events - Internal, Destroyed!
2022-09-28T12:36:26.499Z cpu6:2097772)WARNING: NMP: nmp_RegisterDevice:876: Registration of NMP device with primary uid 'naa.6000d310043320000000000000000009' failed. Not supported
2022-09-28T12:36:26.499Z cpu6:2097772)StorageApdHandler: 1051: Freeing APD handle 0x430816d96f30 []
2022-09-28T12:36:26.499Z cpu6:2097772)StorageApdHandler: 1135: APD Handle freed!

Reply
0 Kudos
Digory34
Enthusiast
Enthusiast
Jump to solution

Hi Tcrosasso,

Could you check if you NMP is always set to SATP_ALUA with PSP Round Robin?

in Version 7.0, i know that HPP was introduce and can cause some error.

VMware High Performance Plug-In and Path Selection Schemes

What is your driver / FW also?

Tags (1)
Reply
0 Kudos
Digory34
Enthusiast
Enthusiast
Jump to solution

Also, i think that i got this issue before and from DSM i could see an issue on port SAS, the FW Version from the Array was not compatible with Esxi 7.0 and need to be updated.

You may check with the storage vendor or the Compellent HCL.

Reply
0 Kudos
maksym007
Expert
Expert
Jump to solution

What about Firmware and Drivers of Network Cards/FibreChannel on ESXi hosts? 

they should be updated too after your upgrade to vSphere7 

Reply
0 Kudos
tcrosasso
Contributor
Contributor
Jump to solution

Hello everyone and really sorry for my extremely late return...

I was finally able to solve my problem by applying this dell kb:

https://www.dell.com/support/kbdoc/en-us/000138029/sc-storage-customer-notification-driver-compatibi...

In summary :

  • Connect in SSH on the problematic ESX
  • Enter this command: esxcli system module parameters set -p issue_scsi_cmd_to_bringup_drive=0 -m lsi_msgpt3
  • Restart the ESX
  • The datastores are visible again !

I hope this will be useful for others, and again thank you all.

Seth70
Contributor
Contributor
Jump to solution

Thank you very much! One of our host which was migrated from 6.7 to 7.03 has lost its Storage connectivity but this solution worked like a charm! Great help, thank you. 

Tags (1)
Reply
0 Kudos