VMware Cloud Community
6655321
Contributor
Contributor
‎02-04-2014 02:55 AM

ESXi 4.1 upgrade to 5.5 - unable to connect to datastores

I upgraded an ESXi host from 4.1 to 5.5.  The server is an HP Proliant DL380 gen7.  The upgrade was done using the HP ESXi 5.5 image.  Prior to the upgrade the server was remediated with Critical and non-critical patches.

When the host restarted after the upgrade, none of the nfs datastores hosted on our netapp were mounted.

DataStores.jpg

I cannot vmkping the datastores from that host.  These datastores are working fine from the other two hosts.

Is this a driver related issue?  Where would you advise I look to troubleshoot this further?

I have attached the vmkernel.log

segment from vmkernel.log which is logging the failed attempts to connect to the datastore:

2014-02-02T12:17:42.629Z cpu2:33260)FSS: 5051: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported

2014-02-02T12:17:42.629Z cpu2:33260)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:17:42.629Z cpu2:33260)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:17:42.629Z cpu2:33260)FSS: 5051: No FS driver claimed device 'control': Not supported

2014-02-02T12:17:42.646Z cpu2:33260)VC: 1958: Device rescan time 6 msec (total number of devices 6)

2014-02-02T12:17:42.646Z cpu2:33260)VC: 1961: Filesystem probe time 23 msec (devices probed 6 of 6)

2014-02-02T12:17:42.646Z cpu2:33260)VC: 1963: Refresh open volume time 0 msec

2014-02-02T12:17:42.746Z cpu2:33260)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:17:42.746Z cpu2:33260)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:17:42.746Z cpu2:33260)FSS: 5051: No FS driver claimed device 'control': Not supported

2014-02-02T12:17:42.747Z cpu2:33260)FSS: 5051: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported

2014-02-02T12:17:42.760Z cpu2:33260)VC: 1958: Device rescan time 8 msec (total number of devices 7)

2014-02-02T12:17:42.760Z cpu2:33260)VC: 1961: Filesystem probe time 21 msec (devices probed 7 of 7)

2014-02-02T12:17:42.760Z cpu2:33260)VC: 1963: Refresh open volume time 0 msec

2014-02-02T12:17:42.800Z cpu11:33648)Loading module nfsclient ...

2014-02-02T12:17:42.801Z cpu11:33648)Elf: 1861: module nfsclient has license VMware

2014-02-02T12:17:42.806Z cpu11:33648)FSS: 936: Registered fs nfs, module 65, fsTypeNum 0xb00f

2014-02-02T12:17:42.807Z cpu11:33648)NFS: 94: Multiple outstanding NFS requests are supported

2014-02-02T12:17:42.807Z cpu11:33648)Mod: 4780: Initialization of nfsclient succeeded with module ID 101.

2014-02-02T12:17:42.807Z cpu11:33648)nfsclient loaded successfully.

2014-02-02T12:17:42.871Z cpu2:33260)NFS: 157: Command: (remount) Server: (192.168.10.50) IP: (192.168.10.50) Path: (/vol/uVMware) Label: (UAT) Options: (None)

2014-02-02T12:17:42.871Z cpu2:33260)StorageApdHandler: 698: APD Handle 20cd0451-fd5e7f6a Created with lock[StorageApd0x41113d]

2014-02-02T12:17:44.758Z cpu2:33007)LVM: 13082: One or more LVM devices have been discovered.

2014-02-02T12:18:13.300Z cpu0:33260)NFS: 168: NFS mount 192.168.10.50:/vol/uVMware failed: Unable to connect to NFS server.

2014-02-02T12:18:13.341Z cpu0:33260)NFS: 157: Command: (remount) Server: (192.168.10.50) IP: (192.168.10.50) Path: (/vol/VMWare2) Label: (VMWare2) Options: (None)

2014-02-02T12:18:13.341Z cpu0:33260)StorageApdHandler: 698: APD Handle e8aec5c1-2d6671b6 Created with lock[StorageApd0x41113d]

2014-02-02T12:18:44.356Z cpu2:33260)NFS: 168: NFS mount 192.168.10.50:/vol/VMWare2 failed: Unable to connect to NFS server.

2014-02-02T12:18:44.398Z cpu2:33260)NFS: 157: Command: (remount) Server: (192.168.10.50) IP: (192.168.10.50) Path: (/vol/VMware) Label: (VMware) Options: (None)

2014-02-02T12:18:44.398Z cpu2:33260)StorageApdHandler: 698: APD Handle 8e5f9a80-e2f1d0f4 Created with lock[StorageApd0x41113d]

2014-02-02T12:19:15.420Z cpu2:33260)NFS: 168: NFS mount 192.168.10.50:/vol/VMware failed: Unable to connect to NFS server.

2014-02-02T12:19:15.462Z cpu2:33260)NFS: 157: Command: (remount) Server: (192.168.10.46) IP: (192.168.10.46) Path: (/vol/vol_nfs01_01) Label: (vol_nfs01_01) Options: (None)

2014-02-02T12:19:15.462Z cpu2:33260)StorageApdHandler: 698: APD Handle 5cd7dd1a-2c9c138d Created with lock[StorageApd0x41113d]

2014-02-02T12:19:46.480Z cpu2:33260)NFS: 168: NFS mount 192.168.10.46:/vol/vol_nfs01_01 failed: Unable to connect to NFS server.

2014-02-02T12:19:46.522Z cpu2:33260)NFS: 157: Command: (remount) Server: (192.168.10.47) IP: (192.168.10.47) Path: (/vol/vol_nfs02_01) Label: (vol_nfs02_01) Options: (None)

2014-02-02T12:19:46.522Z cpu2:33260)StorageApdHandler: 698: APD Handle 47b8d340-397d3cac Created with lock[StorageApd0x41113d]

2014-02-02T12:20:17.539Z cpu0:33260)NFS: 168: NFS mount 192.168.10.47:/vol/vol_nfs02_01 failed: Unable to connect to NFS server.

2014-02-02T12:22:29.159Z cpu0:33785)Config: 446: "Vmknic" = "", Old value: "vmk1" (Status: 0x0)

2014-02-02T12:22:29.159Z cpu14:33454)MigrateNet: 567: Set vMotion to AUTOBIND mode

2014-02-02T12:22:29.159Z cpu14:33454)Tcpip_Vmk: 123: INFO: sosetopt set SO_AUTOBIND

Starting up services2014-02-02T12:22:33.854Z cpu2:34015)FSS: 5051: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported

2014-02-02T12:22:33.854Z cpu2:34015)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:22:33.854Z cpu2:34015)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:22:33.854Z cpu2:34015)FSS: 5051: No FS driver claimed device 'control': Not supported

2014-02-02T12:22:33.918Z cpu2:34015)VC: 1958: Device rescan time 5 msec (total number of devices 7)

2014-02-02T12:22:33.918Z cpu2:34015)VC: 1961: Filesystem probe time 18 msec (devices probed 5 of 7)

2014-02-02T12:22:33.918Z cpu2:34015)VC: 1963: Refresh open volume time 56 msec

2014-02-02T12:22:34.029Z cpu13:34018)<6>usb passthrough enabled; all eligible devices will be unclaimed by kernel drivers

2014-02-02T12:22:34.164Z cpu11:34020)FSS: 5051: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported

2014-02-02T12:22:34.164Z cpu11:34020)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:22:34.164Z cpu11:34020)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:22:34.164Z cpu11:34020)FSS: 5051: No FS driver claimed device 'control': Not supported

2014-02-02T12:22:34.171Z cpu11:34020)VC: 1958: Device rescan time 6 msec (total number of devices 7)

2014-02-02T12:22:34.171Z cpu11:34020)VC: 1961: Filesystem probe time 11 msec (devices probed 5 of 7)

2014-02-02T12:22:34.171Z cpu11:34020)VC: 1963: Refresh open volume time 2 msec

2014-02-02T12:22:34.387Z cpu5:32969)ScsiUid: 273: Path 'vmhba0:C0:T0:L0' does not support VPD Device Id page.

2014-02-02T12:22:34.390Z cpu5:32969)VMWARE SCSI Id: Could not get disk id for vmhba0:C0:T0:L0

2014-02-02T12:22:34.402Z cpu8:34021)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:22:34.402Z cpu8:34021)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:22:34.402Z cpu8:34021)FSS: 5051: No FS driver claimed device 'control': Not supported

2014-02-02T12:22:34.403Z cpu8:34021)FSS: 5051: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported

2014-02-02T12:22:34.407Z cpu8:34021)VC: 1958: Device rescan time 5 msec (total number of devices 7)

2014-02-02T12:22:34.407Z cpu8:34021)VC: 1961: Filesystem probe time 10 msec (devices probed 5 of 7)

2014-02-02T12:22:34.407Z cpu8:34021)VC: 1963: Refresh open volume time 0 msec

2014-02-02T12:22:34.448Z cpu8:34021)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:22:34.448Z cpu8:34021)Vol3: 714: Couldn't read volume header from control: Not supported

2014-02-02T12:22:34.448Z cpu8:34021)FSS: 5051: No FS driver claimed device 'control': Not supported

2014-02-02T12:22:34.449Z cpu8:34021)FSS: 5051: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported

2014-02-02T12:22:34.453Z cpu8:34021)VC: 1958: Device rescan time 5 msec (total number of devices 7)

2014-02-02T12:22:34.453Z cpu8:34021)VC: 1961: Filesystem probe time 10 msec (devices probed 5 of 7)

2014-02-02T12:22:34.453Z cpu8:34021)VC: 1963: Refresh open volume time 0 msec

Thanks in advance.

vmkernel.log.zip
0 Kudos
4 Replies
john23
Commander
Commander
‎02-04-2014 05:46 AM

Are you able to ping your nfs server from esxi server..

check using vmkping.

-A

Thanks -A Read my blogs: www.openwriteup.com
0 Kudos
f10
Expert
Expert
‎02-04-2014 07:30 AM

Hi,

What version of ONTAP are you using and is it 7 Mode or Clustered Data ONTAP?

Since vmkping is not working I would first look at the network before looking at export polices and rules. Can you check the networking on the ESXi host and Storage using command like esxcfg-vmknic -l to check the VMkernel ports and also check esxcfg-nics -l. Check the MTU settings between the controller nics, switch ports and server nics, they should all be 1500 or 9000/9216.

-f10

http://highoncloud.blogspot.in/

About VMware Virtualization on NetApp

Regards, Arun Pandey VCP 3,4,5 | VCAP-DCA | NCDA | HPUX-CSA | http://highoncloud.blogspot.in/ If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".
0 Kudos
6655321
Contributor
Contributor
‎02-04-2014 07:44 AM

Thanks for the replies.

I agree that it is almost certainly a networking issue (vmkping does not work as per my original post) on the host itself as we have had odd issue when vMotioning machines in the past, however connectivity to the datastore was solid prior to the upgrade.

I will try out the esxcfg commands to see if that shows up any config issues.

0 Kudos
6655321
Contributor
Contributor
‎02-07-2014 02:53 AM

Making an adjustment to the NIC teaming settings resolved this.

0 Kudos