Can't mount NFS share - Operation failed, diagnostics report: Unable to get console path for volume

[ThriveTP]

I'm trying to mount an NFS volume on ESXI 6, but keep running into this error.  Googling about hasn't helped, so here I am. Error:

Call "HostDatastoreSystem.CreateNasDatastore" for object "ha-datastoresystem" on ESXi "192.168.xx.xx" failed.

Operation failed, diagnostics report: Unable to get console path for volume, sample name.

The NFS share is located on a synology nas. I've checked permissions and configuration. Everything looks correct based on the various tips and KB articles.

Ideas?

Thanks.

[jpsider]

‌ffrom the synology did you export the NFS share and allow the Esx ip to connect to that mount point?

synology has multiple layers to configure

[ThriveTP]

Yes, I have another esxi host (Host B) that can mount the same NFS share from the same Synology box (Box A), no problem.  I have another synology box (Box B) that the original esxi (Host A) can mount a share from without a problem. Only Host A is throwing the problem with Box A on that particular share.  Rules have been created for both Host A and B to mount the Box A share.

A couple items of note: 1) the share on Box A was mounted at one point on Host A. I don't recall however if I manually dismounted it or if it vanished from the storage configuration list.  2) the subnet schema (everything is on the same subnet) was changed since the share was mounted.  3) Host A CAN mount a different share on box A, just not the desired share.

Maybe after the subnet change, something got bolloxed and the mount point vanished?  and now is throwing this error when trying to remount the same exact share with the different IP address?  Maybe some configuration I can't see in the vsphere client is boinked for this particular share? 

Also, I have not been able to find a reference to the "Unable to get console path for volume" part of the message anywhere on google, so not sure what it even means.

[hussainbte]

don't know much about Synology NAS but below general config I suggest to make sure is OK.

1) Make sure the access permissions on the NAS volume are allowed for multiple hosts and it is not limited to single host access.

2) instead of providing an ESXi vmkernel IP(for hostA) try provided a range which includes HostB as well.

[ThriveTP]

Thanks for the response.  The NFS share can easily be set for discrete IP addresses or an entire subnet. Both hosts A and B were included in the range setup for the share.

UPDATE:  Forgot to mention that that multiple hosts can connect, as well.

[hussainbte]

As the host A can mount another share from the same NAS box, it rules out any networking issues.

Looking at the synology logs while trying to mount the volume might help.

[boss01]

did u check the logs message, vmkernel & hostd.

[ThriveTP]

OK. Found the logs. Thanks for the suggestion (newb, here!). Here's one of the lines from the vmkernel.log after I tried and failed again:

2016-04-11T23:01:01.564Z cpu3:32794)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x9e (0x439d803a58c0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

I couldn't find a correlating entry with the same time stamp in the hostd.log file, but this line appears a few times and mentions vsan"

2016-04-11T23:03:06.793Z info hostd[5F1B1B70] [Originator@6876 sub=Hostsvc opID=a30adcbe] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

-->    accessGenNo = 0,

--> }

[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

The Synology logs don't seem to show any datastore action by the esxi host.  I searched keywords NFS and the ip address of the host.

[hussainbte]

the mpx:vmhba32 stands for messages regarding the local drive that the ESXi host. these messages have no relation with NAS mount. mpx(multi pathing disabled -- aka local).

If you could share the vmkernel  log file. you can file for any IP or host names if you want to keep it secure.
Also there has to be something in the synology logs.

[hussainbte]

I meant to say filter the log file for IPs and names to keep it secure. otherwise you can share it as it is.

[ThriveTP]

I set the log files to save to a share on the NAS (one that works).  I don't think it's an issue to share the IP's or host names, as it's internal to my lab only.  Logs below.

Reading through the logs, I found a reference to the share (DS3 Apps) I was trying to mount.  The explanation was:  "VmkCtl volume already mounted at DS3 Apps." Meaning, this goes back to the share that vanished having been previously mounted before I changed the subnet address scheme.  As a test, I tried to add the exact same share located at /volume1/apps, but using a different data store name of "DS3 Apps 2", and it worked.

So I guess the question now is, how do I reclaim the original datastore name of "DS3 Apps"?  The association with a previous VmkCtl must be listed somewhere, right?

kernel.log

2016-04-12T20:16:27.902Z cpu4:32795)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x9e (0x439d803fd1c0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

2016-04-12T20:16:27.904Z cpu4:32795)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x439d803fd1c0, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE

2016-04-12T20:16:27.907Z cpu4:32795)ScsiDeviceIO: 2645: Cmd(0x439d803fd1c0) 0x1a, CmdSN 0x49998 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2016-04-12T20:16:27.907Z cpu4:32795)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x85 (0x439d803fd1c0, 34494) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

hostd.log

2016-04-12T20:15:12.251Z info hostd[5F1F2B70] [Originator@6876 sub=Solo.Vmomi opID=a30b14a8 user=root] Activation [N5Vmomi10ActivationE:0x5e9331f0] : Invoke done [waitForUpdatesEx] on [vmodl.query.PropertyCollector:ha-property-collector]

2016-04-12T20:15:12.251Z verbose hostd[5F1F2B70] [Originator@6876 sub=Solo.Vmomi opID=a30b14a8 user=root] Arg version:

--> "35"

2016-04-12T20:15:12.251Z verbose hostd[5F1F2B70] [Originator@6876 sub=Solo.Vmomi opID=a30b14a8 user=root] Arg options:

--> (vmodl.query.PropertyCollector.WaitOptions) {

-->    maxWaitSeconds = 600,

-->    maxObjectUpdates = 100

--> }

2016-04-12T20:15:12.251Z info hostd[5F1F2B70] [Originator@6876 sub=Solo.Vmomi opID=a30b14a8 user=root] Throw vmodl.fault.RequestCanceled

2016-04-12T20:15:12.251Z info hostd[5F1F2B70] [Originator@6876 sub=Solo.Vmomi opID=a30b14a8 user=root] Result:

--> (vmodl.fault.RequestCanceled) {

-->    faultCause = (vmodl.MethodFault) null,

-->    msg = ""

--> }

2016-04-12T20:15:12.251Z error hostd[60580B70] [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] Failed to read header on stream <io_obj p:0x5e908ddc, h:43, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

2016-04-12T20:16:08.297Z info hostd[FFFECA80] [Originator@6876 sub=Vimsvc.TaskManager opID=974F8A33-0000007D-151b user=root] Task Created : haTask-ha-host-vim.host.DatastoreSystem.createNasDatastore-160610

2016-04-12T20:16:08.298Z info hostd[FFFECA80] [Originator@6876 sub=Hostsvc.DatastoreSystem opID=974F8A33-0000007D-151b user=root] Creating datastore DS3 Apps

2016-04-12T20:16:08.298Z warning hostd[FFFECA80] [Originator@6876 sub=Hostsvc.FSVolumeProvider opID=974F8A33-0000007D-151b user=root] AddNasVolume: VmkCtl volume already mounted at DS3 Apps

2016-04-12T20:16:08.298Z error hostd[FFFECA80] [Originator@6876 sub=Hostsvc.FSVolumeProvider opID=974F8A33-0000007D-151b user=root] VmkCtl GetNetworkFileSystemByLabel or ProcessNas failed: Unable to get console path for volume, DS3 Apps

2016-04-12T20:16:08.298Z info hostd[FFFECA80] [Originator@6876 sub=Default opID=974F8A33-0000007D-151b user=root] AdapterServer caught exception: vim.fault.PlatformConfigFault

2016-04-12T20:16:08.298Z info hostd[FFFECA80] [Originator@6876 sub=Vimsvc.TaskManager opID=974F8A33-0000007D-151b user=root] Task Completed : haTask-ha-host-vim.host.DatastoreSystem.createNasDatastore-160610 Status error

2016-04-12T20:16:08.298Z info hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=974F8A33-0000007D-151b user=root] Activation [N5Vmomi10ActivationE:0x1f4620c8] : Invoke done [createNasDatastore] on [vim.host.DatastoreSystem:ha-datastoresystem]

2016-04-12T20:16:08.298Z verbose hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=974F8A33-0000007D-151b user=root] Arg spec:

--> (vim.host.NasVolume.Specification) {

-->    remoteHost = "192.168.64.23",

-->    remotePath = "/volume1/apps",

-->    localPath = "DS3 Apps",

-->    accessMode = "readWrite",

-->    type = "nfs",

-->    userName = <unset>,

-->    password = <unset>,

-->    securityType = <unset>,

--> }

2016-04-12T20:16:08.298Z info hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=974F8A33-0000007D-151b user=root] Throw vim.fault.PlatformConfigFault

2016-04-12T20:16:08.298Z info hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=974F8A33-0000007D-151b user=root] Result:

--> (vim.fault.PlatformConfigFault) {

-->    faultCause = (vmodl.MethodFault) null,

-->    faultMessage = (vmodl.LocalizableMessage) [

-->       (vmodl.LocalizableMessage) {

-->          key = "com.vmware.esx.hostctl.default",

-->          arg = (vmodl.KeyAnyValue) [

-->             (vmodl.KeyAnyValue) {

-->                key = "reason",

-->                value = "Unable to get console path for volume, DS3 Apps"

-->             }

-->          ],

-->          message = <unset>

-->       }

-->    ],

-->    text = ""

-->    msg = ""

--> }

[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

2016-04-12T20:16:26.041Z info hostd[FFFECA80] [Originator@6876 sub=Hostsvc opID=a30b1520] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

-->    accessGenNo = 0,

--> }

[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

2016-04-12T20:17:57.203Z info hostd[FFFECA80] [Originator@6876 sub=Hostsvc opID=a30b152c] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

-->    accessGenNo = 0,

--> }

[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

2016-04-12T20:19:02.533Z info hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=974F8A33-00000018-141f user=root] Activation [N5Vmomi10ActivationE:0x1f2e03c8] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:session[526a9dba-8a0b-9784-a023-3de98895e1e7]525c8ce1-8b16-eba4-3bcc-9fe0b8e0b38d]

2016-04-12T20:19:02.533Z verbose hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=974F8A33-00000018-141f user=root] Arg version:

--> "1"

2016-04-12T20:19:02.533Z info hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=974F8A33-00000018-141f user=root] Throw vmodl.fault.RequestCanceled

2016-04-12T20:19:02.533Z info hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=974F8A33-00000018-141f user=root] Result:

--> (vmodl.fault.RequestCanceled) {

-->    faultCause = (vmodl.MethodFault) null,

-->    msg = ""

--> }

2016-04-12T20:19:02.534Z error hostd[5F1F2B70] [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] Failed to read header on stream <io_obj p:0x5ea0bd8c, h:35, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

2016-04-12T20:19:28.364Z info hostd[FFFECA80] [Originator@6876 sub=Hostsvc opID=a30b153c] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

-->    accessGenNo = 0,

--> }

2016-04-12T20:20:12.257Z info hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=a30b1508 user=root] Activation [N5Vmomi10ActivationE:0x5fd52090] : Invoke done [waitForUpdatesEx] on [vmodl.query.PropertyCollector:ha-property-collector]

2016-04-12T20:20:12.257Z verbose hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=a30b1508 user=root] Arg version:

--> "35"

2016-04-12T20:20:12.257Z verbose hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=a30b1508 user=root] Arg options:

--> (vmodl.query.PropertyCollector.WaitOptions) {

-->    maxWaitSeconds = 600,

-->    maxObjectUpdates = 100

--> }

2016-04-12T20:20:12.258Z info hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=a30b1508 user=root] Throw vmodl.fault.RequestCanceled

2016-04-12T20:20:12.258Z info hostd[FFFECA80] [Originator@6876 sub=Solo.Vmomi opID=a30b1508 user=root] Result:

--> (vmodl.fault.RequestCanceled) {

-->    faultCause = (vmodl.MethodFault) null,

-->    msg = ""

--> }

2016-04-12T20:20:12.258Z error hostd[60580B70] [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] Failed to read header on stream <io_obj p:0x5ff299f4, h:38, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>: N7Vmacore15SystemExceptionE(Connection reset by peer)

[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

[hussainbte]

Probably the NAS volume was not cleanly unmounted before the subnet was changed.

Unmount the newly created NAS volume(the one added with a different name) and list the current volumes using the below KB.

you might not have Storage IO control enabled,however I am just sharing the KB so that we could make use of the commands.

VMware KB: Unable to remove an inaccessible NFS datastore with Storage I/O control enabled

I think you can also try and use the below KB to remount the volume by same name, just use with the latest IP.

VMware KB: Remounting a disconnected NFS datastore from the ESXi/ESX command line

[ThriveTP]

Hussainbte-

This is getting more interesting. Thanks for the links. I dug through both. Results:

When I list the datastore from the command line, DS3 Apps doesn't show up.  I went ahead and followed the instructions in link 1 to stop the SIOC server, rescan all, then restart the service, but still can't add the datastore with the same name from the vSphere Client. 

Then, following some of the steps in Link 2, I tried to add the datastore via CLI. This time, instead of the console message, I received "Unable to add new NAS, volume with the label DS3 Apps already exists".  I once more ran "esxcli storage nfs list" to check again, but DS3 Apps still doesn't show up.  Huh. 

So somewhere the host is holding on to this info. Key is to find where. If we listing the datastore via the CLI doesn't show DS3 Apps, so not sure how to find it.

Another piece of information: I did have this host associated to a vCenter server. In my ignorance, when I changed the subnet, I found that I could not change the ip address of the vCenter server because I used it as the system name rather than using a FQDN. I just redeployed, lesson learned, but haven't yet added this host to it. I wonder if the fact that it's not a standalone host (being linked to a no longer existing vCenter server) has something to do with this issue. I read a post on how someone tried to rename a datastore via CLI, but vCenter kept reverting the change to the original name.  No idea, but a thought.

[hussainbte · Accepted solution]

Check for the esx.conf file in /etc/vmware.

If you could also share a copy with me..

VMware KB: Attempting to mount the NFS volume fails with the error: Unable to resolve hostname

Check the below link as well.

http://www.bussink.ch/?p=1640

[ThriveTP]

I saw link one last week, but when I ran esxcfg-nas -l, it doesn't show the (or any) problem share, so I had no idea what to put in the hosts file.

HOWEVER!!!  Link two http://www.bussink.ch/?p=1640  was the ticket.  It didn't give me the exact message he encountered, but close enough. I was able to delete the reference to DS3 Apps.  After that, no more errors when trying to mount the share using the same name. 

Thank you for your time and patience in helping to sort this out.

Update: fixed broken link.

[FengZhaoCHN]

I meet that this issue, done final analysis then know that it was been caused by existing NFS storage in esx.conf file under /etc/vmware folder,  use vi esx.conf removed the items with existing NFS storage name from this file, reboot the Vmware host then can add the NFS datastore.