Restoring an Inaccessible VM

[Mulenga07]

Hello,

I'm fairly new to this so forgive my inexperience.

Our setup is one cluster of 4 ESXi 6.5 hosts, 2 NFS 3 datastores, vSphere HA is turned on

I'd like to start a VM that we've been using to store some backup files but cannot because it is inaccessible. After looking around, I noticed that the NFS 3 datastore it is related to was removed from the VM's host last week. One of the descriptions on the eventslog from last week was that the confriguration file for the VM cannot be found. My theory is: I can get the VM to run if I can get the hosts to see the datastore. I attempted to mount the datastore but did not succeed. When I click "Mount Datastore to Additional Hosts" in the vSphere Web Client, all it does is load and does not allow me to select the hosts I want to mount the datastore on. Any ideas on how I can get this to work. Other VMs related to the second datastore (which is mounted) are running.

I also noticed a couple of alerts on the hosts i.e. "Host hardware system board status" and "The number of vSphere HA hearbeat datastores for this host is 1, which is less than required: 2". I'm not sure if this has an impact on what we are trying to do.

I'll appreciate any help I can get.

Thanks in advance.

[Mulenga07]

That's the VM summary.

That's the datastore's list of related VMs. The first one is the one I'm trying to restore.

I'm not sure why there are no files. I was expecting to see some since there are some related VMs.

That's the datastore summary.

[SureshKumarMuth]

The datastore summary says  0 hosts and there are no files seen in the DS.  Do you know what was used space in the DS before the issue ? because now I see the datastore is almost empty.

While adding the DS again, did you remove this Ds from inventory and added it again ? What is the error you see while adding this DS if it is failing our while adding it again.

The errors related to host system board and insufficient DS for HA are not related to this issue. However, we have to find what happened at the back end to the NFS export where we see no data in it.

[Mulenga07]

The datastore summary says  0 hosts and there are no files seen in the DS.  The ds was somehow removed from all of the hosts last week hence 0 Hosts.

Do you know what was used space in the DS before the issue ? because now I see the datastore is almost empty. I only looked at the contents of the ds after the issue had occurred else had been working on it. However; I have a couple of assumptions: the path to a .vmx file listed under virtual machines in one of the hosts seems to indicate that our backup server stored its files on the ds. My other assumption (and I'm really just guessing here) is that some HA process migrated the files to this ds and then something went wrong.

While adding the DS again, did you remove this Ds from inventory and added it again ? I did not. My assumption was that this was unnecessary because the ds was removed from the hosts.

What is the error you see while adding this DS if it is failing our while adding it again. By adding are you referring to mounting?

The errors related to host system board and insufficient DS for HA are not related to this issue. However, we have to find what happened at the back end to the NFS export where we see no data in it.

[SureshKumarMuth]

The datastore summary says  0 hosts and there are no files seen in the DS.  The ds was somehow removed from all of the hosts last week hence 0 Hosts.

If it was not manual then there could be something went wrong at the storage end, only logs can tell us what happened actually.

Do you know what was used space in the DS before the issue ? because now I see the datastore is almost empty. I only looked at the contents of the ds after the issue had occurred else had been working on it. However; I have a couple of assumptions: the path to a .vmx file listed under virtual machines in one of the hosts seems to indicate that our backup server stored its files on the ds. My other assumption (and I'm really just guessing here) is that some HA process migrated the files to this ds and then something went wrong.

HA wont migrate the files among datastore, it just restarts the VMs when ESXi goes down unexpectedly. You have to confirm by referring your old data (rvtools data/ inventory export) to confirm what was there in this datastore before this issue. guessings wont work out much ,sometimes VM data might be scattered  across multiple datastores.

While adding the DS again, did you remove this Ds from inventory and added it again ? I did not. My assumption was that this was unnecessary because the ds was removed from the hosts.

Since this DS is not mounted on any ESXi hosts, the object is still in vCenter DB which shows in the inventory which is of no use. you may remove the datastore and try adding it again.

What is the error you see while adding this DS if it is failing our while adding it again. By adding are you referring to mounting?

Yes, mount this NFS again on ESXi as datastore, while trying to do that we will come to know the status of the mount, if it is still a healthy NFS mount then we will be able to add the datastore and read the data else it will error out.

The errors related to host system board and insufficient DS for HA are not related to this issue. However, we have to find what happened at the back end to the NFS export where we see no data in it.

Have a word with your storage team to know about the health status of this mount point and check What they see at storage end.

[Mulenga07]

You have to confirm by referring your old data (rvtools data/ inventory export) Is there a way of viewing this from the web client?

Since this DS is not mounted on any ESXi hosts, the object is still in vCenter DB which shows in the inventory which is of no use. you may remove the datastore and try adding it again. Would this process cause any data loss?

Yes, mount this NFS again on ESXi as datastore, while trying to do that we will come to know the status of the mount, if it is still a healthy NFS mount then we will be able to add the datastore and read the data else it will error out. When I attempt to mount the error I'm getting is "Failed to mount NFS datastore ds01 - Operation failed, diagnostics report: Unable to get console path for volume"

Have a word with your storage team to know about the health status of this mount point and check What they see at storage end. Unfortunately, we do not have a storage team and are unable to reach the person that helped us set the environment up 😞

[SureshKumarMuth]

You have to confirm by referring your old data (rvtools data/ inventory export) Is there a way of viewing this from the web client? -- No you will not get complete info

Since this DS is not mounted on any ESXi hosts, the object is still in vCenter DB which shows in the inventory which is of no use. you may remove the datastore and try adding it again. Would this process cause any data loss?

Right now no hosts are connected and it is kind of stale entry, so no harm in removing it.

Yes, mount this NFS again on ESXi as datastore, while trying to do that we will come to know the status of the mount, if it is still a healthy NFS mount then we will be able to add the datastore and read the data else it will error out. When I attempt to mount the error I'm getting is "Failed to mount NFS datastore ds01 - Operation failed, diagnostics report: Unable to get console path for volume"

There is an issue with the NFS mount, unless we fix it ,we cant proceed further. You have to engage someone to check at the storage end fix the issue.

Have a word with your storage team to know about the health status of this mount point and check What they see at storage end. Unfortunately, we do not have a storage team and are unable to reach the person that helped us set the environment up 😞

Same thing , we need someone to work at storage end.

[Mulenga07]

So some disks on the physical machine (Netapp) are not spinning, guess that's why we have this issue. Trying to see how to work round it.

Thanks for your time and help.