Hi!
Three esxi v6.0.0 servers had iSCSI configured on a Synology NAS server that went down. We bought a new NAS server, fixed the old one and migrated the LUN from the old one to the new one. After trying to connect one of the esxi hosts found that the current iSCSI Software Adapter on it contains information about the old connection. I deleted the previous connection in Static Discovery and created a new one by Dynamic Discovery. Under Devices and Path I could not delete the entry, it is frozen:
However, I decided to connect the esxi host to the restored LUN. I added it via Discovery, then clicked Rescan Adapter, then Rescan Storage. On the NAS server, I saw an iSCSI connection. But the problem is that I didn't see the previous configured array that should have appeared in the Datastores section of the esxi host after this procedure. It's gone. Why is that? It seems that ESXi considers it as a new device. How do I get back an FS1-iSCSI named store that was previously attached to an old LUN?
But it is still configured and in an Inaccessible state on the other two esxi hosts.
Screenshot from SAN Manager of Synology NAS (LUN-1 was backuped by Hyper Backup package on Synology and recovered by Hyper Backup Vault):
Another screenshots of 192.168.140.102 esxi host:
Devices:
Path:
Dynamic Discovery:
Static Discovery:
Any help is appreciated.
Accepted Solutions
And is there a way to delete an old disk which is with "Not Cosumed" datastore?
I don't think it's possible to do that after re-signing the snapshot. Especially the snapshot list is empty if I do esxcfg-volume -l. I think a reboot would help here. I can't test it yet. I removed the virtual machine for now from Inventory that was located on a disconnected iSCSI disk and that was inaccessible state. One of the esxi in the cluster immediately saw the snap-4659cc8a-FS1-iSCSI storage as soon as I added the new address and did a Refresh Adapter. Seems everything is ok.
One problem left, a real pain in the ass, is a VM whose disk was referencing an iSCSI array at the time of the sudden outage. The server is loaded with phone calls (Asterisk server is running on it) and shutting it down is not desirable 🙂 The disk is installed as a second (/dev/sdb) in the VM (4 disks in total) and is mounted as ext4 in the OS, the others are in LVMs that were added to / as space ran out, and it looks like the disks were added by name (ie /dev/sda, /dev/sdc, /dev/sdd) in LVM rather than via UUID (I see use_lvmetad=0). This is an old server on CentOS 6.5. Perhaps after a forced shutdown (to remove the iscsi disk and re-add it) the disk order will change both in the VM itself and in the OS, and the VM will most likely stop booting due to LVM issues. For now the server is still running, but I can't request pvs or vgs (LVM), the system hangs. I am afraid to reconfigure it to use UUID disks on a production system, I have never had such experience. I need to somehow reproduce the problem on a test machine and do an experiment.
The second option is to change the disk on the working system. Is it possible to make the VM refer to a resigned iSCSI storage without rebooting? The disk is still the same on a storage, it's just in a different storage due to re-signing. I can't even unmount the disk in the OS yet, it's in busy state. I tried umount -f /data, fuser -km /data and even kill -9 PID - there is an updatedb process that I can't kill, it is using a partition of this hung disk, the process itself is in DN (Uninterruptible sleep with low-priority) state.
Anyway, I'm thinking about what to do with that VM.