VMware Cloud Community
billu
Contributor
Contributor

ESX 4.1 - Unable to Delete Datastore

I have 3 ESX 4.1 servers managed by vCenter ( not clustered ). I have moved all VMs off of ESX1 and am attempting to delete a secondary datastore. The datastore is empty and I have the server in maintenance mode. When I try to delete the datastore, the vSphere client throws this error:

Call "HostDatastoreSystem.RemoveDatastore" for object "datastoreSystem-543" on vCenter Server "ER-VCENTER.myDomain.com" failed.

The vmkernel log contains these entries:

Nov 22 15:17:50 ESX1 vmkernel: 0:05:34:13.160 cpu8:4118)WARNING: ScsiCore: 1399: Invalid sense buffer: error=0x0, valid=0x0, segment=0x0, key=0x5

Nov 22 15:17:50 ESX1 vmkernel: 0:05:34:13.162 cpu8:4118)WARNING: ScsiCore: 1399: Invalid sense buffer: error=0x0, valid=0x0, segment=0x0, key=0x0

Nov 22 15:17:50 ESX1 vmkernel: 0:05:34:13.167 cpu10:4118)WARNING: ScsiCore: 1399: Invalid sense buffer: error=0x0, valid=0x0, segment=0x0, key=0x0

Nov 22 15:17:50 ESX1 vmkernel: 0:05:34:13.168 cpu10:4118)WARNING: ScsiCore: 1399: Invalid sense buffer: error=0x0, valid=0x0, segment=0x0, key=0x0

Nov 22 15:17:52 ESX1 vmkernel: 0:05:34:14.600 cpu8:4120)WARNING: ScsiCore: 1399: Invalid sense buffer: error=0x0, valid=0x0, segment=0x0, key=0x0

Nov 22 15:17:52 ESX1 vmkernel: 0:05:34:14.603 cpu14:4233)DevFS: 3791: Overflow detected (numDiskBlocks 5754945792)

Nov 22 15:17:52 ESX1 vmkernel: 0:05:34:14.663 cpu9:4119)WARNING: ScsiCore: 1399: Invalid sense buffer: error=0x0, valid=0x0, segment=0x0, key=0x0

Nov 22 15:17:52 ESX1 vmkernel: 0:05:34:14.667 cpu4:4236)DevFS: 3791: Overflow detected (numDiskBlocks 5754945792)

I have also tried connecting directly to the ESX box via the client, but get the same error. I should also note that this datastore is on local drives...

Any suggestions would be very much appreciated!

tia,

Bill

Reply
0 Kudos
18 Replies
eeg3
Commander
Commander

Although it is in French, this page (translated using Google Translate) seems to provide a fix for the symptoms you're describing.

>Solution:

>Apparently when eliminating the LUN is still accessed by a write-blocking ESXi 4.1 This method RemoveDatastore.

>To resolve this problem, move the contents of the VMFS and then restart the server to reset the datastore. This workaround should then allow to permanently remove the LUN.

>Note: If you control the operation with a vCenter, connect directly to one of ESXi to remove the datastore.






____________

blog.eeg3.net | Useful VMware-related Links

If you found this or any other post helpful, please consider the use of the Helpful/Correct buttons to award points.

Blog: http://blog.eeg3.net
Reply
0 Kudos
billu
Contributor
Contributor

Thanks for the suggestion, but that didn't seem to help. The datastore is empty, the ESX box is in maintenance mode, I rebooted, I connected directly to the ESX box, and still no go... same error...

Reply
0 Kudos
FranckRookie
Leadership
Leadership

Hi Bill,

The error message is not exactly the same but maybe you could find some information in this discussion.

Good luck!

Regards

Franck

Reply
0 Kudos
idle-jam
Immortal
Immortal

Try this KB http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101710...


iDLE-jAM | VCP 2, VCP 3 & VCP 4

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Reply
0 Kudos
billu
Contributor
Contributor

I tried the procedures in the kb article and still no joy. The commands to delete from the command line seem to have worked ( I googled the mode 8180 msg and vmware kb indicated it was nothing to worry about ), but the datastore continues to live. Rerunning the esxcfg-scsidevs –l command still shows it and so does the vSphere client. Below is a clip of the "deletion" session:

#fdisk /vmfs/devices/disks/naa.6a4badb048d9ab0013ae32070d86e445

last_lba(): I don't know how to handle files with mode 8180

You must set cylinders.

You can do this from the extra functions menu.

Command (m for help): d

Selected partition 1

You must set cylinders.

You can do this from the extra functions menu.

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

# vmkfstools -V

Reply
0 Kudos
billu
Contributor
Contributor

Further information:

I have removed the ESX host from vCenter, rebooted the ESX server, directly connected to the ESX server with vSphere client, and then tried deleting again. Here is the error:

Call "HostDatastoreSystem.RemoveDatastore" for object "ha-datastoresystem" on ESX "xxx.yyy.zzz.aaa" failed.

Thoughts? Suggestions??

Reply
0 Kudos
billu
Contributor
Contributor

Okay... I worked with VMware support this morning and the issue is solved, sort of.

It turns out that all of the issues are occurring because my underlying LUN size is greater than 2TB ( it is actually about 2.6TB ). When I first setup this server I configured the RAID Array into 2 virtual disks - one at 50GB to hold the console and the remainder to be the extra datastore, which is about 2.7TB. That second virtual disk is the datastore I am trying to work with.

I knew there was a 2TB limit, but thought that pertained to the datastore not the underlying drive. I was wrong.

The long term fix is to blow away the virtual disk and recreate multiple disks each 2TB or less.... which is what I am doing this afternoon.

Reply
0 Kudos
kleung
Contributor
Contributor

I have encountered this error as well. Namely:

Call "HostDatastoreSystem.RemoveDatastore" for object "ha-datastoresystem" on ESX "xxx.yyy.zzz.aaa" failed.

However my datastore is less than 2TB in size. What it turned out was beside of some VM still mounting some iso in that datastore, other VM CD/DVD drive setting is refering to iso in that datastore althought they are NOT CONNECTED.  Once that "Datastore ISO File" is being points to a different location that datastore will be able to free up and delete.

A list of VM still holding on to that datastore can be obtain from Home->Inventory->Datastores then Virtual Machines tab.

Reply
0 Kudos
dinxe
Contributor
Contributor

I'm having the same problem!! I can't see why I can't be allowed to delete the emty datastore!!??

It keeps giving me this error:

Call "HostDatastoreSystem.RemoveDatastore" for object "ha-datastoresystem" on ESXi "192.168.x.x" failed.

What shall I do?!?!?

Reply
0 Kudos
bbricker
Contributor
Contributor

kleung wrote:

I have encountered this error as well. Namely:

Call "HostDatastoreSystem.RemoveDatastore" for object "ha-datastoresystem" on ESX "xxx.yyy.zzz.aaa" failed.

However my datastore is less than 2TB in size. What it turned out was beside of some VM still mounting some iso in that datastore, other VM CD/DVD drive setting is refering to iso in that datastore althought they are NOT CONNECTED.  Once that "Datastore ISO File" is being points to a different location that datastore will be able to free up and delete.

A list of VM still holding on to that datastore can be obtain from Home->Inventory->Datastores then Virtual Machines tab.

Thanks, this was my problem as well and they were actually all templates. I converted them to virtual machines, edited each ones settings to change from using the ISO on the VMFS volume (which weren't even existent any more) to using client CD-ROM, and then I was able to remove the VMFS store. Perfect.

Reply
0 Kudos
xtropx
Contributor
Contributor

After hours of frustration on this problem and trying everything in every single google post I decided to do things the old fashioned way. I grabbed a copy of Gparted and tried that, and I am sure that would have worked except I couldn't get X (the GUI) to work right on my system.

http://www.partitionwizard.com/free-partition-manager.html

Partition Wizard Home Edition bootable ISO, deleted the partition on my disk, formatted it to NTFS, rebooted the ESXi machine and was able to re-create the datastore. I recommend trying this first:

http://www.blackmanticore.net/34ab986e70a5425091bcd879dfc98af4

However this did not work for me.

I created an account simply for this problem just because it took me about 4 hours to fix.

Please note you can enable SSH from inside ESXi 4.1 vSphere Client to try the black manticore link:

http://www.thomasmaurer.ch/2010/09/activate-ssh-on-esxi-4-1-via-vsphere-client/

Good luck!

Reply
0 Kudos
marshall28
Contributor
Contributor

thank you sooooo. much. I also struggled like you forever about 4 hours and finally came to your post at the end here and realized what a great idea it was. I booted to a windows 2003 cd and formated the drives ntfs which allowed me to then reformat them in the vsphere client to a vmfs datastore.

thanks

Reply
0 Kudos
zorkani
Contributor
Contributor

One of the reason behind this, is one of the guest vmware.log file remained in the datastore even after migrating it to another datastore. of course this log files is opened by the guest as far is the guest is up and running. The problem you may face is to find this guest that it's log file remained opened on this datastore. once you power off this guest, it will release the log file and create it on the original datastore. I have noticed these log files remained on the root of the datastore.

--Ashraf

Reply
0 Kudos
NTShad0w
Enthusiast
Enthusiast

Hi mates,

I observe same problem 1st time on my server but on ESXi v5.0 (migrated from ESX v4.1 where there was no such problem).

"Call "HostDatastoreSystem.RemoveDatastore" for object "ha-datastoresystem" on ESXi "x.x.x.x" failed.

Operation failed, diagnostics report: Sysinfo error on operation returned status : Busy. Please see the VMkernel log for detailed error information

"

it's a local storage (mpx...) on controller HP P800, sleeping every vm and going server in Maintenance Mode doesn't solve an issue, even server restart in Maintenance Mode dont help!!! Smiley Sad deleting from cmd/cli not solve the issue too (but it looks from cli like it delete a partition, but when I enter fdisk again partition still exeist!!),

so this a bug I think, VMware please look at this.

if someone want more info abouut HW/config please contact me or post question here.

1 time, after deleting it from CLI/CMD and then after going to Maintenance Mode I can delete datastore from GUI, but now I have second datastore (this time is build from 2 lun's) and I'm fighting with this from 2 hours and... nothing help Smiley Sad

So now I need help, Smiley Sad

In my opinion it's a bug in  ESX v5.0, can anyone help with this?

little update...

afer a lot of try... I found a solution, heeh it seems that there is a bug when you Migrate suspended vm from datastore You want to delete to new one, and somewhere in ESX or vm machine state there is still info about this old datastore, and it cant let you delete old free (empty) datastore UNTIL... you just run every suspended machines on the new one - vmware/esx probably update some machines/datastores info and let you delete these old empty datastore...:P - its a bug.

regards

NTShad0w

Reply
0 Kudos
zorkani
Contributor
Contributor

Hi, you better check the VMs stored on this DataStore or WAS stored on it. Then edit the "vm-name".vmx file. make sure of the attribute log.fileName that should refer directly to "vmware.log". IE, it has to be log.fileName = "vmware.log". (no path before vmware.log).

Ashraf

Reply
0 Kudos
NTShad0w
Enthusiast
Enthusiast

Hi zorkani,

I don't have such attribute in my vm's... (in .vmx file), do You think it may be a problem?

A lot of my vms are the old one, migrated/upgraded even from ESX v3.0/3.5 and 4.x but there are 30% of newest one created under ESXi v5.0 and they also don't have such attribute, should i add one?

kind regards

NTShad0w

Reply
0 Kudos
khalil53
Contributor
Contributor

Hei,

I am a bit late with my answer but better late than never Smiley Happy

I hade the same problem and I did everything that mentioned here without success.

Finally succeeded via iSCSI EqualLogic Group Manager GUI,

Deleted binding between Volume-Shadow and the iSCSI initiators in Access Control List under Access.

Then set the volume Offline, and finally deleted it. BINGO!

/Khalil

Reply
0 Kudos
Vikram11
Contributor
Contributor

I had the same issue where I could not delete the datastore.

"Call "HostDatastoreSystem.RemoveDatastore" for object "datastoreSystem-738" on vCenter Server "xyz.abc.org.au" failed."

Cause for me was - VDR. VDR was setup to snapshot machines which were on the datastore in question.

I had VDR Shutdown for few months and I migrated (vStorage Motion) few machines while VDR was shutdown.

So I think VDR was still looking at that Datastore and hence locking it up. I tried turning the VDR server on so that it can update the links but could not start it up.

As VDR was test only, I deleted it (not Ideal for Production Environment..) and tried removing the datastore. Worked fine

Regards,


Vikram

Reply
0 Kudos