Pool deletion deleted another VM

BarryUWSEFS · ‎05-19-2020

Horizon 7.11 linked clone pools.

This morning I deleted a retired pool in the Horizon Administrator. I checked later and indeed it had been cleanly deleted.

I later discovered that the Parent VM of a different pool was missing. A search of the event logs revealed that the parent VM was deleted at the same time as the VM's in the pool. It was not even the Parent for the deleted pool.

Template	: False
Key	: 27993157
ChainId	: 27993153
CreatedTime	: 5/19/2020 7:21:13 AM
UserName	: XXXX\Administrator
Datacenter	: VMware.Vim.DatacenterEventArgument
ComputeResource	: VMware.Vim.ComputeResourceEventArgument
Host	: VMware.Vim.HostEventArgument
Vm	: VMware.Vim.VmEventArgument
Ds	:
Net	:
Dvs	:

FullFormattedMessage : Removed Staff-1909-M on 172.25.xxx.xxx from Madrona

I also noticed a replica was deleted. The only thing these VM's had in common is that they are all on a Nutanix cluster.

Does anyone have a clue how this could happen?

Shreyskar · ‎05-19-2020

Hi BarryUWSEFS

Deleting a desktop pool will never delete master/gold image. It must have been done either manually Or may be during any cleanup done on vCenter.

You need to review vcenter server and host logs during that timeperiod to find out the possible reason.

BarryUWSEFS · ‎05-19-2020

I know it seems impossible. The deleted VM that was deleted was a master for a different pool altogether. Check this query and you can see the obvious VM's from the pool that were deleted, the the staff-1909-M and a couple replicas. Look at the time stamps. I marked it in red towards the bottom.

$start = (Get-Date).AddDays(-6)

Get-VIEvent -Start $start -MaxSamples ([int]::MaxValue) |

where{$_ -is [VMware.Vim.VmRemovedEvent]}

Template : False

Key : 27992993

ChainId : 27992989

CreatedTime : 5/19/2020 7:16:42 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-016 on xxx.25.182.204 from Madrona

ChangeTag :

Template : False

Key : 27992995

ChainId : 27992990

CreatedTime : 5/19/2020 7:16:42 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-003 on xxx.25.182.204 from Madrona

ChangeTag :

Template : False

Key : 27992997

ChainId : 27992991

CreatedTime : 5/19/2020 7:16:42 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-004 on xxx.25.182.204 from Madrona

ChangeTag :

Template : False

Key : 27993024

ChainId : 27993016

CreatedTime : 5/19/2020 7:17:13 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-009 on xxx.25.182.202 from Madrona

ChangeTag :

Template : False

Key : 27993029

ChainId : 27993019

CreatedTime : 5/19/2020 7:17:14 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-018 on xxx.25.182.202 from Madrona

ChangeTag :

Template : False

Key : 27993035

ChainId : 27993031

CreatedTime : 5/19/2020 7:17:15 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-015 on xxx.25.182.204 from Madrona

ChangeTag :

Template : False

Key : 27993037

ChainId : 27993030

CreatedTime : 5/19/2020 7:17:15 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-017 on xxx.25.182.204 from Madrona

ChangeTag :

Template : False

Key : 27993040

ChainId : 27993038

CreatedTime : 5/19/2020 7:17:16 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-005 on xxx.25.182.204 from Madrona

ChangeTag :

Template : False

Key : 27993043

ChainId : 27993041

CreatedTime : 5/19/2020 7:17:18 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-019 on xxx.25.182.204 from Madrona

ChangeTag :

Template : False

Key : 27993063

ChainId : 27993061

CreatedTime : 5/19/2020 7:17:45 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-022 on xxx.25.182.202 from Madrona

ChangeTag :

Template : False

Key : 27993068

ChainId : 27993066

CreatedTime : 5/19/2020 7:17:50 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-024 on xxx.25.182.204 from Madrona

ChangeTag :

Template : False

Key : 27993081

ChainId : 27993075

CreatedTime : 5/19/2020 7:17:55 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-025 on xxx.25.182.203 from Madrona

ChangeTag :

Template : False

Key : 27993085

ChainId : 27993082

CreatedTime : 5/19/2020 7:17:57 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-021 on xxx.25.182.201 from Madrona

ChangeTag :

Template : False

Key : 27993087

ChainId : 27993083

CreatedTime : 5/19/2020 7:17:57 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-020 on xxx.25.182.203 from Madrona

ChangeTag :

Template : False

Key : 27993103

ChainId : 27993101

CreatedTime : 5/19/2020 7:18:29 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Staff-023 on xxx.25.182.204 from Madrona

ChangeTag :

Template : False

Key : 27993152

ChainId : 27993106

CreatedTime : 5/19/2020 7:21:11 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed replica-bd5bb126-58c0-4bdd-ad00-4739b4ea0e8d on xxx.25.182.203 from Madrona

ChangeTag :

Template : False

Key : 27993157

ChainId : 27993153

CreatedTime : 5/19/2020 7:21:13 AM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed Staff-1909-M on xxx.25.182.202 from Madrona

ChangeTag :

Template : True

Key : 27838858

ChainId : 27838858

CreatedTime : 5/14/2020 11:47:51 PM

UserName : User

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed FV-Student-12 on vxrail4.xxx from Madrona

ChangeTag :

Template : False

Key : 27831062

ChainId : 27831016

CreatedTime : 5/14/2020 7:33:13 PM

UserName : xxx\Administrator

Datacenter : VMware.Vim.DatacenterEventArgument

ComputeResource : VMware.Vim.ComputeResourceEventArgument

Host : VMware.Vim.HostEventArgument

Vm : VMware.Vim.VmEventArgument

Ds :

Net :

Dvs :

FullFormattedMessage : Removed replica-a47a5597-8b03-4ee0-9c23-05a8152c5b9d on xxx.25.182.204 from Madrona

ChangeTag :

PS C:\WINDOWS\system32>

BarryUWSEFS · ‎05-21-2020

I opened a VMware support case. Unfortunately the first tier Horizon tech was pretty unhelpful, and never even recognized my question of how this could happen. He escalated to a vSphere engineer who confirmed the VM was not recoverable, but not being a Horizon engineer could not speak to why this happened. The case is still open so if I get any answer I will update, otherwise this will go down as the strangest mystery in my years of Horizon/View administration.

sjesse · ‎05-21-2020

xxx\administrator, are you using a special account for horizon or using a generic administrator account? I'm not saying one thinig or another happened, but if xxx\administrator is not dedicated to horizon it may hinder your attemtps in finding the problem as you can't really tell where it came from, only assuming it was the pool deletion.

BarryUWSEFS · ‎05-21-2020

I was logged into Horizon Administrator with a Windows domain account when I deleted the pool. xxx\administrator is the vCenter SSO. I assume Horizon invokes the vCenter account for the actual VM removal.

I am the only person on our team who supports VMware/vSphere/Horizon so no one else would have been involved.

Thanks!

sjesse · ‎05-21-2020

That's my point, with it being the SSO administrator account you can never be 100% sure, I'd suggest changing that to a service account. Mine has view right in the name, and appvol in the name for appvolumes. That way we cant track what those accounts do it cases like this. Anything Horizon does is through the account you give it, and it will help your case with GSS when the events show that account doing it, not an account someone may be using in general.

BarryUWSEFS · ‎05-21-2020

I see, so when adding the vCenter to Horizon, create a special account, good idea. I see that now in the documentation, thanks. I do still believe however, since I am the only person who does anything in vSphere/Horizon, the command to delete the pool in Horizon, nevertheless, also deleted this other VM, for whatever reason. Do you think using a dedicated account would have prevented this? Or still important, just made it easier to analyze? Thanks for your input, much appreciated!

sjesse · ‎05-21-2020

Possibly I'd need to check the permissions VMware recommends, but if you have your parent images in a different folder which that account can't delete it shouldn't be possible

BarryUWSEFS · ‎05-21-2020

That sounds reasonable. The images may have been in the same folder. I had created a brand new image and pool for the user group "staff" and it was when I deleted the old staff pool that the new VM image was also removed. Being in the same folder was really the only thing I can think of they would have in common. Still does not make sense, but you have given me some good ideas to further lock down Horizon.

All

Pool deletion deleted another VM