hostasaurus's Posts

For sure; if we couldn't live migrate, it would be a nightmare dealing with this.  We can do our shrinking without taking the guests offline.  They've really dropped the ball on this issue, and I... See more...
For sure; if we couldn't live migrate, it would be a nightmare dealing with this.  We can do our shrinking without taking the guests offline.  They've really dropped the ball on this issue, and I've gotten no hint of anything in the pipeline to fix it.
It certainly is a common request.  Only logic I can see behind them doing nothing for years and years now is the fact that EMC bought them; maybe they want people to waste tons of disk space so t... See more...
It certainly is a common request.  Only logic I can see behind them doing nothing for years and years now is the fact that EMC bought them; maybe they want people to waste tons of disk space so they buy more storage.  We decided to task one of our staff with zeroing and live migrating forward and back bloated VM's every few months and it never fails to recover several terabytes of wasted space on our arrays. That is one of the few nice things about the horrendous new 5.5 interface; on the Home -> vCenter -> Virtual Machines screen, it gives you a chart of all of your machines and two of the columns are provisioned space and used space, and you can sort it by them.  Makes it very easy to spot the bloated guests to target for cleanup.
I've never heard that expression he used but he gave the command he used to accomplish it with the offline vmdk: vmkfstools -K someserverdisk.vmdk
The option to disable mac changes won't help because this is not what the guests are doing.  Basically what I'm wanting to prevent is a compromised server (guest) from sending out fraudulent arp ... See more...
The option to disable mac changes won't help because this is not what the guests are doing.  Basically what I'm wanting to prevent is a compromised server (guest) from sending out fraudulent arp responses or gratuitous arp packets for ip addresses that are not on that guest.  It's not changing it's mac, it's just trying to announce it's mac for ip's it doesn't have.  vmware tools has knowledge of every ip address configured on the guest OS, so I was hoping there'd be some automated way to let vsphere block what vmware tools would know to be bogus arp's.
Thanks; I took a quick glance at vShield but I don't think it would work.  It comes close in that it lets you segment traffic, but I've effectively already done that by putting different machines... See more...
Thanks; I took a quick glance at vShield but I don't think it would work.  It comes close in that it lets you segment traffic, but I've effectively already done that by putting different machines on different vlan's, so when one causes a problem, it is at least limited to that vlan.  It also appears to let you firewall down to the vm level, but I'm not sure if the rules could be granular enough to accomplish what I need, and even if it could, it would likely be a very heavy administrative burden; i.e. I would need it to support adding a rule to every guest that states "if a gratuitous arp is generated by this VM that announces it has IP addresses other than X, Y and Z, drop it."  With that much work, I'd probably be better off just hard coding the arp table on the routers since I can automate that. I'm going to investigate vshield further with our sales rep though just to make sure.
Hi all, in a physical world, there exists technologies on the switching side to help deal with arp poisoning, such as Cisco's dhcp snooping and dynamic arp inspection.  We can of course protect a... See more...
Hi all, in a physical world, there exists technologies on the switching side to help deal with arp poisoning, such as Cisco's dhcp snooping and dynamic arp inspection.  We can of course protect all servers, physical and virtual, from the effects of attempted arp poisoning of the router mac by hard coding the router's mac address on each system.  What I'm really after though is a way to prevent a VM from sending malicious responses to broadcasts from the router looking for the MAC of a given IP; i.e. vm1 has mac 0011.2233.4455 and IP 192.0.2.1, vm2 has mac 2233.4455.6677 and IP 192.0.2.2.  vmware tools are running and vmware knows this.  I want to prevent vm2 from sending a response to a broadcast from the router for who has 192.0.2.1 with its mac address.  I could of course hard code the arp table on the router to protect router to vm communications, but we're talking way too many addresses to do that to begin with, and it still wouldn't protect other intra-vlan communications between systems on that vlan. I'm running 5.5 enterprise plus.
Found the issue; the version of RTOOLS on the server was still 5.8 and it needed to be updated to match the PowerPath/VE version.  Issue resolved.
Cross posted this to EMC forums but they tend to be a lot less active.  Will post a followup here if I find the answer elsewhere first. I've got a vCenter 5.5 build 1378903 install and a host ... See more...
Cross posted this to EMC forums but they tend to be a lot less active.  Will post a followup here if I find the answer elsewhere first. I've got a vCenter 5.5 build 1378903 install and a host I just upgraded to 5.5 build 1331820.  I did the upgrade by putting it in maintenance mode, detaching my PowerPath/VE 5.8 extension group from the host, attaching my new PowerPath/VE 5.9 group to the host, disabled lockdown, applied all outstanding ESXi 5.1 patches and reboot, applied the PowerPath/VE 5.9 patches and reboot.  I used 'rpowermt check_registration' to verify it was okay and it seems to think it is. However, having not taken it out of maintenance mode yet for fear something is wrong, I checked the EMC VSI tab in my vSphere Client and it reports this at the top of the screen in a bright color: The PowerPath/VE service is not managing any LUNs on this host. If I use 'rpowermt host=192.0.2.1 display dev=all' it seems to think things are working: Pseudo name=emcpower1 VNX ID=1234 [LUN1] Standard UID=naa.1234 [BLOCK_LUN_20] state=alive; policy=CLAROpt; queued-IOs=0 Owner: default=SP A, current=SP A       Array failover mode: 4 ============================================================================== --------------- Host ---------------   - Stor -  -- I/O Path --   -- Stats --- ###  HW Path               I/O Paths    Interf.  Mode     State   Q-IOs Errors ==============================================================================    1 vmhba32                C1:T6:L0    SP B4    active   alive      0      0    1 vmhba32                C0:T7:L0    SP B5    active   alive      0      0    1 vmhba32                C1:T4:L0    SP A4    active   alive      0      0    1 vmhba32                C0:T5:L0    SP A5    active   alive      0      0 I would assume the output would have been empty if PowerPath was not actually managing I/O to that LUN? On the Configuration tab, Storage, 'Devices' button, pick one; in the lower window "Device Details" it shows owner as PowerPath. My other PowerPath/VE 5.8 hosts on the EMC VSI tab continue to state "Found PowerPath/VE version 5.8 (build342) installed on this ESX host..." I dont want to take it out of maintenance mode and risk live guests moving back to it if something is actually wrong with PowerPath.  Anything else I can check?
Any chance you've found a way to move Tasks and Alarms back down to the bottom so I can actually read them along with what is currently in the center panel without constant shrinking/expanding or... See more...
Any chance you've found a way to move Tasks and Alarms back down to the bottom so I can actually read them along with what is currently in the center panel without constant shrinking/expanding or side scrolling sub-windows?  So far really hating this interface.
Upgraded the test lab to 5.5 today to check out the new features, only to find out that to make use of the new features, you have to use the web interface; the old client only gives you the same ... See more...
Upgraded the test lab to 5.5 today to check out the new features, only to find out that to make use of the new features, you have to use the web interface; the old client only gives you the same features you already had accessed to and none of the new ones.  The web interface conveniently requires Adobe Flash to use, right after we mandated pulling Flash off the internal management computers due to Adobe's horrible track record on security in Flash.  Thanks VMware... :smileyangry:
The fact that it's trying to download some of the things in my list of pre-reqs makes it sound like you may have skipped that part?  UUID for example.  You need to start the cpan shell and instal... See more...
The fact that it's trying to download some of the things in my list of pre-reqs makes it sound like you may have skipped that part?  UUID for example.  You need to start the cpan shell and install all the things in the list I gave or the vCLI will fail to install.
Hi all, we recently replaced an EMC CX4-480 with a VNX5500 and saw a perceptible improvement in the performance of all our VM's.  I now have 200 gigs of fastcache that I'm about to enable but bef... See more...
Hi all, we recently replaced an EMC CX4-480 with a VNX5500 and saw a perceptible improvement in the performance of all our VM's.  I now have 200 gigs of fastcache that I'm about to enable but before doing so, I'd like to collect some data to determine what kind of effect it has.  Are there any whitepapers, forum posts, etc. that would give me some guidelines on what data I should collect in vsphere before and after to see if I can spot improvement, such as latency figures, etc.? Thanks, David
Ah, yep, it was a few stuck instances of install/upgrade vmware tools that never completed.  Thanks!
Hi all, I'm trying to retire a LUN and the storage it sits on but have several guests that are showing as being on that datastore in vSphere.  The weird thing is they all show 0.00 bytes used spa... See more...
Hi all, I'm trying to retire a LUN and the storage it sits on but have several guests that are showing as being on that datastore in vSphere.  The weird thing is they all show 0.00 bytes used space, and if I browse the datastore there are no folders related to those guests.  I do believe the guests in question may have had their primary storage on that datastore at some point in the past.  I successfully storage vMotioned the guests in question from one datastore to another but all that moved were the real files associated with the guest from the datastore they're really on to the new datastore I moved them to, while the presence on the problem datastore remains. Any idea what I can look at to get them off of it?  I'm scared to delete the datastore if there's something really there that I can't see. Thanks, David
Not that it will be any easier, or less annoying, but if you'd like to avoid taking your machine offline to run the vmdkfs stuff, you could set up a linux box as an iscsi server, hook your vmware... See more...
Not that it will be any easier, or less annoying, but if you'd like to avoid taking your machine offline to run the vmdkfs stuff, you could set up a linux box as an iscsi server, hook your vmware host to it and format it as VMFS v3 with a different block size than your production disk, then storage vMotion the VM to it and back; that will shrink it.  Like I said, just as annoying and tedius, but it avoids an outage if you need to.
Yup, unfortunately that's the only way.  The zero'ing the space does make sense so I'm not holding that against them.  The underlying operating system (ESXi) would have no way of knowing what is ... See more...
Yup, unfortunately that's the only way.  The zero'ing the space does make sense so I'm not holding that against them.  The underlying operating system (ESXi) would have no way of knowing what is in use or not in use within the guest's filesystem in the VMDK, so setting all the free space to zero is the only way to give it an indicator it can then see to be able to shrink.  However, we're about 4+ years over due for a way to do this that doesn't involve vmotion'ing the guest to a VMFS of a different block size, or doing an offline operation; that's simply ridiculous, especially in large environments. Maybe storage vendors are paying them to not implement the feature lol.
Doh!  That was stupid of me, I breezed through the volume create wizard so many times setting up the new LUNs that I completely overlooked the fact that you're allowed to still create VMFS v3 vol... See more...
Doh!  That was stupid of me, I breezed through the volume create wizard so many times setting up the new LUNs that I completely overlooked the fact that you're allowed to still create VMFS v3 volumes.  I'll set one up just for this maintenance task and problem solved. Thanks! (Still wish there was a specific shrinking process that could be run on a live vm)
Do you know if there's a way to create one from vCenter/ESXi 5.1?  Or do I need to install a 'new' 4.1 server and hook it to the san just to create the file system? Thanks!
Hi all, I'm trying to find a way to shrink bloated thin-provisioned guests in an all VMFS5 environment. We upgraded from 4.1 to 5.1 and then also storage vmotioned all of our vm's to newly cre... See more...
Hi all, I'm trying to find a way to shrink bloated thin-provisioned guests in an all VMFS5 environment. We upgraded from 4.1 to 5.1 and then also storage vmotioned all of our vm's to newly created VMFS5 volumes before deleting the old VMFS3 volumes.  While doing this, everyone forgot that one of our volumes was intentionally set to a differnet block size than all the others for the purposes of shrinking guests that had ballooned up for whatever reason.  Now we have no way to create anything but a 1 MB block size volume which prevents us from doing the cleanup routine.  Is there any alternative way to shrink a guest that does not involve taking it offline? For those not familiar with the procedure I'm describing, here's how it works: Say you have a 100 GB guest that only has 10 GB of real data in it  but due to something having gone wrong, temporary files, clean-up of old data performed,  etc., that 100 GB had been used at one point by real files but is now only 10 GB in use. You would like to recover what is now 90 wasted gigs of space since your thin-provisioned VMDK is sitting there at 100 GB in size.  With VMFS3, you could create a volume with a  block size that was different than the volume the guest currently lives  on.   In our environment all of our LUN’s were 1 MB except for one we  kept at 2 MB just for this purpose.   When we were getting a bit low on  space and found guests like the one in question, we’d run a simple  command to write zeroes to all of the free space: cat /dev/zero > bigfile ; rm -f bigfile Then all you have to do is storage vMotion the guest to the 2 MB  volume and back to its normal home and suddenly the VMDK goes from 100  gigs down to 10 gigs, all without any impact on the guest.  Without support for creating a file system with a different block  size in VMFS5, now I am not aware of a way to reclaim this space in a manner that doesn’t  require an outage for the guest.
You need to zero the freespace in the guest filesystems, then move to a filesystem with a different block size when going from thin to thick and back or it won't recover the space.