My understanding of DEFRAG is that it moves blocks around on your disk in order to increase performance. ie. My file sits in 9 blocks sitting across the disk in random locations. Defrag would move these 9 blocks to create 9 contiguous blocks on the spindle. This allows the disk arm to move less, thus improving performance on the disk.
Your understanding WOULD be correct IF it applied to a physical machine. Remember that VM has a Virtual Disk. The file system inside that virtual disk is the same as any other hard drive. The VM's do not have acess to the underlying SAN file system or VMFS, so they are ONLY defragging the internal disk.
A VM is a file sitting on another file system. So it's a file, just like files on your computer. You can defrag files and you can defrag blocks. Since the VM can only defrag what it can see, the only thing it can see is the File System within the OS of that running VM. THOSE are the files / blocks that it can defrag.
So it's correct. You are thinking about the ESX server as a whole, but it's not taken as a whole, EACH VM is separate.
It's most certainly is NOT pointless to defrag.
What about when using say something like WAFL and thin provisioning and working from snapshots of base disk's?? Starts to get a bit more confusing...
How so? Those are SAN utilities. It's not confusing you are over thinking this.
A SAN is a set of disks. It can take several disks and parse those out to other systems. Those other systems utilize that data at a LUN (block level) or existing file system (NFS) level. EACH system then can carve up a slick of that space allocated to a VM. That VM can only see a portion of that disk, but a VM uses it's OWN system.
You drop a zip file on that SAN, that zip file may be fragemented, but that underlying file system is fine. You can simple defrag that zip and leave the rest of that drive alone. Same way with a VM. A VM has it's own container, within that container it's formatted with the VM OS (ext3, zfs, NTFS, etc.). So your Windows machine sees an NTFS file system that IT can access and defrag, but it's self contained, and within a file sitting on your SAN. WAFL defrags (scrubs) the disks which is separate from the contents of those disks. That's a low level function maintained separately.
Since the whole VMDK is not read into RAM, the physical heads would have to move around within that file to read and write the data wouldn't they? So defragging could be good in that case.
Yes that's just about it. Defrag is good, it's rearranging files within the VM (vmdk) which is accurate.
Or, is the VMFS file system smart enough to know how to write the file in a way that it is not fragmented at that level?
The VMFS is agnostic, it has NO idea what's going on WITHIN that vmdk. VMFS holds a series of VMDK files. That VMDK can become fragmented, and does because Windows file system can be fragmented. So think of VMFS as a repository. You put objects into that repository, it's not going to analyze each and every one to ensure they are organized, that would be too much overhead.
So VMFS can ALSO be fragmented, but there is no tool to defragment VMFS or ext3 file systems currently. Which is where WAFL comes in, it can scrub and clear blocks of data beneath VMFS at the hard drive level, which is invisible to VMFS, so there are 3 or 4 layers of visibility here.
Hard drives can be grouped to contain data, those hard drives can be RAID. RAID can hold NFS / LUN data. Within LUN/s you can have OS file systems (VMFS). Within VMFS you can FURTHER have vmdk files. Within THOSE files you can have OTHER file systems. At each layer of access they can be managed separately and independently. Depending on what you use to manage those hard drive/ raids / file systems.
Defragging increases performance by decreasing DISK ARM travel time. If I can arrange blocks next to each other on a spindle, then the disk arm can read contiguous blocks without jumping all over the disk.
Still thinking in terms of physical realm. Virtual realm works exactly the same. You can defrag files, not even talking about blocks of data. Defragging a single SQL database can improve performance, so that in and of itself proves my point. Nothing about ARM travel or any of that physical stuff, it's about IO, the more time spent gathering data from a non-continguous piece of data, is LESS time able to do real work elsewhere, period. It's not any simpler than that.
If it takes more IO to read/write a file because it's strewn all over your datastore/file system/drive it's going to cause the hard drives to work harder. Making your hard drive work harder for even 1 VM is going to affect your ENTIRE RAID.
But defragging the VMDK does not decrease disk arm travel time because its HIGHLY abstracted from the vmdk.
Therefore my above statement subjugates this statement, and it's completely false. Defragging DOES and WILL improve overall performance, don't care if it has over all effect on the armature or not. defragging is about performance, virtual, physical or otherwise. 1 IO spent reading a file is 1 MORE IO not reading some thing else.
But SANs respond to scsi commands, read/write blocks. A defrag at the san level is going to look like a bunch of read/writes which will just move the blocks around to different locations.
Ah ha that's where this 'theory' breaks down right there. Cache. Familiar with that? NOT ALL reads come direct IO from SAN, it could be a cached read / write. A file sitting on a fragmented file system (i.e. VMFS) is still defragmenting, at least at the FILE level. It's still MORE efficient than leaving it fragmented.
There isn't a performance hit for virtual scsi commands going to a vmdk because there arent disk arms in a vmdk. The performance hit wil come from the virtualization layer when this scsi command is turned in to a physical fc/iscsi command and sent to the disk array.
That's not true, there is a performance hit. Find a VM, check it for fragments, run a benchmark. Defragment it using a highly recommended defrag like diskkeeper, run benchmark again. Improvement? Yes.
But SANs are built to handle the placement of blocks efficiently without any work from higher level file systems/operating systems. Is this incorrect?
Correct, that's incorrect 
SAN's do not manage DATA, they manage DISKS. What YOUR systems do with DATA is up to your systems, the SAN will not and can not efficiently move data within a file system contained in a file. If that file occupies 50GB of space on the SAN, to the SAN that's just a really large block of data. WITHIN that large block of data, since it's not visible to the SAN architecture, that data COULD and probably IS fragmented. So you need to defragment that large chunk of data, otherwise the SAN is inneficient. Your over all IO will suffer, I promise you.
simple diagram to explain.
Take 2 pieces of paper, put them on your desk.
On the first piece of paper, write the numbers 1 thru 10. Write them in sequence at the top left corner of the paper.
On the second piece of paper write the numbers 1thru 10, this time spread them out, out of sequence all over the page as far apart as possible.
The paper represents a vMDK file, the desk represents the SAN RAID. You can MOVE that paper as a unit to anywhere on the SAN, but when you add contents to that paper, how would the SAN know what you are writing on that paper? It can't, and it doesn't. Now on that first piece of paper wirte a letter of the corresponding letter for that number beneath the number. A=1, B=2, etc..
do the same thing to both pieces of paper. Now how long does it take to write paper 1? Second paper takes significantly longer doesn't it? Now I know that computers can go MUCH faster but that explains what I am talking about. The paper sits on a RAID, the SAN only manages that paper not the CONTENTS of the paper. You can still defrag the CONTENTS to make your programs / data more efficient.
That's what I am talking about.
ejward, you're almost there
Defragging allows your OS to move the blocks closer together so the physical heads dont have to move around so much, thus increasing performance. This falls apart in a SAN environment since not even ESX knows EXACTLY where the blocks are placed. ESX can see a LUN on your san, which is a virtual SCSI hard disk. If you create a 1TB LUN on your san, your SAN will carve out 1TB of blocks and generate a "virtual disk". This virtual disk, or LUN, is presented to your physical host. In our case, the ESX server now thinks it has a 1TB SCSI disk connected to it. ESX sends a SCSI WRITE command to the 1TB disk, stating it wants to store data in block #1. This is written in VMFS so it knows where to find the data later for a read. But when this command reaches the SAN, your san controllers are responsible for the logical mapping between the LUN blocks, and the physical blocks on the disks. This mapping can place data all over your spindles, regardless of where vmfs thinks the data lives in its vmfs formatted lun.
This is why I think defragging is pointless in a guest. You have no way to control where your san is going to place the data. And since defragging increases performance by placing blocks next to each other, there is no way for this to occur since the physical spindle HEADS and ARMS are so abstracted from the guest living MANY layers above.
Granted, In windows you will see a beautiful defragmented disk in the defrag GUI. But did performance actually increase? The blocks are still scattered about the physical spindles many layers below.
I contend that defragging serves no purpose in a SAN environment. As you previously stated, defragging moves file blocks around so the physical heads do not have to jump around the disk. But in a virtual environment, there are NO physical heads.
Uh wrong... Your data is spread out over that SAN. There is STILL some movement with regards to reading / writing data. So organizing the data so that it's sequential still involves going from place to place on a hard drive, in some manner.
So inside that file, which is occupying hundreds perhaps thousands of blocks of data, still means that hard drive and RAID must access the data spread out all over that block of data. So if your RAID is 10' wide, and your VMDK occupies 3' of that RAID, and within that VMDK your data occupies fragmented information, reading from 1" all the way over to 3' for the rest of the data REQUIRES movement by the disks still. Just because the Virtual Machine has no moving parts, doesn't imply that there isn't movement at another level.
So in essence you still must move that arm in the hard drive somewhere to get TO the data elsewhere on that RAID, it's just confined to that single vMDK.
Multiply this by 30 VM's and you can see where this can cause IO issues, and thereby causing immense performance problems IF you don't defragment that data.
In Windows when you do a defrag on NTFS, Windows copies the blocks to a new location then updates the file allocation table that points to those blocks. Doesn't VMFS have something like that?
no it does not. VMFS is like a cookie sheet, you put cookies anywhere on that sheet. You can move them around. But suppose you want chocolate chips in the cookies. You think the cookie is going to care how many chips are in each cookie? no. Do you think the sheet is going to care how they are inside the cookie? No because it can't 'see' the cookies, it only knows there are a few of them,and VMFS does NO defragmentation.
It has a FAT table of sorts, but that data IS still and can be fragmented. VMDK files can fragment, because data isn't moving to make other data contiguous, SANS don't work like that. If you have 50GB of space, and you put 2 VM's in there of 20GB each, then you delete one. You have 20GB free, 20GB occupied by a VM, and 10GB free right?
What happens when you add another VM of 30GB? Where does the data go? that's right 20Gb before the existing VM, and 10GB AFTER hence fragmentation. NO SAN is going to FIRST move that data just to make the data sequentially, that's a LOT of overhead to maintain, and they do not do this. VMFS is just space, it's not managed by ANY fragment prevention.