9 Replies Latest reply on Oct 18, 2012 4:00 AM by jklick

    Per-VM I/O Monitoring On NFS Datastores

    deltajoka Novice

      I realize that this issue has been up on the forums before, but no real alternative solution has been posted. So the purpose of this post is to hopefully stir up some discussion.

       

      We have recently migrated from local VMFS datastores to NFS datastores on a IBM System Storage N3300 Model A20 filer (basically a rebranded NetApp). It is no longer possible to get statistics of per-VM I/O in vCenter or by using esxtop on the ESX hosts. We have filed a Support Case to VMware about this, but there is apparently no solution on the VMware side of things. VMware just refers to monitoring this on the NFS filer side. On the IBM filer, we can only monitor I/O per volume (and having a volume per VM is just not feasible). There is no way (that we know of) to monitor I/O per file on the NFS filer.

       

      We think this is a huge drawback for NFS datastores (which is quite popular). Quite frankly, we are surprised that VMWare don't seem to be interesting in implementing this.

       

      Has anyone managed to implement an alternative solution to monitor this? We are currently trying to build a solution that promiscuously captures NFS traffic on the storage network with tools like tshark/wireshark. But progress is slow due to the hassle of tracking back filehandlers to specific files and directories.

       

      Any constructive input is most welcomed!

       

      Regards,

       

      Johan Karlsson

        • 1. Re: Per-VM I/O Monitoring On NFS Datastores
          frank_wegner Master
          VMware Employees

           

          > We think this is a huge drawback for NFS datastores (which is quite popular). Quite frankly, we are surprised that VMWare don't seem to be interesting in implementing this.

           

           

          AFAIK, VMware does look into this topic very deeply right now. It has been recognized as a missing feature. And I see more and more customers move towards NFS-based storage for various reasons. I am not sure when exactly VMware will actually ship a solution, but I'd suggest raising this topic whenever you talk to a VMware representative (Sales, Professional Services, whatever). And I wish it would be sooner than later. You can help driving the priorities. Until then I guess we have to live with workarounds.

           

           

          1 person found this helpful
          • 2. Re: Per-VM I/O Monitoring On NFS Datastores
            astorrs Enthusiast

             

            I had a conversation last month at VMworld with John Blumenthal about this exact issue. He mentioned that VMware is well aware of customer concerns in this area and is working with some of the vendors (presumably NetApp and EMC, perhaps others) - but the implication was that any solution is >12 months out.

             

             

            The best I've come up with is monitoring I/O per volume and watching which host is talking the most to a specific NFS export, anything more like you said requires wireshark. I greatly miss esxtop everytime I'm troubleshooting an NFS environment.

             

             

            • 3. Re: Per-VM I/O Monitoring On NFS Datastores
              deltajoka Novice

              Thanks for the replies!

               

              Good to know that VMware is aware of the issue and seem to be working on a solution at least. Too bad that it seems so far away though.

               

              I've been able to trace the file handlers used in NFS Lookup request/replies to specific files/directories (and thus determine the VMs). I've managed to filter out packets matching these file handlers with tshark in a way that i can see (and measure) the specific traffic when a VM powers on/off. However, further operations to the "disk" in the OS of the VMs don't use these file handlers. I can't trace the actual I/O, because most of that traffic is NFS FILE_SYNC packages (which have no human readable payload displaying file or directory, like the lookup packages have) and they use different file handlers than the NFS lookup packages. Too bad. Maybe there are some NFS gurus around that can nudge me in the right direction? Or suggestions about other possible workarounds?

              • 4. Re: Per-VM I/O Monitoring On NFS Datastores
                karunk Lurker

                Hi

                 

                Is there any methodology to identify the I/O intensive VMs that were running on NFS Datastores or on both NFS and VMFS datastores.If there exists something plz let me know.

                 

                Thank you

                • 5. Re: Per-VM I/O Monitoring On NFS Datastores
                  rickardnobel Virtuoso

                  kmaster wrote:

                   

                  Is there any methodology to identify the I/O intensive VMs that were running on NFS Datastores or on both NFS and VMFS datastores.

                   

                  You should be able to analyze this through ESXTOP in the "v" view to see the IOs per VM.

                  • 6. Re: Per-VM I/O Monitoring On NFS Datastores
                    karunk Lurker

                    Thanks but How do we get those results in csv file. In bash mode I was able to generate a csv file but it was giving all the fields approximatly 1000, where it was highly difficult to get those fields which belong to VM's IOPS. If there is any possibility to get it plese let me know.

                     

                    thanks

                    • 7. Re: Per-VM I/O Monitoring On NFS Datastores
                      mcowger Champion

                      Have you considered vCenter Operations?

                      • 8. Re: Per-VM I/O Monitoring On NFS Datastores
                        durganet Novice

                        That sounds like a Job for esxplot !

                        • 9. Re: Per-VM I/O Monitoring On NFS Datastores
                          jklick Enthusiast

                          The good news is that as of vSphere 4.1, you should be able to see most - if not all - the IO data for NFS datastores inside vCenter. However, even then, I think we're all aware vCenter isn't the easiest solution for aggregating/monitoring storage performance data. These are probably the reports you're looking for:

                           

                          DatastorePerformance.jpeg

                          DatastoreDrillIn.jpeg

                           

                          If so, the latter is simply a drill-in from the former and both are default functionality in VKernel vOPS (feel free to abuse a 30-day free trial). Additionally, keep an eye out for a free tool in the near future which will also help tackle this problem area.

                           

                          Full disclosure: I can't forget to let everyone know I'm a VKernel employee or else the powers that be will be unhappy with me.