4 Replies Latest reply on Apr 11, 2011 1:35 AM by bulletprooffool

    Should iSCSI have its own physical NICs?

    taylorb Expert

      Trying to decide what the best course of action is here.  I have about half my VMs on an HP Lefthand system.  I am (like most of us) running out of NICs on my VM host.   I have 4 gigabit ports left to share between the main internal networks and ISCSI.   I am trying to decide if I pair them on into two virtual switches, one with 2 links to the ISCSI VLAN and 2 links trunked with my internal VLANs.  Or should I lump them altogether in one big Vswitch with 4 links and use port groups?   See crude drawing below for the two options I am considering.    Any pros and cons would be helpful.  I know either way works, but I am just wondering what the most efficient setup would be as well as any pros/cons. 



        • 1. Re: Should iSCSI have its own physical NICs?
          a.p. Guru
          User ModeratorsvExpertCommunity Warriors

          IMO you should use dedicated NICs (unless you run 10 GBit/s) for iSCSI traffic. In addition to this, consult the Lefthand (sorry HP) best practices. You may get a better iSCSI throughput with two separate vSwitches (1 NIC each) for iSCSI.



          • 2. Re: Should iSCSI have its own physical NICs?
            depping Champion
            User ModeratorsVMware Employees

            I would recommend separating them on a vSwitch as well. Although technically speaking you can achieve the same with portgroups and setting nics in active / unused or active / standby from a management perspective it is easier when you separate them.



            Yellow-bricks.com | HA/DRS technical deepdive - the ebook

            • 3. Re: Should iSCSI have its own physical NICs?
              anjaneshbabu Novice

              Following various best practice  articles and bit of our own testing , we settled on a model as given in  the figure below. the highlights of the setup are

              - all nics are broadcom iscsi offload (Dependent HBA) type

              - seperate switches (physical) for storage and vm traffic

              - broadcom iscsi adapters are driven by built in s/w adaptors

              - vSwitches are setup for round robin load balancing for external Nics


              Few concepts we eliminated en route to our deployment

              - jumbo frames : not worth the effort and inconsistent performance with loss of service.

              -  shared switch with storage and vm traffic separated using vlans :   degraded performance since it is still within the same switch fabric

              - SFP+ connectors on server  : not sure about backwards compatibility with SFP slots.


              To  answer your query , yes it is imperative that you have seperate  dedicated nics for iscsi traffic. You could get away with vlans on a  single nic but that would not be ideal in a production environment.


              anjanesh babu


              • 4. Re: Should iSCSI have its own physical NICs?
                bulletprooffool Virtuoso

                As a rule, I always seperate storage traffic -whether this be iSCSI or NFS.

                Storage Nics invariably carry more traffic than any other NIC - meaning large disk read / writes can heavily affect your nbetwork bandwidth utilisation.


                As an add-on to seperating iSCSI, wherever possible, I also make my storage networks NON-routable - to even further isolate network traffic and also provide additional security.