10 Replies Latest reply on Sep 26, 2006 7:56 AM by DougBaer

    pSwitch Network redundancy for VMs?

    DougBaer Master
    VMware Employees

      In the neverending quest to eliminate single points of failure, I've got a question about utilizing multiple switches and how to best configure vSwitches and pSwitches.

       

      If I've got 4 NICs (aside from the 2 for SC and VMotion) and 2 pSwitches, in ESX 2.5.x, I would have bonded the NICs into two sets of two, put one set into each pSwitch and attached both bonds to a vSwitch.

       

      I've read that ESX 3 automatically teams/bonds the NICs associated with a vSwitch. How smart is ESX in this case? Will it/Can it figure out how the NICs are paired?

       

      In my case, it seems like I'd have to associate all 4 NICs with the vSwitch and hope that ESX figures out how they're paired. Will it do that properly? Aside from configuring compatible load balancing on the pSwitches and vSwitches, is there anything else I need to do?  So far, documentation on this has been hard to come by, and /etc/vmware/hwconfig doesn't exist in ESX 3.0.

       

      I will have the ports bonded in a static 802.3ad group on each pSwitch side, unless breaking them up and leaving them as 4 independent NICs makes more sense.

       

      Thanks!

        • 1. Re: pSwitch Network redundancy for VMs?
          wunderon Hot Shot

          I'd say for each vswitch, select one nic connected to each physical switch, and assign them both to the vswitch. You can configure them as both active, or active and standby.

           

          Some will combine the vmkernel and Service Console onto a single vswitch with a single NIC, depending on SC loading and vmotion(or NAS/iSCSI) loads.  This would leave one of your 4 nics available for another vswitch (different subnet for example), or backup SC, etc.

           

          more discusstion here:

          http://www.vmware.com/community/thread.jspa?messageID=478154

          • 2. Re: pSwitch Network redundancy for VMs?
            DougBaer Master
            VMware Employees

            To clarify, I have 6 NICs in the box. I use one for the SC, a separate NIC for VMotion, and have 4 NICs available for the VMs. I'd like to utilize the 4 NICs -- I could do active/standby, but I have the ability to aggregate them and that seems[/i] better because there is no need to deal with activating a standby NIC in the event of a failure.

             

            I'm generally suspicious of things that work 'automatically'...

            • 3. Re: pSwitch Network redundancy for VMs?
              wunderon Hot Shot

              I would assign a backup SC to one of the other vswitches as well, in case your single NIC for SC fails, perhaps to the vmotion vswitch.

               

              pp 56 of the Server Configuration PDF discusses failover policy for the NICS, as well as traffic shaping, network failure detection, failover types, etc.

               

              You could leave the NICS independent and configure their behavior through the VIC. Single point of configuration, etc.

              1 person found this helpful
              • 4. Re: pSwitch Network redundancy for VMs?
                DougBaer Master
                VMware Employees

                I understand the failover policy, and have read the manual. Unfortunately, there is not a lot out there regarding a redundant switch configuration that also leverages 802.3ad portgroups.

                 

                The suggestion to make the SC redundant is appreciated, but will only be possible if the network utilized by the SC is presented on the VM network trunk. That is not currently the case, but there is definitely a strong argument for doing so.

                • 5. Re: pSwitch Network redundancy for VMs?
                  Ken.Cline Champion

                  pSwitch diversity with 802.3ad port aggregation (Etherchannel) requires an inter-switch link and vendor support from your pSwitch manufacturer.

                   

                  With ESX 2.5, you could misconfigure 802.3ad and it would work (mismatched ends), with ESX 3, you've got to get it right.

                   

                  I usually recommend going with simple MAC-out load balancing. You can then have pSwitch diversity with even the most simple of pSwitches - and it requires no configuration on the pSwitch...

                  1 person found this helpful
                  • 6. Re: pSwitch Network redundancy for VMs?
                    DougBaer Master
                    VMware Employees

                    Ken,

                     

                    I think I'm confused about the pSwitch "diversity".  In this case, we have a pair of Cisco 6509s with 2 1000Base-T interfaces going from each switch to each of the ESX servers.  I don't want to use aggregation between the switches, rather, I would like to aggregate the 2 ports that go to each[/i] switch, but connect all 4 NICs to the same vSwitch.

                     

                    What I'm considering at this point is either 4 independent NICs or aggregating two sets:

                     

                    A and A'  both into the A-side switch = A-bond

                    B and B'  both into the B-side switch = B-bond

                     

                    1. Configure the A-bond as primary NICs for the vSwitch with B-bond NICs as the standby...

                     

                    2. Configure two port groups: one with A-bond Active/B-bond as Standby, the other with B-bond Active/A-bond Standby

                     

                     

                    It sounds like I should just forget the complexity, attach all 4 NICs as independent NICs, use out-mac, and be done with it! 

                     

                    Message was edited by:

                            DougBaer ... for runaway italics!

                    • 7. Re: pSwitch Network redundancy for VMs?
                      Ken.Cline Champion

                      I would like to aggregate the 2 ports that go to each  switch,

                      but connect all 4 NICs to the same vSwitch.

                       

                      No can do...

                       

                      It sounds like I should just forget the complexity, attach all 4 NICs

                      as independent NICs, use out-mac, and be done with it!

                       

                      Gets my vote

                      • 8. Re: pSwitch Network redundancy for VMs?
                        DougBaer Master
                        VMware Employees

                        Ah, well... one can dream, right?

                         

                        Thanks for the assistance guys

                        • 9. Re: pSwitch Network redundancy for VMs?
                          jhanekom Master

                          The only place I've seen that type of functionality is with HP's Intelligent Networking Pack (only available on Windows servers.)  Agreed, would be nice if VMware implemented something similar. http://h18000.www1.hp.com/products/servers/networking/teaming.html

                           

                          If you're splitting over two switches, consider turning on beacon monitoring/probing on ESX - whether 2.x and 3.x.

                           

                          Just a comment on an earlier statement... if you're going to be running ESX 3 (you haven't said yet), the general consensus seems to be that you should no longer dedicate a single adapter to the service console.  Put it in a load-balanced team - the adapter you're currently dedicating for VMotion is a candidate for teaming with the service console (or see the suggestion below.)

                           

                          Also, the necessity of having more than two adapters per vSwitch is debateable.  Very few environments need to output more than two Gb/s of network traffic and those that do generally don't virtualise well (high CPU overhead on networking.)  Keep in mind that with transmit load balancing, you'll only have 1Gb/s of bandwidth coming back in to the ESX server and a maximum outbound capacity of 1Gb/s per single MAC address you're communicating with (4Gb aggregate in your case.)

                           

                          If you're using ESX 3, have the six NICs anyway and want to go overboard with redundancy, consider the following possibility:

                          2 x SC

                          2 x VMotion

                          2 x VM Guests

                           

                          Take a look at this excellent post by Quotient: http://www.vmware.com/community/message.jspa?messageID=463425

                          • 10. Re: pSwitch Network redundancy for VMs?
                            DougBaer Master
                            VMware Employees

                            Thank you. Yes, the posts from Quotient have been most helpful in this area.