VMware

This Question is Not Answered

1 "correct" answer available (10 pts)
12 Replies Last post: Jun 25, 2009 12:33 PM by mtsm  

HP C7000 + ESX 3.5 + GBE2C posted: Jun 23, 2009 7:20 PM

Click to view mtsm's profile Enthusiast 98 posts since
Aug 24, 2006
Hi guys..i need a little help , my company just bought a new c7000 full with sixteen small blades each one with 2 internal nics and two mezzanine cards for hbas and nics , so each blade will have 4 nics and 2 hbas , of course they didnt involve my team in buying this equipment so i was thinking the following, the first enclosure (where the sixteen blades are) have 4 GBE2C switches with 16 ports each :

two nics in etherchannel for SC + vmkernel (vlan tagging enabled sc and vmkernel in distincts vlans)


two nics in etherchannel for vm traffic (many vlans passing through)


so the hp technicians powered on the blade and gave me access...when i see portmapping i see a little problem , each nic is connected to a distinct switch (nics 0 to switch 0 , nic 1 to switch 1 , nic 2 to switch 2 , nic 3 to switch 3) so my idea of creating the etherchannel/trunking is gone as i cant aggregates with nics in different switches , does anyone know if its possible to change that mapping logically ? so i can put nic0/nic1 to the same switch? i didnt find anything on manuals , on the console i can just view. i am attaching an image to show what i am talking about..


does anyone have an idea for a better arquiteture than the one i was proposing?


http://communities.vmware.com/servlet/JiveServlet/downloadImage/6089/myblade.tiff


here is the link for the image in case the attach doesnt work , http://xs140.xs.to/xs140/09262/myblade156.png

Re: HP C7000 + ESX 3.5 + GBE2C

1. Jun 23, 2009 9:18 PM in response to: mtsm
Click to view CiscoKid's profile Hot Shot 127 posts since
Apr 18, 2006
Me again, I would do the following, configure vSwitch0 with portgroups Service Console and VMKernel and then assign one of each NIC type (onboard and mezzanine). Then create vSwitch1 with the remaining NICs and assign all of your Virtual Machine Network portgroups to this switch. As for etherchanneling, you are correct, you cannot do this at the access layer (server to switch). However, you can have multiple ISLs that can be etherchanneled at the distribution layer. How many VMs do you plan on running per server?

Re: HP C7000 + ESX 3.5 + GBE2C

3. Jun 23, 2009 9:43 PM in response to: mtsm
Click to view CiscoKid's profile Hot Shot 127 posts since
Apr 18, 2006

Okay, so you have BL480c blades? If that is the case you should have the following mapping if you are using Cisco 3020s for ICB1&2:

Server1 in Bay1/9 will have ports G0/1 and G/09 in ICB1&2

Server2 in Bay2/10 will have ports G0/2 and G0/10 in ICB1&2

Server3 in Bay3/11 will have ports G0/3 and G0/11 in ICB1&2

....

Re: HP C7000 + ESX 3.5 + GBE2C

4. Jun 23, 2009 9:50 PM in response to: CiscoKid
Click to view CiscoKid's profile Hot Shot 127 posts since
Apr 18, 2006
My bad, I researched your switches and it looks like these are basic switches and not Cisco. In this case you will not have the ability to have your 4 onboards to connect to ICB1&2...you have to use it in multiple ICBs. We use the Cisco 3020s and it allows us to use all 4 ports on ICB1&2. What happens to the iLO port map if you were to remove the other 2 switches? Does it see the other NICs as disconnected?

Re: HP C7000 + ESX 3.5 + GBE2C

5. Jun 23, 2009 9:51 PM in response to: mtsm
Click to view CiscoKid's profile Hot Shot 127 posts since
Apr 18, 2006
As far as number of VMs per host, I think you will be fine with the 2x1GB ports that you will use on vSwitch1 without etherchannel to the access layer.

Re: HP C7000 + ESX 3.5 + GBE2C

8. Jun 23, 2009 10:04 PM in response to: mtsm
Click to view CiscoKid's profile Hot Shot 127 posts since
Apr 18, 2006
Well, it's not the server, it's the interconnect switch that is the limitation. I really enjoy BL480c blades as ESX hosts as they provide an excellent platform for a mid-range ESX host as a 2-CPU server. Best bang for the buck. We started out with 8 BL480c blades configured with 32GB RAM with the intention that the consolidation ratio was going to be 10:1. Our initial scope was for 166 physical servers to be virtualized on 16 BL480c servers and found that the consolidation ratio is more like 15:1-17:1 based on workloads. We started running out of memory before CPU so we upgraded memory to 48GB and everything is nice and stable. We have not implemented 8 of the 16 hosts, but when we do we will employ them as a separate HA/DRS cluster and making sure to also stagger the physical servers across 3 C7000 chassis to ensure we are protected from chassis failure.

Re: HP C7000 + ESX 3.5 + GBE2C

9. Jun 23, 2009 11:04 PM in response to: mtsm
Click to view sclingan's profile Novice 2 posts since
Jun 23, 2009

I have the same setup pretty much.

You cant use therchannel, but your vm's will split across the nics in the vm vswitch. So you get 2 x 1Gb for your vm's. Not sure how esx balances across the nics, but they are assigned to a particular nic and therefore interconnect switch, so each vm will get a share of maximum 1Gb.

I used to run rack mount 4 cpu boxes with heaps more nics than two, and these were trunked. But that might have been overkill and I find the BL460 goes ok for bandwidth with only 2Gb and maybe 16 vm's max. I get about 10 vms typically per blade.

As I think was mentioned you can trunk out of each nortel into your other network infrastructure to give more bandwidth to the entire chassis.

If you run a lot of esx hosts in that chassis you will need this.

This basic config gives you link failure redundancy for your nics in your vswitch, but since the link is in the backplane its really only giving you redundancy for when an interconnect switch fails. Upstream path failures will not cause the vswitch to swap to the other nic, as I discovered recently.

If you want more redundancy it gets a little complicated. There are a few options

In one scenario you need to enable the crossbar between the switches (ports 17 and 18) and enable STP. Note this will mean you are only using the trunk from one of the interconnects to provide all your upstream bandwidth. If that trunk fails, STP will cut you over to the other trunk coming out of the other interconnect.

esx hosts connected to the interconnect switch with the inactive trunk will connect upstream via the crossbar to the interconnect with the active trunk.

In this scenario your trunk might need to be 3 or 4 ports to get the required bandwidth.

If you need more than 2 x 1Gb bandwidth for your esx host then you can use 10Gb ethernet I guess (flex10 option), but 2 x 10Gb is a lot of traffic for a 2 CPU blade and as mentioned your going to run out of memory or CPU before you run out of bandwidth.

Oh, there is a quad port card option but you need another two interconnect switches.

Theres a doco from HP on how to configure your network for redundancy, cant find it though.


Re: HP C7000 + ESX 3.5 + GBE2C

11. Jun 24, 2009 9:51 PM in response to: mtsm
Click to view sclingan's profile Novice 2 posts since
Jun 23, 2009

Yes thats correct. You need to create the vlan on all switches, the uplinks/trunks, the crosslink ports 17, 18, and esx ports (1-16 as necessary)

But rather than me tell you read this doco (found it).

http://h71028.www7.hp.com/ERC/downloads/4AA1-0779ENW.pdf

In the examples they use cisco border switches (like you) with the GbE2c interconnect switch so just pick one of the topolgoes and follow the instructions.

We use HP Procurve but I figured it all out...

Chees

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities