VMotion VLAN shared with Oracle RAC Heartbeat Network

[boamek]

Hello all,

We are implementing Oracle RAC (on physical servers) in our environment for the first time and are trying to determine the network design. The original plan from our DBA was to create a new VLAN for the Oracle RAC Heartbeat Network, however, our Cisco Engineer stepped in and stated that he wanted to use the same VLAN that we are currently using for VMotion.

Here is what our Cisco Expert had to say when I questioned the idea:

    

Bandwidth contention doesn't come into picture within a vlan, it is  all on 720 Gbps backplane within a switch.

It is peer to peer and vlan can accommodate up to 254  end points.

Up to this point this VLAN has ONLY been for VMotion based on every best  practice I have ever seen, what I wanted to know is: Does anyone  have any concrete data that would show that combining this data is going  to wreck havoc on both technologies or conversly if it is a NON-ISSUE!

Thoughts?!

[admin]

The main reasons to separate the vMotion traffic are:

Bandwith and security.

If bandwith and security is not a problem in your case. You should check so the RAC heartbeat and vMotion ports don't conflicts.

If the port's don't conflict, I can't see a reason why you can't run both in same VLAN.

[depping]

There are no issues known, however as vMotion traffic is unencrypted someone with access to the Oracle RAC environment could potentially read all data flying from one host to the other. As such it is recommended to isolate vMotion traffic from any other type of traffic, same goes for management traffic for instance... or does the Cisco guy also combine his management layer with production traffic?

Duncan

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive

[logiboy123]

Separating management from data layers in any type of environment is always a good idea if possible.

Let me put it this way; One time I asked a girl who was a friend "What do women think about men with money"... What she said back to me I will never forget, she said to me "money has never made a man less attractive".

I think you understand the point.

[ravirhce]

I have little knowledge of Oracle RAC installation and working on VMware, so want to share some help if I can.

In my understanding Heartbeat network is private network which we use for heartbeat in RAC between cluster nodes?

1)      Public network (is used for client access)

2)      Private network (cluster heartbeat)? discussion about this network

3)      VIP (virtual IP network)

Above 3 networks which we configured in /etc/hosts file in all nodes of RAC

Please clarify me, if I am right then I will share the network connection in VMware environment on ESX, which will work perfectly.

Thanks

Ravi

[melwong78]

Its always best practice to isolate data traffic, management traffic and network control traffic.

Unfortunately, RAC Heartbeat falls between management and data traffic for keepalive and database sync.

It doesn't matter how much the switch backplane can provide in a single vlan. As long as theres no vlan isolation and a broadcast storm occurs...unnecessary chaos will occur. Your Cisco Engineer/Expert just want to make life easier during implementation and has no concern on the management and post maintenance.

My advice to you is nothing beats going to layer 3 IP instead of vlan. If resources permit, and security is a concern, use VPN for RAC. That way, you have tremendous flexibility for RAC link redundancy and QoS.

[boamek]

To clarify a few points:

We have a number of VLANs set up for "Public" traffic. IE standard server VLANs.

We also have a VLAN dedicated to Management - this is where the Service Console lives as well as other "Management" tools.

Up to this point we have a Layer 2 VLAN dedicated to VMotion - this is where the Cisco guy wants to put the RAC interconnect traffic.

While I understand that VMotion traffic is unencrypted I'm not particularly concerned with Security being an issue. It's a pretty big stretch for a DBA to get on this VLAN and hack into the VMotion stream. Besides all the juicy data is already in the Oracle databases - right?

It does sound like there is a technical concern around a broadcast storm interfiering with VMotin traffic. Has anyone expereinced this?

Don't get me wrong here - I'm fighting for VMotion to stay isolated - I'm just trying to establish what my worst case would be if I lose the fight!

[mcowger]

You should push for the separation, but if you can't get it, things will be fine.

[akumar86]

Hello Matt,

I use Boxbe to prioritize my email. While I did receive your email about "[vMotion & Resource Management] New message: "VMotion VLAN shared with Oracle RAC Heartbeat Network" ", you are not currently listed in my priority Guest List.

Click here to be added <https://www.boxbe.com/crs?tc=6268049058_1639422148>

Once added, future messages you send will go directly to my Inbox.

Thank you,

anupam@kumargroups.org

<http://www.boxbe.com/how-it-works?tc=6268049058_1639422148>

Powered by Boxbe -- "End Email Overload"

[Gleed]

Security issues aside, out the gate I think this is largely a non-issue as it would be difficult for both RAC and vSphere to push enough traffic to consume all 720Gbps.

The recommendation for using a dedicated network for vMotion traffic is not because vMotion has problems sharing a network with other traffic, but rather to ensure there is adequate bandwidth available when vMotion needs it.  In your case, it looks like you have plenty of bandwidth so the added RAC traffic shouldn't be an issue.  Especially since you will be using VLANs to segregate the traffic. 

The concerns I would have:

1.  In the future, as the size of your vSphere and RAC environments grow and as 10Gbps links become more common, you will see traffic/congestion increase.  So you'll want to keep an eye on it and monitor the growth.  Down the road you may need to revisit and separate the RAC/Vmotion traffic.

2.  What happens when/if Oracle recommends some low level network tuning for RAC that could conflict or adversely impact performance for vMotion (or vice versa)?   Make sure the network folks don't do any changes without running them by both the Oracle RAC and vSphere teams.

Regards,

-Kyle