How to force VCB backup using certain network interface

[matioswald]

Hello,

we have a ESX 3.5 einvironment wit three nodes and a NetApp storage behind.

All ESX nodes and also the backup server have two network interfaces, one on 100BT and one on GB.

We are backing up with BackupExec and VCB trying to force the traffic of the VCB backup between the ESX host and the backup server through the 1000BT interface, without success.

The ESX server receives the data through the 1000BT network, but sends it through the 100BT to the VCB/Backup server.

Is there a way to configure / force the network traffic between VCB and ESX ?

Thanks a lot for your help,

Matthias Oswald

[AndreTheGiant]

Do you have a COS on both network?

Also do you have a custom hosts file on VCB with "internal" IP?

Andre

[LEslinger]

Is this a dual homed computer to separate IP subnets?

Check the IP Routing Table and possibly rearrange your NIC binding order on the VCB server. You can also try adjusting the "Inteface metric" assigned to each NIC.

Dual homing is all about knowing how routing tables work and how the NICs are ordered.

[matioswald]

It is not dual homed, we just want to use the GB interface for the data traffic to the SAN storage and the othet NIC for management.

The NIC binding order on the two WIndows boxes (Backup server, vCenter server) looks ok, the GB interface comes first.

The routing table looks also fine I guess, pinging the Backup server returns the 192.168.1 GB network:

# route

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

192.168.2.0 * 255.255.255.0 U 0 0 0 vswif2

192.168.113.0 * 255.255.255.0 U 0 0 0 vswif0

192.168.1.0 * 255.255.255.0 U 0 0 0 vswif1

169.254.0.0 * 255.255.0.0 U 0 0 0 vswif2

default router-sw.snct. 0.0.0.0 UG 0 0 0 vswif0

# ping srvbck01

PING srvbck01.snct.lux (192.168.1.220) 56(84) bytes of data.

64 bytes from srvbck01.snct.lux (192.168.1.220): icmp_seq=0 ttl=128 time=1.45 ms

64 bytes from srvbck01.snct.lux (192.168.1.220): icmp_seq=1 ttl=128 time=0.325 ms

64 bytes from srvbck01.snct.lux (192.168.1.220): icmp_seq=2 ttl=128 time=0.336 ms

--- srvbck01.snct.lux ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2003ms

rtt min/avg/max/mdev = 0.325/0.704/1.451/0.528 ms, pipe 2

# ping srvbck01

When looking at the network performance of the ESX server with vCenter client, I see incoming traffic from the SAN through the GB interface

but outgoing traffic to the backup server through the 100BT network.

Any other idea wherw I can look at ?

Thanks

Matthias

[LEslinger]

This appears to the route that the ESX server is issuing.

192.168.1.0 * 255.255.255.0 U 0 0 0 vswif1

What is the config of vswif1? Is it using the 1000BT or the 100BT? Or both? If it is both then configure the 100BT to be a standbay in the portgroup.

If it is the 100BT then you need to change the config to use the 1000BT.

[matioswald]

vswif1 is a team of two NIC, both on 1000BT.

This interface on the ESC is used to receive the data from the storage, but the transmission to the backup server,

which has also has one 100BT and a team of two 1000BT, is going through the 100BT.

The VCB attacks the vCenter through the 1000BT and the ESX sends back to the VCB/Backup server on the 100BT.

I'm still not sure wehere the config problem is ?

ESX networking, VCB config, vCenter, Backup Exec config, host networking (host files, ???

Thanks for your support so far.

[LEslinger]

Is srvbck01.snct.lux (192.168.1.220) the ipaddress of the Teamed GB's or the the 100BT?

Do both interfaces have separate ip address registered as the same name in DNS? Does your DNS server do round robin?

If so try changing the 100BT interfact to something else like srvbck01B.snct.lux, or try configuring a host entry on the ESX server to point to the GB interfaces of the backup server.

If that isn't it provide the IP configuration for the backup server, routing table, and nic binding information of the backup server.

[matioswald]

I launch the backup from the backup server with

vcbMounter.exe -h 192.168.1.221 -u snct\backup -p ??? -a ipaddr:192.168.1.102 -r f:\mnt\vmdbs02 -t fullvm -m nbd

-

Rgs,

Matthias

moswald@snct.lu

+352 357 214 341

[matioswald]

Hi there,

thanks a lot for your efforts,

we noticed that both IPs of the srvbck01 192.168.113.189 (100BT) and 192.168.1.220 (GB) were in the DNS.

We changed the setting on the network and removed the GB from the DNS.

On the ESX server I added the GB interface in the hosts file

# cat /etc/hosts

1. Do not remove the following line, or various programs

2. that require network functionality will fail.

127.0.0.1 localhost.localdomain localhost

192.168.113.188 vclnode03.snct.lux vclnode03

192.168.1.220 srvbck01.snct.lux srvbck01

# route

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

192.168.2.0 * 255.255.255.0 U 0 0 0 vswif2

192.168.113.0 * 255.255.255.0 U 0 0 0 vswif0

192.168.1.0 * 255.255.255.0 U 0 0 0 vswif1

169.254.0.0 * 255.255.0.0 U 0 0 0 vswif2

default router-sw.snct. 0.0.0.0 UG 0 0 0 vswif0

# ping srvbck01

PING srvbck01.snct.lux (192.168.1.220) 56(84) bytes of data.

64 bytes from srvbck01.snct.lux (192.168.1.220): icmp_seq=0 ttl=128 time=1.32 ms

64 bytes from srvbck01.snct.lux (192.168.1.220): icmp_seq=1 ttl=128 time=0.525 ms

#

The backup server srvbck01 has a host file

127.0.0.1 localhost

#::1 localhost

127.0.0.1 srvbck01.snct.lux

192.168.1.221 srvvm01-san

192.168.1.213 vclnode03

The vCenter server srvvm01 has an empty host file

127.0.0.1 localhost

::1 localhost

Backup server configuration:

C:\Users\Backup>hostname

srvbck01

C:\Users\Backup>ipconfig

Windows IP Configuration

Ethernet adapter GBBCK:

Connection-specific DNS Suffix . :

IPv4 Address. . . . . . . . . . . : 192.168.1.220

Subnet Mask . . . . . . . . . . . : 255.255.255.0

Default Gateway . . . . . . . . . :

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :

IPv4 Address. . . . . . . . . . . : 192.168.113.189

Subnet Mask . . . . . . . . . . . : 255.255.255.0

Default Gateway . . . . . . . . . : 192.168.113.254

Ethernet adapter Local Area Connection 2:

Media State . . . . . . . . . . . : Media disconnected

Connection-specific DNS Suffix . :

Tunnel adapter Local Area Connection* 8:

Media State . . . . . . . . . . . : Media disconnected

Connection-specific DNS Suffix . :

Tunnel adapter Local Area Connection* 9:

Media State . . . . . . . . . . . : Media disconnected

Connection-specific DNS Suffix . :

Tunnel adapter Local Area Connection* 13:

Media State . . . . . . . . . . . : Media disconnected

Connection-specific DNS Suffix . :

Tunnel adapter Local Area Connection* 14:

Media State . . . . . . . . . . . : Media disconnected

Connection-specific DNS Suffix . :

C:\Users\Backup>route print

===========================================================================

Interface List

17 ...00 24 e8 37 10 95 ...... BASP Virtual Adapter

10 ...00 24 e8 37 10 91 ...... Broadcom BCM5709C NetXtreme II GigE (NDIS VBD Client)

11 ...00 24 e8 37 10 93 ...... Broadcom BCM5709C NetXtreme II GigE (NDIS VBD Client) #2

1 ........................... Software Loopback Interface 1

14 ...00 00 00 00 00 00 00 e0 isatap.{8E6AF430-DF3F-4580-A664-B50167CF2F67}

18 ...00 00 00 00 00 00 00 e0 isatap.{4DCFB2B1-5225-48BC-A8C0-C911F1DE046A}

15 ...02 00 54 55 4e 01 ...... Teredo Tunneling Pseudo-Interface

16 ...00 00 00 00 00 00 00 e0 isatap.{90BD23AA-B239-4011-B4C7-49C4F70C7931}

===========================================================================

IPv4 Route Table

===========================================================================

Active Routes:

Network Destination Netmask Gateway Interface Metric

0.0.0.0 0.0.0.0 192.168.113.254 192.168.113.189 276

127.0.0.0 255.0.0.0 On-link 127.0.0.1 306

127.0.0.1 255.255.255.255 On-link 127.0.0.1 306

127.255.255.255 255.255.255.255 On-link 127.0.0.1 306

192.168.1.0 255.255.255.0 On-link 192.168.1.220 266

192.168.1.220 255.255.255.255 On-link 192.168.1.220 266

192.168.1.255 255.255.255.255 On-link 192.168.1.220 266

192.168.113.0 255.255.255.0 On-link 192.168.113.189 276

192.168.113.189 255.255.255.255 On-link 192.168.113.189 276

192.168.113.255 255.255.255.255 On-link 192.168.113.189 276

224.0.0.0 240.0.0.0 On-link 127.0.0.1 306

224.0.0.0 240.0.0.0 On-link 192.168.1.220 266

224.0.0.0 240.0.0.0 On-link 192.168.113.189 276

255.255.255.255 255.255.255.255 On-link 127.0.0.1 306

255.255.255.255 255.255.255.255 On-link 192.168.1.220 266

255.255.255.255 255.255.255.255 On-link 192.168.113.189 276

===========================================================================

Persistent Routes:

Network Address Netmask Gateway Address Metric

0.0.0.0 0.0.0.0 192.168.113.254 Default

===========================================================================

IPv6 Route Table

===========================================================================

Active Routes:

If Metric Network Destination Gateway

1 306 ::1/128 On-link

1 306 ff00::/8 On-link

===========================================================================

Persistent Routes:

None

C:\Users\Backup>ping vclnode03

Pinging vclnode03 http://192.168.1.213 with 32 bytes of data:

Reply from 192.168.1.213: bytes=32 time<1ms TTL=64
Reply from 192.168.1.213: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.1.213:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
^C
C:\Users\Backup>

I checked again the binding order:

GB (team)

1000BT

1000BT

100BT

-

Rgs,

Matthias

moswald@snct.lu

+352 357 214 341

[LEslinger]

Ok, both of your servers are "Dual Homed" meaning they have connections to separate subnets 192.168.1.0 and 192.168.113.0 respectively.

It appears your ESX server may have two Service Consoles? One for each IP Subnet? 192.168.1.213 and 192.168.113.118 repectively.

If these are not both Service Consoles, then one must be a vmKernel and the other the SC. Please let me know what that configuration is.

Your ESX server Hosts file is using the IP address of 192.168.113.188 in the hosts file....which should correspond to a Service Console.

On both server the Default Route is the 113 subnet (100BT), and not the GB subnet.

Your Backup server is missing a Host entry for itself, you may want to put that in as the 1.220 IP Address.

[LEslinger]

Previously you stated. The ESX server receives the data through the 1000BT network, but sends it through the 100BT to the VCB/Backup server.

How do you know this?

Assuming this is true, that would potentially mean (if my assumptions are correct) that the Service Console is responsible for sending data to the VCB server.

From my previous post, if you have two service consoles already configured, then configer the second SC as the "Default" OR

If you currently only have one SC, configure a second Service console on the GB interface, but keep the orginial as the "Default"

[matioswald]

We see the network traffic in the vCenter clinet -> node -> Performance -> Network ...

During vcb backup the measurements on the GB RX is close to the 100BT TX and also

under the Windows Task Manager -> Networking on the backup server, only the 100BT is used.

Now after looking with esxcfg-route -l,

what strikes me is that the interface on the storage system (NetApp) is

actually defined as the default route, not the ESX GB.

This might be a mistake ???

Shouldn't it be the 1000BT of the ESX itself?

Is the default route actually affecting since the subnets should be properly defined ?

The Linux route command returns a port on the switch (100BT) as the default route.

-

Rgs,

Matthias

moswald@snct.lu

+352 357 214 341

[LEslinger]

In ESX the default gateway(s) can be defined on a Service Console or vmKernel Ports for Storage or vMotion. Which get used depends on the associated service using the IP. Make sure you have Default Gateways defined on both these configurations. Normally, you can only have one Defautl GW on a host, so I think the SC takes precendence...but not sure....your last post may contradict that.

This is also why i asked if you have two SC's or One Service Console and one vmKernel? Let me know which IP is which interface.

Your most recent post is telling me this...which is different than previusly thought.

1) The ESX Server is sending and receiving data via the GB

2) The Backup Sever is sending and recieving on the 100BT.

The issue appears to be the backup server.

What is the vcbMounter command you are using? Post the real IP Addresses in the command.

[matioswald]

Sorry for the confusion.

The ESX server is receiving from the storage on the GB but sends to the backup server on 100BT.

The vcb command is

vcbMounter.exe -h vCenterServer -u backup -p ??? -a ipaddr:theVirtualMavhine IP -r f:\mnt\vmdbs02 -t fullvm -m nbd

# esxcfg-nics -l

Name PCI Driver Link Speed Duplex MTU Description

vmnic3 0a:00.00 bnx2 Down 0Mbps Half 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic6 24:00.00 e1000 Down 0Mbps Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic8 25:00.00 e1000 Down 0Mbps Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic9 25:00.01 e1000 Down 0Mbps Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic0 03:00.00 bnx2 Up 100Mbps Full 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic4 23:00.00 e1000 Up 1000Mbps Full 1500 Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic5 23:00.01 e1000 Up 1000Mbps Full 1500 Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic7 24:00.01 e1000 Down 0Mbps Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller

vmnic1 05:00.00 bnx2 Up 100Mbps Full 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic2 08:00.00 bnx2 Down 0Mbps Half 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

# esxcfg-vswitch -l

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch0 64 4 64 1500 vmnic0

PortGroup Name VLAN ID Used Ports Uplinks

VM Network 0 0 vmnic0

Service Console 0 1 vmnic0

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch1 64 8 64 1500 vmnic5,vmnic4

PortGroup Name VLAN ID Used Ports Uplinks

SAN Network 0 2 vmnic4,vmnic5

Servicekonsole ISCSI0 1 vmnic4,vmnic5

VMkernel ISCSI 0 1 vmnic4,vmnic5

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch2 64 5 64 1500 vmnic7

PortGroup Name VLAN ID Used Ports Uplinks

Servicekonsole 0 1 vmnic7

VMkernel VMotion 0 1 vmnic7

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch3 64 10 64 1500 vmnic2,vmnic1

PortGroup Name VLAN ID Used Ports Uplinks

Virtual Machine Network 1130 6 vmnic1,vmnic2

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch4 64 1 64 1500

PortGroup Name VLAN ID Used Ports Uplinks

Temp VSwitch 0 0

# esxcfg-route -l

VMkernel Routes:

Network Netmask Gateway

192.168.1.0 255.255.255.0 Local Subnet "the GB network"

192.168.2.0 255.255.255.0 Local Subnet "the Vmotion subnet"

default 0.0.0.0 192.168.1.11 "the IP of the NetApp stoarge 1000BT interface" which is an interafce not on theESX

#

comments in ""

Not sure what should be set in the esxcfg-route ?

[LEslinger]

Is the 192.168.1.0 subnet routable? Or is it just subnet used for the storage and backup?

I'm pretty sure your issue appears to be that a Service Console is not connected to the 192.168.1.0 subnet. As far as i know vcbMounter will only communicate with a Service Console IP.

Since the 192.168.113.0 network holds the SC, that would explain why ESX is sending data and the Backup Server is receiving data on that on those interfaces.

vcbMounter will communicate with VC server. What ever IP address (SC) is registered in VC will be the one that the vcbMounter uses.

If 192.168.1.0 is routable you should create a second SC on that network, then register the ESX host (disconnect change DNS entry to use 192..168.1.0 Ip Address and readd) to VC Server with the new IP configuration.

If 192.168.1.0 is not routable you shouldn't use the VC server to do the backup use the ESX host directly via a Second SC port on the 192.168.1.0 subnet.

Make sense?

Assuming 192.168.1.0 is not routable already.

A third option would be to dual home your VC server to the 192.168.1.0 subnet as well....and only register the 192.168.1.0 SC's in VC server.

A fouth option would be to dual home your VC server and drop you 100BT interfaces all together.

A fifth option would be make 192.168.1.0 routable with the rest of the network and drop the 100BT interfaces all together.

As for the default 0.0.0.0 192.168.1.11 "the IP of the NetApp stoarge 1000BT interface" which is an interafce not on theESX

I think you mistakenly added in the 192.168.1.11 ip address as the Default Gateway for the vmKernel storage network. If this is not routable you should remove the default gateway setting. If it is routable you should add in the correct router ip address.

[matioswald]

It is just a subnet for storage and backup.

There is a Service Console on both VSwitch0 (100BT) and vSwitch1 (GB).

Changing the DNS entry is not necessarily what I want to do,

management access trough 100BT but backup/storage through GB.

I will do a test vcb backup with -h to the ESX (GB) first.

# esxcfg-vswitch -l

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch0 64 4 64 1500 vmnic0

PortGroup Name VLAN ID Used Ports Uplinks

VM Network 0 0 vmnic0

Service Console 0 1 vmnic0

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch1 64 8 64 1500 vmnic5,vmnic4

PortGroup Name VLAN ID Used Ports Uplinks

SAN Network 0 2 vmnic4,vmnic5

Servicekonsole ISCSI0 1 vmnic4,vmnic5

VMkernel ISCSI 0 1 vmnic4,vmnic5

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch2 64 5 64 1500 vmnic7

PortGroup Name VLAN ID Used Ports Uplinks

Servicekonsole 0 1 vmnic7

VMkernel VMotion 0 1 vmnic7

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch3 64 9 64 1500 vmnic2,vmnic1

PortGroup Name VLAN ID Used Ports Uplinks

Virtual Machine Network 1130 5 vmnic1,vmnic2

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch4 64 1 64 1500

PortGroup Name VLAN ID Used Ports Uplinks

Temp VSwitch 0 0

#

[matioswald]

I did the following tests:

1. vcbmounter -h to the esx server GB interface

GB ist used for both RX and TX on ESX host

2. vcbmounter -h to the vCenter server GB interface

100BT is used on backup server and ESX TX, GB on ESX RX from storage

3 changed the vCenter host IP in DNS to the GB interface

GB is used , same as above under 1.

Changing the DNS entry of the backup server to the GB interface did the trick. hostname / DNS entry point to GB

That is fine so far.

Only if I would need to have the host name / DNS entry on 100BT for isolation of managament traffic from backup/storage traffic???

[LEslinger]

Glad to hear it is working for you now.

I would buy GB NIC for your SC and dump the 100BT NIC. As you can tell the SC is used for more than just managing the server, it requires bandwith to do backups.

[matioswald]

Thanks a lot for your patience / help.

Have a great weekend.

-

Rgs,

Matthias