Hello,
we have a ESX 3.5 einvironment wit three nodes and a NetApp storage behind.
All ESX nodes and also the backup server have two network interfaces, one on 100BT and one on GB.
We are backing up with BackupExec and VCB trying to force the traffic of the VCB backup between the ESX host and the backup server through the 1000BT interface, without success.
The ESX server receives the data through the 1000BT network, but sends it through the 100BT to the VCB/Backup server.
Is there a way to configure / force the network traffic between VCB and ESX ?
Thanks a lot for your help,
Matthias Oswald
Do you have a COS on both network?
Also do you have a custom hosts file on VCB with "internal" IP?
Andre
Is this a dual homed computer to separate IP subnets?
Check the IP Routing Table and possibly rearrange your NIC binding order on the VCB server. You can also try adjusting the "Inteface metric" assigned to each NIC.
Dual homing is all about knowing how routing tables work and how the NICs are ordered.
It is not dual homed, we just want to use the GB interface for the data traffic to the SAN storage and the othet NIC for management.
The NIC binding order on the two WIndows boxes (Backup server, vCenter server) looks ok, the GB interface comes first.
The routing table looks also fine I guess, pinging the Backup server returns the 192.168.1 GB network:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.0 * 255.255.255.0 U 0 0 0 vswif2
192.168.113.0 * 255.255.255.0 U 0 0 0 vswif0
192.168.1.0 * 255.255.255.0 U 0 0 0 vswif1
169.254.0.0 * 255.255.0.0 U 0 0 0 vswif2
default router-sw.snct. 0.0.0.0 UG 0 0 0 vswif0
PING srvbck01.snct.lux (192.168.1.220) 56(84) bytes of data.
64 bytes from srvbck01.snct.lux (192.168.1.220): icmp_seq=0 ttl=128 time=1.45 ms
64 bytes from srvbck01.snct.lux (192.168.1.220): icmp_seq=1 ttl=128 time=0.325 ms
64 bytes from srvbck01.snct.lux (192.168.1.220): icmp_seq=2 ttl=128 time=0.336 ms
--- srvbck01.snct.lux ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.325/0.704/1.451/0.528 ms, pipe 2
When looking at the network performance of the ESX server with vCenter client, I see incoming traffic from the SAN through the GB interface
but outgoing traffic to the backup server through the 100BT network.
Any other idea wherw I can look at ?
Thanks
Matthias
This appears to the route that the ESX server is issuing.
192.168.1.0 * 255.255.255.0 U 0 0 0 vswif1
What is the config of vswif1? Is it using the 1000BT or the 100BT? Or both? If it is both then configure the 100BT to be a standbay in the portgroup.
If it is the 100BT then you need to change the config to use the 1000BT.
vswif1 is a team of two NIC, both on 1000BT.
This interface on the ESC is used to receive the data from the storage, but the transmission to the backup server,
which has also has one 100BT and a team of two 1000BT, is going through the 100BT.
The VCB attacks the vCenter through the 1000BT and the ESX sends back to the VCB/Backup server on the 100BT.
I'm still not sure wehere the config problem is ?
ESX networking, VCB config, vCenter, Backup Exec config, host networking (host files, ???
Thanks for your support so far.
Is srvbck01.snct.lux (192.168.1.220) the ipaddress of the Teamed GB's or the the 100BT?
Do both interfaces have separate ip address registered as the same name in DNS? Does your DNS server do round robin?
If so try changing the 100BT interfact to something else like srvbck01B.snct.lux, or try configuring a host entry on the ESX server to point to the GB interfaces of the backup server.
If that isn't it provide the IP configuration for the backup server, routing table, and nic binding information of the backup server.
I launch the backup from the backup server with
vcbMounter.exe -h 192.168.1.221 -u snct\backup -p ??? -a ipaddr:192.168.1.102 -r f:\mnt\vmdbs02 -t fullvm -m nbd
-
Rgs,
Matthias
moswald@snct.lu
+352 357 214 341
Hi there,
thanks a lot for your efforts,
we noticed that both IPs of the srvbck01 192.168.113.189 (100BT) and 192.168.1.220 (GB) were in the DNS.
We changed the setting on the network and removed the GB from the DNS.
On the ESX server I added the GB interface in the hosts file
Do not remove the following line, or various programs
that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.113.188 vclnode03.snct.lux vclnode03
192.168.1.220 srvbck01.snct.lux srvbck01
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.0 * 255.255.255.0 U 0 0 0 vswif2
192.168.113.0 * 255.255.255.0 U 0 0 0 vswif0
192.168.1.0 * 255.255.255.0 U 0 0 0 vswif1
169.254.0.0 * 255.255.0.0 U 0 0 0 vswif2
default router-sw.snct. 0.0.0.0 UG 0 0 0 vswif0
PING srvbck01.snct.lux (192.168.1.220) 56(84) bytes of data.
64 bytes from srvbck01.snct.lux (192.168.1.220): icmp_seq=0 ttl=128 time=1.32 ms
64 bytes from srvbck01.snct.lux (192.168.1.220): icmp_seq=1 ttl=128 time=0.525 ms
The backup server srvbck01 has a host file
127.0.0.1 localhost
#::1 localhost
127.0.0.1 srvbck01.snct.lux
192.168.1.221 srvvm01-san
192.168.1.213 vclnode03
The vCenter server srvvm01 has an empty host file
127.0.0.1 localhost
::1 localhost
Backup server configuration:
C:\Users\Backup>hostname
srvbck01
C:\Users\Backup>ipconfig
Windows IP Configuration
Ethernet adapter GBBCK:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 192.168.1.220
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . :
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 192.168.113.189
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.113.254
Ethernet adapter Local Area Connection 2:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Tunnel adapter Local Area Connection* 8:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Tunnel adapter Local Area Connection* 9:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Tunnel adapter Local Area Connection* 13:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Tunnel adapter Local Area Connection* 14:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
C:\Users\Backup>route print
===========================================================================
Interface List
17 ...00 24 e8 37 10 95 ...... BASP Virtual Adapter
10 ...00 24 e8 37 10 91 ...... Broadcom BCM5709C NetXtreme II GigE (NDIS VBD Client)
11 ...00 24 e8 37 10 93 ...... Broadcom BCM5709C NetXtreme II GigE (NDIS VBD Client) #2
1 ........................... Software Loopback Interface 1
14 ...00 00 00 00 00 00 00 e0 isatap.{8E6AF430-DF3F-4580-A664-B50167CF2F67}
18 ...00 00 00 00 00 00 00 e0 isatap.{4DCFB2B1-5225-48BC-A8C0-C911F1DE046A}
15 ...02 00 54 55 4e 01 ...... Teredo Tunneling Pseudo-Interface
16 ...00 00 00 00 00 00 00 e0 isatap.{90BD23AA-B239-4011-B4C7-49C4F70C7931}
===========================================================================
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.113.254 192.168.113.189 276
127.0.0.0 255.0.0.0 On-link 127.0.0.1 306
127.0.0.1 255.255.255.255 On-link 127.0.0.1 306
127.255.255.255 255.255.255.255 On-link 127.0.0.1 306
192.168.1.0 255.255.255.0 On-link 192.168.1.220 266
192.168.1.220 255.255.255.255 On-link 192.168.1.220 266
192.168.1.255 255.255.255.255 On-link 192.168.1.220 266
192.168.113.0 255.255.255.0 On-link 192.168.113.189 276
192.168.113.189 255.255.255.255 On-link 192.168.113.189 276
192.168.113.255 255.255.255.255 On-link 192.168.113.189 276
224.0.0.0 240.0.0.0 On-link 127.0.0.1 306
224.0.0.0 240.0.0.0 On-link 192.168.1.220 266
224.0.0.0 240.0.0.0 On-link 192.168.113.189 276
255.255.255.255 255.255.255.255 On-link 127.0.0.1 306
255.255.255.255 255.255.255.255 On-link 192.168.1.220 266
255.255.255.255 255.255.255.255 On-link 192.168.113.189 276
===========================================================================
Persistent Routes:
Network Address Netmask Gateway Address Metric
0.0.0.0 0.0.0.0 192.168.113.254 Default
===========================================================================
IPv6 Route Table
===========================================================================
Active Routes:
If Metric Network Destination Gateway
1 306 ::1/128 On-link
1 306 ff00::/8 On-link
===========================================================================
Persistent Routes:
None
C:\Users\Backup>ping vclnode03
Pinging vclnode03 http://192.168.1.213 with 32 bytes of data:
Reply from 192.168.1.213: bytes=32 time<1ms TTL=64
Reply from 192.168.1.213: bytes=32 time<1ms TTL=64
Ping statistics for 192.168.1.213:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
^C
C:\Users\Backup>
I checked again the binding order:
GB (team)
1000BT
1000BT
100BT
-
Rgs,
Matthias
moswald@snct.lu
+352 357 214 341
Ok, both of your servers are "Dual Homed" meaning they have connections to separate subnets 192.168.1.0 and 192.168.113.0 respectively.
It appears your ESX server may have two Service Consoles? One for each IP Subnet? 192.168.1.213 and 192.168.113.118 repectively.
If these are not both Service Consoles, then one must be a vmKernel and the other the SC. Please let me know what that configuration is.
Your ESX server Hosts file is using the IP address of 192.168.113.188 in the hosts file....which should correspond to a Service Console.
On both server the Default Route is the 113 subnet (100BT), and not the GB subnet.
Your Backup server is missing a Host entry for itself, you may want to put that in as the 1.220 IP Address.
Previously you stated. The ESX server receives the data through the 1000BT network, but sends it through the 100BT to the VCB/Backup server.
How do you know this?
Assuming this is true, that would potentially mean (if my assumptions are correct) that the Service Console is responsible for sending data to the VCB server.
From my previous post, if you have two service consoles already configured, then configer the second SC as the "Default" OR
If you currently only have one SC, configure a second Service console on the GB interface, but keep the orginial as the "Default"
We see the network traffic in the vCenter clinet -> node -> Performance -> Network ...
During vcb backup the measurements on the GB RX is close to the 100BT TX and also
under the Windows Task Manager -> Networking on the backup server, only the 100BT is used.
Now after looking with esxcfg-route -l,
what strikes me is that the interface on the storage system (NetApp) is
actually defined as the default route, not the ESX GB.
This might be a mistake ???
Shouldn't it be the 1000BT of the ESX itself?
Is the default route actually affecting since the subnets should be properly defined ?
The Linux route command returns a port on the switch (100BT) as the default route.
-
Rgs,
Matthias
moswald@snct.lu
+352 357 214 341
In ESX the default gateway(s) can be defined on a Service Console or vmKernel Ports for Storage or vMotion. Which get used depends on the associated service using the IP. Make sure you have Default Gateways defined on both these configurations. Normally, you can only have one Defautl GW on a host, so I think the SC takes precendence...but not sure....your last post may contradict that.
This is also why i asked if you have two SC's or One Service Console and one vmKernel? Let me know which IP is which interface.
Your most recent post is telling me this...which is different than previusly thought.
1) The ESX Server is sending and receiving data via the GB
2) The Backup Sever is sending and recieving on the 100BT.
The issue appears to be the backup server.
What is the vcbMounter command you are using? Post the real IP Addresses in the command.
Sorry for the confusion.
The ESX server is receiving from the storage on the GB but sends to the backup server on 100BT.
The vcb command is
vcbMounter.exe -h vCenterServer -u backup -p ??? -a ipaddr:theVirtualMavhine IP -r f:\mnt\vmdbs02 -t fullvm -m nbd
Name PCI Driver Link Speed Duplex MTU Description
vmnic3 0a:00.00 bnx2 Down 0Mbps Half 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic6 24:00.00 e1000 Down 0Mbps Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic8 25:00.00 e1000 Down 0Mbps Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic9 25:00.01 e1000 Down 0Mbps Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic0 03:00.00 bnx2 Up 100Mbps Full 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic4 23:00.00 e1000 Up 1000Mbps Full 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic5 23:00.01 e1000 Up 1000Mbps Full 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic7 24:00.01 e1000 Down 0Mbps Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic1 05:00.00 bnx2 Up 100Mbps Full 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic2 08:00.00 bnx2 Down 0Mbps Half 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 64 4 64 1500 vmnic0
PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 0 vmnic0
Service Console 0 1 vmnic0
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 64 8 64 1500 vmnic5,vmnic4
PortGroup Name VLAN ID Used Ports Uplinks
SAN Network 0 2 vmnic4,vmnic5
Servicekonsole ISCSI0 1 vmnic4,vmnic5
VMkernel ISCSI 0 1 vmnic4,vmnic5
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch2 64 5 64 1500 vmnic7
PortGroup Name VLAN ID Used Ports Uplinks
Servicekonsole 0 1 vmnic7
VMkernel VMotion 0 1 vmnic7
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch3 64 10 64 1500 vmnic2,vmnic1
PortGroup Name VLAN ID Used Ports Uplinks
Virtual Machine Network 1130 6 vmnic1,vmnic2
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch4 64 1 64 1500
PortGroup Name VLAN ID Used Ports Uplinks
Temp VSwitch 0 0
VMkernel Routes:
Network Netmask Gateway
192.168.1.0 255.255.255.0 Local Subnet "the GB network"
192.168.2.0 255.255.255.0 Local Subnet "the Vmotion subnet"
default 0.0.0.0 192.168.1.11 "the IP of the NetApp stoarge 1000BT interface" which is an interafce not on theESX
comments in ""
Not sure what should be set in the esxcfg-route ?
Is the 192.168.1.0 subnet routable? Or is it just subnet used for the storage and backup?
I'm pretty sure your issue appears to be that a Service Console is not connected to the 192.168.1.0 subnet. As far as i know vcbMounter will only communicate with a Service Console IP.
Since the 192.168.113.0 network holds the SC, that would explain why ESX is sending data and the Backup Server is receiving data on that on those interfaces.
vcbMounter will communicate with VC server. What ever IP address (SC) is registered in VC will be the one that the vcbMounter uses.
If 192.168.1.0 is routable you should create a second SC on that network, then register the ESX host (disconnect change DNS entry to use 192..168.1.0 Ip Address and readd) to VC Server with the new IP configuration.
If 192.168.1.0 is not routable you shouldn't use the VC server to do the backup use the ESX host directly via a Second SC port on the 192.168.1.0 subnet.
Make sense?
Assuming 192.168.1.0 is not routable already.
A third option would be to dual home your VC server to the 192.168.1.0 subnet as well....and only register the 192.168.1.0 SC's in VC server.
A fouth option would be to dual home your VC server and drop you 100BT interfaces all together.
A fifth option would be make 192.168.1.0 routable with the rest of the network and drop the 100BT interfaces all together.
As for the default 0.0.0.0 192.168.1.11 "the IP of the NetApp stoarge 1000BT interface" which is an interafce not on theESX
I think you mistakenly added in the 192.168.1.11 ip address as the Default Gateway for the vmKernel storage network. If this is not routable you should remove the default gateway setting. If it is routable you should add in the correct router ip address.
It is just a subnet for storage and backup.
There is a Service Console on both VSwitch0 (100BT) and vSwitch1 (GB).
Changing the DNS entry is not necessarily what I want to do,
management access trough 100BT but backup/storage through GB.
I will do a test vcb backup with -h to the ESX (GB) first.
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 64 4 64 1500 vmnic0
PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 0 vmnic0
Service Console 0 1 vmnic0
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 64 8 64 1500 vmnic5,vmnic4
PortGroup Name VLAN ID Used Ports Uplinks
SAN Network 0 2 vmnic4,vmnic5
Servicekonsole ISCSI0 1 vmnic4,vmnic5
VMkernel ISCSI 0 1 vmnic4,vmnic5
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch2 64 5 64 1500 vmnic7
PortGroup Name VLAN ID Used Ports Uplinks
Servicekonsole 0 1 vmnic7
VMkernel VMotion 0 1 vmnic7
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch3 64 9 64 1500 vmnic2,vmnic1
PortGroup Name VLAN ID Used Ports Uplinks
Virtual Machine Network 1130 5 vmnic1,vmnic2
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch4 64 1 64 1500
PortGroup Name VLAN ID Used Ports Uplinks
Temp VSwitch 0 0
I did the following tests:
1. vcbmounter -h to the esx server GB interface
GB ist used for both RX and TX on ESX host
2. vcbmounter -h to the vCenter server GB interface
100BT is used on backup server and ESX TX, GB on ESX RX from storage
3 changed the vCenter host IP in DNS to the GB interface
GB is used , same as above under 1.
Changing the DNS entry of the backup server to the GB interface did the trick. hostname / DNS entry point to GB
That is fine so far.
Only if I would need to have the host name / DNS entry on 100BT for isolation of managament traffic from backup/storage traffic???
Glad to hear it is working for you now.
I would buy GB NIC for your SC and dump the 100BT NIC. As you can tell the SC is used for more than just managing the server, it requires bandwith to do backups.
Thanks a lot for your patience / help.
Have a great weekend.
-
Rgs,
Matthias