VMware Cloud Community
StephanS80
Contributor
Contributor

vCenter / Timed-out while waiting to get datastore information from host

Hi Community,

  • My environment consist of two ESXi 4.1 machines.
  • Both machines have one DAS and  NAS datastore.
  • Both machines hosts two virtual machines (DC and vCenter) for testing purposes (I want to test vCenter Heartbeat)
  • The virtual machines vCenter- and DC-VM resides on the DAS datastore. 

When I use the vSphereClient and make a direct connection to the ESXi machines I can browse the NAS datastore without problems. But when I connect to the vCenter VM and try to use/browse the NAS Datastore it fails with a "Timeout" message. I  tried to remove the NAS datastore on the ESXi machine when I have a direct connection to the ESXi machines.

Then I tried to create a NFS datastore via vCenter.

[2011-04-15 14:58:08.120 03008 error 'App' opID=110E0D33-000000B1] [WaitForDatastore] Timed-out while waiting to get datastore netfs://10.210.0.84//ifs/data/PS_Customers/200107/ information from host
[2011-04-15 14:58:08.195 03008 error 'App' opID=110E0D33-000000B1] vmodl.fault.SystemError
------ In-memory logs end   --------
[2011-04-15 14:58:09.686 03008 info 'App' opID=110E0D33-000000B1] [VpxLRO] -- FINISH task-182 -- host-136 -- vim.host.DatastoreSystem.createNasDatastore -- 2087A1D6-4CB3-4E11-AD75-EBE50348F69F(A15B1CB4-60C5-44B5-BCBB-58CE825298B4)
[2011-04-15 14:58:09.686 03008 info 'App' opID=110E0D33-000000B1] [VpxLRO] -- ERROR task-182 -- host-136 -- vim.host.DatastoreSystem.createNasDatastore: vmodl.fault.SystemError:
Result:
(vmodl.fault.SystemError) {
   dynamicType = <unset>,
   faultCause = (vmodl.MethodFault) null,
   reason = "Timed-out while waiting to get datastore information from host",
   msg = "",
}
Args:
Arg spec:
(vim.host.NasVolume.Specification) {
   dynamicType = <unset>,
   remoteHost = "storage1.plusserver.de",
   remotePath = "/ifs/data/PS_Customers/200107",
   localPath = "Test",
   accessMode = "readWrite",
   type = "nfs",
   userName = <unset>,
   password = <unset>,
}

Anyway, vCenter shows the datastore, see pictures in section 1. But it's not "really accessible" , see pictures in section 2.

1)

1.jpeg

4.jpeg

2)

2.jpeg

Service-Console View:

~ # esxcfg-route -l
VMkernel Routes:
Network          Netmask          Gateway          Interface
10.254.10.240    255.255.255.240  Local Subnet     vmk2
62.75.232.0      255.255.255.128  Local Subnet     vmk0
10.1.1.0         255.255.255.0    Local Subnet     vmk1
10.210.0.0       255.255.255.0    10.254.10.241    vmk2
default          0.0.0.0          62.75.232.1      vmk0
~ # ls -lha /vmfs/
/vmfs/devices/  /vmfs/volumes/
~ # ls -lha /vmfs/volumes/Isilon/
drwxr-xr-x    1 root     root           56 Apr  8 10:16 .
drwxr-xr-x    1 root     root          512 Apr 15 13:19 ..
dr-xr-xr-x    1 root     root            0 Apr 15 02:00 .snapshot
drwxr-xr-x    1 root     root          143 Apr 13 14:38 ISO Images
-rw-r--r--    1 root     root        18.0k Mar 23 13:47 index.html
~ #

~ # esxcfg-vmknic -l
Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type   
vmk0       10                  IPv4      62.75.232.6                             255.255.255.128 62.75.232.127   00:26:2d:01:79:c6 1500    65535     true    STATIC 
vmk1       271                 IPv4      10.1.1.1                                255.255.255.0   10.1.1.255      00:50:56:7e:c0:9a 1500    65535     true    STATIC 
vmk2       655                 IPv4      10.254.10.244                           255.255.255.240 10.254.10.255   00:50:56:75:ef:2f 1500    65535     true    STATIC 
~ # esxcfg-vswitch -l
DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks
dvSwitch         256         10          256               1500    vmnic7,vmnic6,vmnic5,vmnic4,vmnic1,vmnic0

  DVPort ID           In Use      Client
  1                   1           vmnic0
  2                   0
  3                   0
  4                   0
  5                   0
  6                   0
  7                   0
  8                   0
  10                  1           vmk0
  137                 0
  138                 1           vmnic1
  139                 1           vmnic4
  140                 1           vmnic5
  141                 1           vmnic6
  142                 1           vmnic7
  271                 1           vmk1
  655                 1           vmk2
  101                 0
  400                 0
  100                 0
  399                 0

~ # esxcfg-nas -l
Isilon is /ifs/data/PS_Customers/200107 from storage1.plusserver.de mounted
~ #


There is one special. I added a new route via: esxcfg-route -a 10.210.0.0/24 10.254.10.241 to access the NAS datastore.

But the datastore is accessable without any problems when I make a direct connection to the ESXi machine.

3.jpeg

Any Hints?

Thanks in advance,

Stephan

0 Kudos
2 Replies
StephanS80
Contributor
Contributor

Okay,

I figured something out. My shared storage (Isilon/EMC²) uses DNS Loadbalancing-technology. In other terms, the domain storage1.plusserver.de has a TTL of 0, see below:

[root@host01 ~]# dig storage1.plusserver.de

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_5.3 <<>> storage1.plusserver.de
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62571
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;storage1.plusserver.de.                IN      A

;; ANSWER SECTION:
storage1.plusserver.de. 0       IN      A       10.210.0.52

;; Query time: 78 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 15 16:33:26 2011
;; MSG SIZE  rcvd: 56

[root@host01 ~]#

In other words the Forward Lookup of storage1.plusserver.de changes every second.

[root@host01 ~]# host storage1.plusserver.de
storage1.plusserver.de has address 10.210.0.58
[root@host01 ~]# host storage1.plusserver.de
storage1.plusserver.de has address 10.210.0.87
[root@host01 ~]# host storage1.plusserver.de
storage1.plusserver.de has address 10.210.0.65
[root@host01 ~]# host storage1.plusserver.de
storage1.plusserver.de has address 10.210.0.99
[root@host01 ~]# host storage1.plusserver.de
storage1.plusserver.de has address 10.210.0.59
[root@host01 ~]#

[....]

vmdx.log:

[2011-04-15 14:25:07.562 173C6B90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-04-15 14:25:07.563 173C6B90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength 129)
[2011-04-15 14:25:07.563 173C6B90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Auto-completing at 129/129 bytes
[2011-04-15 14:25:07.563 17650B90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-04-15 14:25:07.563 17650B90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
[2011-04-15 14:25:07.563 FFFB7B10 info 'App' opID=8C6319D4-00000245-67] [VpxLRO] -- BEGIN task-internal-272 --  -- vpxapi.VpxaService.searchDatastore -- 52cfe477-24b8-df9f-555a-c176d73aaa37
[2011-04-15 14:25:07.618 FFFB7B10 error 'App' opID=8C6319D4-00000245-67] [VpxURL::GetLocalPath] locator 10.210.0.81 in URL not found on host: netfs://10.210.0.81//ifs/data/PS_Customers/200107/
[2011-04-15 14:25:07.619 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxaDatastoreContext] Resolved URL netfs://10.210.0.81//ifs/data/PS_Customers/200107/ to localPath
[2011-04-15 14:25:07.661 FFFB7B10 error 'App' opID=8C6319D4-00000245-67] [VpxURL::GetLocalPath] locator 10.210.0.81 in URL not found on host: netfs://10.210.0.81//ifs/data/PS_Customers/200107/
[2011-04-15 14:25:07.671 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxURL] Resolved localPath /var/log/vmware/vpx/ to URL netfs://62.75.232.6//var/log/vmware/vpx/
[2011-04-15 14:25:07.671 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxURL] Resolved localPath /var/log/vmware/vpx/ to URL netfs://62.75.232.6//var/log/vmware/vpx/
[2011-04-15 14:25:07.671 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxURL] Resolved localPath /var/log/vmware/vpx/ to URL netfs://62.75.232.6//var/log/vmware/vpx/
[2011-04-15 14:25:07.671 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxURL] Resolved localPath /var/log/vmware/vpx/ to URL netfs://62.75.232.6//var/log/vmware/vpx/
[2011-04-15 14:25:07.672 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxURL] Resolved localPath /var/log/vmware/vpx/ to URL netfs://62.75.232.6//var/log/vmware/vpx/
[2011-04-15 14:25:07.672 FFFB7B10 verbose 'SoapAdapter.HTTPService' opID=8C6319D4-00000245-67] HTTP Response: Complete (processed 782 bytes)
[2011-04-15 14:25:07.672 FFFB7B10 info 'App' opID=8C6319D4-00000245-67] [VpxLRO] -- FINISH task-internal-272 --  -- vpxapi.VpxaService.searchDatastore -- 52cfe477-24b8-df9f-555a-c176d73aaa37

I tried to add the datastore with an IP instead of the dns record... And voila it works!

Somebody encountered the same problem?

Iam still a bit confused... Why does it work when I make a direct connection to the ESXi hosts? First I thought i was luckily with the load-balancing, but it works EVERY time. Some ideas? For me it seems, this is a feature of storage systems which was not taken into account by VMware.

Best regards,

Stephan

0 Kudos
StephanS80
Contributor
Contributor

Hi,

today i had a Webex Session with a VMware engineer. The summary is that VMware don't support all "Isilon Smartconnect" features. Sadly there is no hint in the HCL of VMware, but I found a paragraph in the the Isilion documentation, see attachment. I will create a feature request, and hopefully VMware will consider this feature in one of the next vSphere releases.

Best regards,

Stephan

0 Kudos