topic vSphere Replication 5.1 Error in ESXi Discussions

vSphere Replication 5.1 Error

Southernunion — Thu, 11 Oct 2012 02:30:14 GMT

I recently intalled and setup the vSphere replication appliance, but when I try to setup a replication of a VM I get the following eror:

(VM_NAME) replication error: There is a storage issue with datastore path (datastore_name)

This datastore is in the same vCenter instance on a remote host over the WAN. I have tried other datastores and I get the same error. When I try a local datastore on a host that is closer to where the VM resides, it works.

Any ideas?

Re: vSphere Replication 5.1 Error

Dazprior — Tue, 23 Oct 2012 02:18:10 GMT

Hi, were you able to find a solution for this particular issue as we are now having the exact same issue?

Re: vSphere Replication 5.1 Error

Southernunion — Tue, 30 Oct 2012 13:22:41 GMT

No, nothing yet. I called VMware support and apparently I have baffled them as well. They have no clue to why this was happening. At first they thought that I needed another "Datacenter" setup for the replication to work. That didn't change anything. Then I went ahead re deployed the newest appliance 5.1.1 and I'm still getting the same error messages. Anybody else have any luck with this one...

Re: vSphere Replication 5.1 Error

sfarrell — Sat, 10 Nov 2012 23:46:50 GMT

bump

any updates ?

having same issue

Re: vSphere Replication 5.1 Error

Southernunion — Sun, 11 Nov 2012 19:46:50 GMT

Suppot is still looking for an answer. I am told that vSphere Replication support is so new, that support is trying to figure everything out for themselves.

Re: vSphere Replication 5.1 Error

0zzY — Tue, 13 Nov 2012 16:25:38 GMT

I have the same problem, looking for a solution..

Re: vSphere Replication 5.1 Error

jdishantha — Tue, 27 Nov 2012 00:56:54 GMT

I had the same issue and managed to resolve. I looked into the logs and noted that the path to the host of the datastore was not reachable and I found this to be a known issue http://www.vmware.com/support/vsphere5/doc/vsphere-replication-51-release-notes.html#knownissues. The issue was the first point where I had two management networks. In fact I was using only one for management and the other was used for NFS. I just disabled Management Traffic on the NFS network and replication stared working straight away. Hope this will help.

Re: vSphere Replication 5.1 Error

Dazprior — Tue, 27 Nov 2012 05:43:26 GMT

Hi, Thanks heaps, this fixed our issues!!

Re: vSphere Replication 5.1 Error

Southernunion — Tue, 27 Nov 2012 17:06:05 GMT

This workaround didn't fix it for me. Did you have your hosts on the same subnet? My configuration is one host that is hosting VM's is connected through a VPN to a DR location where Host2 is. Host2 is on a differenet subnet. There is routing between and everything else works except Replication. I wonder if the hosts have to on the same managment subnet?

Re: vSphere Replication 5.1 Error

mvalkanov — Thu, 06 Dec 2012 11:18:09 GMT

Hi,

could you provide the SR number, so that I can take a look?

The VR server at the secondary site needs to have access to the target datastores through port 902 of at least one host, on which the datastore is attached. However, VRMS might have picked up IP addresses of the host, that are not reachable by the VR server. There should be more info on the inaccessible datastore issue in the /var/log/vmware/*hbrsrv* logs, included in the VR support bundle when grepped by datastore uuid.

Re: vSphere Replication 5.1 Error

Southernunion — Mon, 10 Dec 2012 14:24:24 GMT

Here is the SR#: 12232795310

Port 902 is open. I guess it could be picking up the wrong ip address, but I have rebooted the VR appliance a couple times. How would one make sure that it's picking up the right IP?

Re: vSphere Replication 5.1 Error

mvalkanov — Mon, 10 Dec 2012 15:12:31 GMT

Hi,

I don't find any VR support bundles associated with the SR, only a single screenshot (Capture.PNG).

Please upload VR support bundles of both the protected site and the recovery site:

http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.replication_admin.doc/GUID-98AFCDF0-7FBB-493F-A23A-33A121C2DAAD.html

Regards,

Martin

Re: vSphere Replication 5.1 Error

Southernunion — Tue, 11 Dec 2012 20:21:02 GMT

Uploaded. Thanks.

Re: vSphere Replication 5.1 Error

mvalkanov — Wed, 12 Dec 2012 10:31:24 GMT

Hi,

I see uploaded only support bundles for one of the sites. Perhaps the other support bundle logs can tell about other errors? I see entries only for Host4_Local2 datastore at the VRMS logs. Nothing named Host2_ something.

Looking at the VR server logs (hbr-support-*.tgz/files/_var_log_vmware_hbrsrv-*.log and _var_log_vmware_hbrsrv-*.log.gz), I see that two datastores are not accessible:

No accessible host for datastore 501a8c94-0a9d4dc8-4d0f-0019bbe60128
No accessible host for datastore 502a06a4-63fbcaa9-df99-0018fe72e666

There are also a lot of connection errors to hosts, failing with "Internal Server Error" :

2012-12-11T20:08:25.942Z [7F1BD899E700 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:00000000049b9630, TCP:10.100.4.108:80>; cnx: (null), error: N7Vmacore4Http13HttpExceptionE(HTTP error response: Internal Server Error)
2012-12-11T20:08:25.942Z [7F1BD5450700 error 'HostAgentConnection'] Connection failed to host host-87 (10.100.4.108): HTTP error response: Internal Server Error

12-12-11T20:08:25.944Z [7F1BD8891700 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:0000000004991bd0, TCP:10.100.3.105:80>; cnx: (null), error: N7Vmacore4Http13HttpExceptionE(HTTP error response: Internal Server Error)
2012-12-11T20:08:25.945Z [7F1BD88D2700 error 'HostAgentConnection'] Connection failed to host host-98 (10.100.3.105): HTTP error response: Internal Server Error
2012-12-11T20:08:25.945Z [7F1BD88D2700 info 'vmomi.soapStub[204240]'] Resetting stub adapter for server <cs p:0000000004991bd0, TCP:10.100.3.105:80> : Closed
2012-12-11T20:08:25.945Z [7F1BD534C700 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:00007f1bc800e550, TCP:10.100.2.104:80>; cnx: (null), error: N7Vmacore4Http13HttpExceptionE(HTTP error response: Internal Server Error)
2012-12-11T20:08:25.946Z [7F1BD538D700 error 'HostAgentConnection'] Connection failed to host host-211 (10.100.2.104): HTTP error response: Internal Server Error

These are the IP addresses of the hosts registered at the VR server:

host-222 10.100.2.106
host-98 10.100.3.105
host-45 172.16.0.104
host-92 10.100.4.107
host-87 10.100.4.108
host-211 10.100.2.104

The first column is the vCenter Server managed object id. To see the host name, you could open https://vc_ip/mob/?moid=host-45.

Regards,

Martin

Re: vSphere Replication 5.1 Error

Southernunion — Wed, 12 Dec 2012 18:52:36 GMT

I only have one site, not two. One vCenter server. Let's take the details of our SR offline and then post the resolution on the forum. I don't want to risk some security information being posted on the forum. Thanks.

Re: vSphere Replication 5.1 Error

RGEnortec — Wed, 07 Aug 2013 14:48:23 GMT

Did this ever get resolved? I have a similar issue - but with 2 sites. The DR site in SRM is also having a strange problem. I can actually connect to the vSphere host from the client over its NFS vmkernel port.... even though management is definitely disabled for that VMK.... I even recreated the whole NFS vSwitch - same issue.

Re: vSphere Replication 5.1 Error

Gr1m — Tue, 07 Jan 2014 21:42:30 GMT

Did anybody ever figure this out? I am having the same issues with the same error message. I don't have two management networks on the hosts. I have tried replication to another site and same site with the same error. I am using distributed switch on my protected site and standard vSwitch on the DR site. My VR server is on another subnet the my Hosts. But I can ping back and forth.

any ideas?

Re: vSphere Replication 5.1 Error

stainboy — Thu, 20 Mar 2014 09:13:36 GMT

Hello all,

I'm having the same error.

Version 5.1.2

Tried having only one nic in the hosts, both primary and recovery sites. I can setup replication from site A to B and it syncs. Fro B to A I get that error "storage issue datastore path...."

Checked for aditional vmkernel ports that could have the "management traffic" tag but I could only find one for each host.

Have any one come up with a different approach? Some lateral thinking on this?

Found the error in the logs:

2014-03-20 14:00:54.197 WARN hms [hms-jobs-scheduler-thread-1] (..hms.util.HmsLock) | Timeout for candidate: (Owner: 18:IssueCalcProcessor.java:228:isGroupLocked,IssueCalcProcessor.java:210:isTargetObjectLocked(GID-6f74785a-d383-4233-8a67-19d73ee9f8ed), elapsed: 0 msec) The current owner is: (Owner: 17:SecondaryGroupImpl.java:4162:lockEntity,SecondaryGroupImpl.java:4176:onLastGroupErrorChanged(GID-6f74785a-d383-4233-8a67-19d73ee9f8ed), elapsed: 86 msec) Additional candidates waiting on the same lock: [] 2014-03-20 14:00:54.198 DEBUG hms.issue.IssueCalc [hms-jobs-scheduler-thread-1] (..hms.issue.IssueCalcProcessor$SingleCalculationProcessor) | The target object HmsGroup 'GID-6f74785a-d383-4233-8a67-19d73ee9f8ed' is in use. Will try to calculate issues for it later! 2014-03-20 14:00:54.199 WARN hms.monitor.hbr.datastore[525f51f7-46ad-21ae-40b7-ca8667324215] [hms-pcm-dispacher-thread-1] (..monitor.hbr.HbrDatastoreMonitor) operationID=9993e8a5-2630-48b2-956e-3c385f87e391 | Datastore VNX_L133_ATKU-PD01_DS002(MoRef: type = Datastore, value = datastore-83, serverGuid = null) become inaccessible. 2014-03-20 14:00:54.199 TRACE hms [hms-pcm-dispacher-thread-1] (..hms.util.EventPoster) operationID=9993e8a5-2630-48b2-956e-3c385f87e391 | POSTING EVENT:com.vmware.vcHms.datastoreInaccessibleEvent TARGET:datastore-83

Checked on th VR and ALL hosts are correctly added with correct ip addresses and all match the management interface ONLY.

21-03-2014

On the site where the VRA should access the hosts, to access the Datastores, there are certificates erroros, found in the VRA logs. Those errors are preventing the VRA to access the datastores and as such, giving inaccessible events.

Thx,

Carlos