vSphere Replication 5.1 Error

Southernunion · ‎10-10-2012

I recently intalled and setup the vSphere replication appliance, but when I try to setup a replication of a VM I get the following eror:

(VM_NAME) replication error: There is a storage issue with datastore path (datastore_name)

This datastore is in the same vCenter instance on a remote host over the WAN. I have tried other datastores and I get the same error. When I try a local datastore on a host that is closer to where the VM resides, it works.

Any ideas?

Dazprior · ‎10-22-2012

Hi, were you able to find a solution for this particular issue as we are now having the exact same issue?

Southernunion · ‎10-30-2012

No, nothing yet. I called VMware support and apparently I have baffled them as well. They have no clue to why this was happening. At first they thought that I needed another "Datacenter" setup for the replication to work. That didn't change anything. Then I went ahead re deployed the newest appliance 5.1.1 and I'm still getting the same error messages. Anybody else have any luck with this one...

sfarrell · ‎11-10-2012

bump

any updates ?

having same issue

Southernunion · ‎11-11-2012

Suppot is still looking for an answer. I am told that vSphere Replication support is so new, that support is trying to figure everything out for themselves.

0zzY · ‎11-13-2012

I have the same problem, looking for a solution..

jdishantha · ‎11-26-2012

I had the same issue and managed to resolve. I looked into the logs and noted that the path to the host of the datastore was not reachable and I found this to be a known issue http://www.vmware.com/support/vsphere5/doc/vsphere-replication-51-release-notes.html#knownissues. The issue was the first point where I had two management networks. In fact I was using only one for management and the other was used for NFS. I just disabled Management Traffic on the NFS network and replication stared working straight away. Hope this will help.

Dazprior · ‎11-26-2012

Hi, Thanks heaps, this fixed our issues!!

Southernunion · ‎11-27-2012

This workaround didn't fix it for me. Did you have your hosts on the same subnet? My configuration is one host that is hosting VM's is connected through a VPN to a DR location where Host2 is. Host2 is on a differenet subnet. There is routing between and everything else works except Replication. I wonder if the hosts have to on the same managment subnet?

mvalkanov · ‎12-06-2012

Hi,

could you provide the SR number, so that I can take a look?

The VR server at the secondary site needs to have access to the target datastores through port 902 of at least one host, on which the datastore is attached. However, VRMS might have picked up IP addresses of the host, that are not reachable by the VR server. There should be more info on the inaccessible datastore issue in the /var/log/vmware/*hbrsrv* logs, included in the VR support bundle when grepped by datastore uuid.

Southernunion · ‎12-10-2012

Here is the SR#: 12232795310

Port 902 is open. I guess it could be picking up the wrong ip address, but I have rebooted the VR appliance a couple times. How would one make sure that it's picking up the right IP?

mvalkanov · ‎12-10-2012

Hi,

I don't find any VR support bundles associated with the SR, only a single screenshot (Capture.PNG).

Please upload VR support bundles of both the protected site and the recovery site:

http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.replication_admin.doc/GUID-98AFCDF0-7FBB-...

Regards,

Martin

Southernunion · ‎12-11-2012

Uploaded. Thanks.

mvalkanov · ‎12-12-2012

Hi,

I see uploaded only support bundles for one of the sites. Perhaps the other support bundle logs can tell about other errors? I see entries only for Host4_Local2 datastore at the VRMS logs. Nothing named Host2_ something.

Looking at the VR server logs (hbr-support-*.tgz/files/_var_log_vmware_hbrsrv-*.log and _var_log_vmware_hbrsrv-*.log.gz), I see that two datastores are not accessible:

No accessible host for datastore 501a8c94-0a9d4dc8-4d0f-0019bbe60128
No accessible host for datastore 502a06a4-63fbcaa9-df99-0018fe72e666

There are also a lot of connection errors to hosts, failing with "Internal Server Error" :

2012-12-11T20:08:25.942Z [7F1BD899E700 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:00000000049b9630, TCP:10.100.4.108:80>; cnx: (null), error: N7Vmacore4Http13HttpExceptionE(HTTP error response: Internal Server Error)
2012-12-11T20:08:25.942Z [7F1BD5450700 error 'HostAgentConnection'] Connection failed to host host-87 (10.100.4.108): HTTP error response: Internal Server Error

12-12-11T20:08:25.944Z [7F1BD8891700 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:0000000004991bd0, TCP:10.100.3.105:80>; cnx: (null), error: N7Vmacore4Http13HttpExceptionE(HTTP error response: Internal Server Error)
2012-12-11T20:08:25.945Z [7F1BD88D2700 error 'HostAgentConnection'] Connection failed to host host-98 (10.100.3.105): HTTP error response: Internal Server Error
2012-12-11T20:08:25.945Z [7F1BD88D2700 info 'vmomi.soapStub[204240]'] Resetting stub adapter for server <cs p:0000000004991bd0, TCP:10.100.3.105:80> : Closed
2012-12-11T20:08:25.945Z [7F1BD534C700 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:00007f1bc800e550, TCP:10.100.2.104:80>; cnx: (null), error: N7Vmacore4Http13HttpExceptionE(HTTP error response: Internal Server Error)
2012-12-11T20:08:25.946Z [7F1BD538D700 error 'HostAgentConnection'] Connection failed to host host-211 (10.100.2.104): HTTP error response: Internal Server Error

These are the IP addresses of the hosts registered at the VR server:

host-222 10.100.2.106
host-98 10.100.3.105
host-45 172.16.0.104
host-92 10.100.4.107
host-87 10.100.4.108
host-211 10.100.2.104

The first column is the vCenter Server managed object id. To see the host name, you could open https://vc_ip/mob/?moid=host-45.

Regards,

Martin

Southernunion · ‎12-12-2012

I only have one site, not two. One vCenter server. Let's take the details of our SR offline and then post the resolution on the forum. I don't want to risk some security information being posted on the forum. Thanks.

RGEnortec · ‎08-07-2013

Did this ever get resolved? I have a similar issue - but with 2 sites. The DR site in SRM is also having a strange problem. I can actually connect to the vSphere host from the client over its NFS vmkernel port.... even though management is definitely disabled for that VMK.... I even recreated the whole NFS vSwitch - same issue.

Gr1m · ‎01-07-2014

Did anybody ever figure this out? I am having the same issues with the same error message. I don't have two management networks on the hosts. I have tried replication to another site and same site with the same error. I am using distributed switch on my protected site and standard vSwitch on the DR site. My VR server is on another subnet the my Hosts. But I can ping back and forth.

any ideas?

stainboy · ‎03-20-2014

Hello all,

I'm having the same error.

Version 5.1.2

Tried having only one nic in the hosts, both primary and recovery sites. I can setup replication from site A to B and it syncs. Fro B to A I get that error "storage issue datastore path...."

Checked for aditional vmkernel ports that could have the "management traffic" tag but I could only find one for each host.

Have any one come up with a different approach? Some lateral thinking on this?

Found the error in the logs:

2014-03-20 14:00:54.197 WARN hms [hms-jobs-scheduler-thread-1] (..hms.util.HmsLock) | Timeout for candidate: (Owner: 18:IssueCalcProcessor.java:228:isGroupLocked,IssueCalcProcessor.java:210:isTargetObjectLocked(GID-6f74785a-d383-4233-8a67-19d73ee9f8ed), elapsed: 0 msec) The current owner is: (Owner: 17:SecondaryGroupImpl.java:4162:lockEntity,SecondaryGroupImpl.java:4176:onLastGroupErrorChanged(GID-6f74785a-d383-4233-8a67-19d73ee9f8ed), elapsed: 86 msec) Additional candidates waiting on the same lock: [] 2014-03-20 14:00:54.198 DEBUG hms.issue.IssueCalc [hms-jobs-scheduler-thread-1] (..hms.issue.IssueCalcProcessor$SingleCalculationProcessor) | The target object HmsGroup 'GID-6f74785a-d383-4233-8a67-19d73ee9f8ed' is in use. Will try to calculate issues for it later! 2014-03-20 14:00:54.199 WARN hms.monitor.hbr.datastore[525f51f7-46ad-21ae-40b7-ca8667324215] [hms-pcm-dispacher-thread-1] (..monitor.hbr.HbrDatastoreMonitor) operationID=9993e8a5-2630-48b2-956e-3c385f87e391 | Datastore VNX_L133_ATKU-PD01_DS002(MoRef: type = Datastore, value = datastore-83, serverGuid = null) become inaccessible. 2014-03-20 14:00:54.199 TRACE hms [hms-pcm-dispacher-thread-1] (..hms.util.EventPoster) operationID=9993e8a5-2630-48b2-956e-3c385f87e391 | POSTING EVENT:com.vmware.vcHms.datastoreInaccessibleEvent TARGET:datastore-83

Checked on th VR and ALL hosts are correctly added with correct ip addresses and all match the management interface ONLY.

21-03-2014

On the site where the VRA should access the hosts, to access the Datastores, there are certificates erroros, found in the VRA logs. Those errors are preventing the VRA to access the datastores and as such, giving inaccessible events.

Thx,

Carlos

All

vSphere Replication 5.1 Error