Toes
Contributor
Contributor

vSphere replication RPO

Hi everyone,

I have the following setup:

Production & DR Sites

vCenter 6 build 3634793

ESXi 6 build 3825889

SRM 6.0.0.1 build 2700459

vReplication 6.0.0.3 build 3845888

Quite a number of times I have noticed that a bunch of replicated VMs will start showing RPO violations.  if you take a look at one of the RPO flagged VMs from within the web client and check the replication details you see only a few KB replicated over a period of time instead of the GBs that should have already replicated.

I usually vmotion the problem VMs to a new host and the replication then runs within normal observed times to completion.

Anyone come across a similar problem? I have yet to figure out what causes it.

regards,

Craig

0 Kudos
5 Replies
Sreec
VMware Employee
VMware Employee

You need to do a basic health check both from Network and Storage perspective

1. Login to the hosts were VM's are residing on both source and destination and check vmkernal and hostd logs for that time period to confirm if there were any connectivity issues reported . Live ESXTOP would also be a better solution when RPO is getting reported - Watch out for Network counters Rx Tx etcc.

2. Any storage connectivity issues or disk latency issues should be checked

3.Any fluctuation in network link ?  Is this a dedicated connection for replication traffic?

4. Also do understand how RPO works -- >Understanding vSphere Replication (VR) Scheduling and RPO Violations - VMware vSphere Blog

I usually vmotion the problem VMs to a new host and the replication then runs within normal observed times to completion-> Good observation,does that mean RPO violation never got reported in the next replication cycle on the new host ? OR do you need to migrate the VM again ?

Cheers,
Sree | CKA|CKAD|VCIX-3X| VCAP-4X| VExpert 5x
Please KUDO helpful posts and mark the thread as solved if answered
0 Kudos
Toes
Contributor
Contributor

Thanks for the feedback!

I will need to get the network and storage teams involved too.

I will provide feedback as I discover issues.

Craig

0 Kudos
Toes
Contributor
Contributor

A bit of feedback.

One of the VMs is currently in vReplication "limbo", the VM shows the following:

pastedImage_0.png

on the target site replication appliance they following is seen regarding the above vm:

pastedImage_3.png

after a short period of time:

pastedImage_2.png

the above entries keeps on going through the same loop until the VM is vmotioned to another host and then the replication runs successfully.

TIA,

Craig

0 Kudos
admin
Immortal
Immortal

Check the logs of the ESXi host on which the replicating VM was registered, when replication was not progressing.


There may be some clue about why it was not working. try /var/log/vmkernel.log or maybe /var/log/hostd.log. You could try 'grep -i hbr' in either of these logs to narrow down to replication activity.

Is it possible the problem is associated with certain ESXi hosts or does it seem to affect all of them?

0 Kudos
Toes
Contributor
Contributor

This is across all hosts, a mix of BL460c Gen9 and DL380 Gen9.

The host logs do not seem to give much insight as to the problem.

Thanks,

0 Kudos