Slow throughput on vSphere and Arcserve r15 VDDK

[BradDHancock]

Hey Guys,

I am recently experiencing a performance issue with Arcserve R 15.0 on vSphere 4.1 using VDDK. Without any documented change to our backup environment certain VM Guests have slowed down to 300mb/min on backing up from our FC SAN.

Ive ensured they are no previous snapshots and that other vm guests on the same datastore backup as expected. I have also verified other vm guests on the same hosts as the troubled guests backup as expected, and I looked inside the guest's os at the fragmentation on the drives and everything looks normal.

I have all the necessary logging turned on in Arcserve and do not see anything really unusually. I have verified the VMDatastoreTypes.ini file has the SAN setting of 0 for the problem vm's.

Can anyone shed some light on this issue or point me in the right direction?

thanks

[DSTAVERT]

Not a solution but perhaps an explanation. http://communities.vmware.com/message/1613708#1613708

You might want to place a support call to CA.

[BradDHancock]

Thanks for the response.

I was afraid someone was going to say that. My experience with CA has been horrible. I seem to get the most inexperienced techs and we have to go back through what I had for breakfast before they tell me they need to call me back. I find it incredibly painful to call them.

[alefestaedist]

Did you try to use VCB manually to do a snapshot of the VM affected? the idea is to check if the performance issue is related to the vsphere or arcserve.

I know VCB it is not anymore supported/suggested but if your performance issues are related to only some VM could be something else than Arcserve.

Antoher question is, the job it is always slow or it is more a random issue?

[BradDHancock]

I tried doing a backup the vcb way in file mode and it was about the same speed for those vm's.

Seems like the issue is related the backups going over the network instead of the san.

[alefestaedist]

I would say that so it is not Arcserve at least you may avoid call CA support LOL.

Do you use a dedicated network segment for VCB? could  be a netwrok congestion or the size of the snapshot.

Try this.. do a first vcb snapshot of a VM and immediatly after a second vcb snapshot, it will have, obviously, a smaller size but you may check the network speed, if increase than your issue is related to the size of the delta, if continue to be the same speed it is  a network congestion issue.

[BradDHancock]

The problem seems to be it's using NDB for these guests in question:

In the VMDKIO log :

VixDiskLib:  Advanced transport modes not available for opening moref=vm-973.

NBD_ClientOpen: attempting to create connection to vpxa-nfc://

Not sure about the Network congestion because I don't want it to use the network. The backup machine has a SAN connection to these guests.

[alefestaedist]

okay so try again the vcb using the -m san

i.e.:

vcbmounter -h <ip of the host/vcenter> -u <user> -p <password> -a ipaddr:<IP or hostname of the VM> -t fullvm -r <PATHWHERETOSAVETHEVM> -m "san"

does this time continue to use the nbd?

In this case check in the config file of VCB that the transport mode is not NBD.

On another end can you check the even the other VM are not using NBD instead of san?

[BradDHancock]

When using VCB manually it uses the SAN.

[BradDHancock]

The issue was related to libexpat.dll. It was not installed on the machine, nor was it documented in Arcserve's documentation for the vmware agent.

I found this article explaining the issue:

https://communities.ca.com/web/ca-recovery-management-storage-global-user-community/message-board/-/...

Notice author John dogs CA too.

[BradDHancock]

What I don't understand is why some of the machines would use the SAN and according to the link posted on CA's forum it should not.

[aruntejaswi · Accepted solution]

Hi All,

This is Arun. I work with CA Technologies on the CA ARCserve productline.

From the response to a related query that we have from VMware, whenever advanced transport mode is used a temporary directory is created for the particular VM to mount the Disk for hotadd and SAN mode. If for some reason the backup were to fail, this directory does not get deleted and the next time advanced transport mode is used for the same VM it fails with the error message "Advanced transport modes not available for opening ...'. According to VMware, the backup fails because it is unable to create a temporary directory since the directory already exists. The user might need to delete the temporary directory and then try again.

The TEMP directory created during VDDK install is "C:\Documents and Settings\Administrator\Local Settings\Temp\vmware-Administrator". This path is used to create directory for advanced transport modes automatically, if the user does not specify one in the VixDiskLib_InitEx or VixMntapi_Init.

Insted of deleting the Temporary Directory, I would recommend to simply rename the directory and try again. Let me know if this works. We are very determined to get your backups back on track and you can also email us if you need further assistance. Please use our twitter email, we still do not have one specific to this forum, twitter_arcserve@ca.com

Regards,

Arun

[BradDHancock]

Yeah by deleting the directory it finally started to work successfully.

[aruntejaswi]

Delighted to know that the suggestion helped resolve the case Also, the behaviour described on the above link to CA's forum has not been experienced during any of our tests with SAN based backups using the Agent for Virtual Machines.

We understand that you would have ideally got this information via CA Support and regret the negative experiences you have had in the past. Please note that we are more eager than ever at CA Technologies for any feedback that will help us improve on the quality of the offerings we deliver to you and to the thousands of other ARCserve customers. The CA Support Management is interested to speak to you to better understand your experience and thoughts.

Anyone else on the thread wishing to share your thoughts, also please feel to email your contact details to twitter_arcserve@ca.com

Sharing some internal info, in the most recent CA Support survey ARCserve support was found to have the best satisfaction ratings among all products in CA's portfolio. With your feedback, we are determined to take it even higher. 

[Heinrich928]

Looks like I have the same problem. Meanwhile I'm on Arcserve r16 and on vsphere 5. My case with CA is now open for 2,5 months, because I'm experiencing occasionally TCP timeouts during SAN backup!

Today I watched the makeup job for the failed VM with TCP timeout and saw that the performance was very poor - about 1/10 of the usual performance.

Had a look at the network traffic and saw that a constant network flow goes from the ESX host interface (on which the VM resides) to the backup server.

In my case a router connects the VM network and the ESX host management network, So the performance of the router limits the throughput and probably is the reason for the TCP timeouts.

Per luck I came over this thread, and BANG! So I will delete the temp files. I will report if this helped.

BTW: please dont ask me on my experience with CA support 😉

Update: The deletion of the temp directory was the key. In my case it was in the %windir%\temp\vmware... After erasing it, the backup goes through SAN again. Unfortunatly this doesnt solve my TCP timeout problem too...