How many VMs can you backup through one vCenter server?

[peetz]

Hi all,

we have a fairly large vSphere 5.0 environment with 2x vCenter servers, each with ca. 2000 VMs using about 600 TB of FC-attached SAN storage.

The majority (~90%) of these VMs are backed up with Symantec NetBackup 7.5 via the vStorage API (doing VMware snapshots through vCenter).

While the environment was growing the backup environment also grew by adding more and more media servers to it. But we are still using only one vCenter server per location.

Since a while we are having trouble with a growing number of backup job failures: It looks like the backup servers' connections to vCenter sometimes fail or time out in the middle of a job.

And the vCenter servers (VMs with 8 vCPUs and 32 GB RAM) appear to be heavily loaded (esp. CPU, but also RAM).

My question here is: Does this happen, because we have reached an architectural limit of such kind of backup setup? Or in other words: Do we need to add additional vCenter servers and spread hosts and VMs among them?

We expect further growth of this environment and need to take a decision quickly. I cannot provide any details about how the backups are done right now. I'd just like to hear some opinions from people who run comparable environments and also use snapshot based backups. Should one vCenter server be able to handle backups of more than 2000 VMs? Is it worth trying to scale up the one vCenter server?

Thanks for any comments!

Andreas

[admin]

According to my sources-

With an external database, the vCenter Server 5.x can support up to 10,000 powered on Virtual Machines and/or 15,000 registered virtual machines (http://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf). Its not necessarily what we can support, but rather what the backup application can handle. That said, we only support ~100-180 concurrent connections to the vCenter Server depending on the API choice - see page 7 from the maximum guide - so that could limit the number of connections to the vCenter Server in one swing (and not to mention their backup vendor better have a good cleanup mechanism or it'll lock up the vCenter Server if they try to push the limit).

So, long story short, they need to be engaging with their backup vendor on those numbers in terms of number of connections that can be perf ormed by the backup 'appliance' as well as the number of VMs that can be done in one connection/session; some backup vendors allow for multiple appliances to be deployed to allow for more backups to be performed at one time.

[peetz]

Thanks Rick,

I was aware of the vCenter maximums for hosts and VMs (and that we are far away from them), but I was not aware of the client connection limit. I'm not sure if every connection through the API counts like a vSphere client or Web Client connection, but this might be part of the explanation, because - apart from backup - we have also many other things going on and connecting to vCenter.

However, I'm still interested in hearing some numbers of experience from other customers: Assuming that you have already optimized the settings of your backup software, how many VMs are people able to backup through one vCenter server?

Andreas

[rash127]

Hi Andreas,

   Rick had gotten you closer to where the bottleneck is. The client connections from vStorage APIs, vSphere client, web client etc. consume the connections from the same pool. There are worker threads in vCenter server catering to these connections. A good backup application should optimally use these connections such that connection consumption is minimized.

   NetBackup is designed (I work for SYMC, but opinions here are mine) and optimized to take this into account. In fact, NetBackup is perhaps the only product that has done so many scalability tests and benchmarks for large scale VM environments and added several features to get you the most out of it. Examples of such innovations driven by scalability requirements of customers are intelligent orphaned snapshot handling, resource limits and throttling etc.

   By the very nature of a vStorage based backup, you have at least one session active when a backup is going on. That means it is not practical to have more than 300 (assuming connection limit is 300) sessions concurrently. This is theoretical limit limit for a vCenter server, in practice the limit will be lower than 300 as there will be other consumers (vSphere clients, other applications etc.) connected to vCenter server.

  The situation here is not about "how many VMs can a backup software like NetBackup can handle through vCenter", it really is about "how many VMs can be concurrently backed up through a given vCenter". There are configurations with 10,000+ VMs protected by NetBackup. In fact, a single entry level NetBackup 5220 appliance had proven to protect 4800 VMs in an 8 hour backup window.

Hence what I would recommend is to limit the time for which the backup session should be active (in other words, do things to improve backup streaming throughput so that connection is released quickly).

   As you know, traditional full backups take longer to finish thereby keeping the connection longer. You have a few ways to solve this problem.

   1. Liberally use incremental backups with vSphere CBT enabled (versions < NetBackup 7.6)

   2. Use NetBackup Accelerator (NetBackup 7.6, this makes use of vSphere CBT, Symantec's own V-Ray based file system internals and Optimized Synthetic engine to provide you full backups at the time it takes to do an incremental backup)

   3. Use NetBackup Replication Director (NetBackup 7.6, for VMs on NetApp datastores only). This eliminates the need for transporting the data to secondary storage while session is active.

  More on NetBackup 7.6 is available here (validated by ESG labs): http://www.esg-global.com/lab-reports/symantec-netbackup-76-for-vmware/

   It is possible to increase the client connection limit for vCenter server if you are thinking about scaling up vCenter server. I wouldn't recommend to go that way unless you run out of other options. If you must, talk to VMware technical support.

Warm regards,

Rasheed

VMware vExpert

@AbdulRasheed

[cfizz34vmware]

Has anyone found a "sweet" spot for the various resource limits to set on the NB side of the house so that VC/SAN and NB are all happy?