Skip navigation
2017

Monitoring Veeam Backup & Replication with vROps

 

         

 

Backup servers, I think we can all agree, are of primary importance. We rely upon them for business continuity as much as we rely upon other systems of primary record like Active Directory or DNS. So why is it that we monitor the latter systems with such fervor and the former not at all? Often times, we don’t know that backup systems are impacted until we go to attempt a restore when it’s necessary only to find it doesn’t work or you don’t have the needed restore point. We need to bring the same type of monitoring visibility to data protection systems that we bring to others. In order to do so, let’s look at monitoring Veeam Backup & Replication with vRealize Operations Manager. In this article, I’m going to illustrate how you can do so with Veeam Backup & Replication 9.5 Update 2 and vROps 6.6.1. The basic idea is to use the Endpoint Operations (EpOps) agent, which I’ll install on the Veeam server, and report back to vROps with the status on the various Veeam services.

 

We obviously need a working vROps system, but in addition we need to download the EpOps agent for Windows from MyVMware (my.vmware.com). Once you’ve grabbed the Windows EXE package that contains Java, download it to the Veeam B&R server. Before you start the installer, however, let’s get a plugin for vROps that can monitor the SQL database portion of Veeam as an added bonus. Navigate to this page on the Solution Exchange and download the plugin. Install into vROps under Administration -> Solutions. There are many guides that illustrate how to install a management pack, and the same process can be used here for an EpOps plug-in so I won’t cover it. Go back to your Veeam server and begin the installation wizard and input the values it requests. The second step in the wizard asks for the certificate thumbprint.

 

This you’ll have to get by logging into the administrative interface of vROps at https://<FQDN>/admin and selecting the button in the upper-right-hand corner. From there, it’s usually going to be the second certificate you see in the list.

 

In my case, this is a lab so I’m just using a self-signed certificate. Copy the thumbprint and paste it into the installer. Complete the installation by providing vROps credentials that have EpOps agent permissions. After installation, login to vROps and go to Administration -> Configuration -> End Point Operations and ensure your system is showing up and you have green icons. Beware it may take five minutes or so for everything to turn green.

 

Once all is green, the Veeam server is now reporting directly to vROps through use of this nifty agent. Now, browse to Environment -> Operating Systems -> Windows and expand the list. You should now see your Veeam server there and it will look something like this.

 

The first object in the tree with the “OS” in the monitor is the entry for the operating system entity, followed by the EpOps agent, and finally the SQL Express instance (or SQL Standard if you opted to use a full-blown SQL server). Clicking on any of these objects will give you the basic Summary dashboard, and under All Metrics you can see distinct metrics that each is reporting.

 

We can also see the databases on the system courtesy of the EpOps SQL plug-in we installed earlier, as well as metrics specific to each database. Notice how we did nothing to accomplish this? The SQL instance and databases are discovered by the plug-in. Very handy! And by the way, if you want even deeper visibility into how Veeam is using its SQL database, I’d recommend looking at the Blue Medora management pack for SQL as it contains a wealth of metrics that really allow you to dig deep into the application.

 

            Next, we need to add in the various Veeam services we want to monitor. If you’ve installed Veeam in a single-system setup, you’ll want to monitor everything there (most likely) including things like the Data Mover and Mount services.

 

In order to pick and choose the ones you want, we need to add them to vROps. Start by double-clicking a service in the list and note its service name (not the display name). Of primary importance is probably the Veeam Backup Service, so let’s start there.

 

The name is “VeeamBackupSvc” and this is what we must copy. From within vROps, click the Actions menu when you’re on the OS entity object and go to Monitor OS Object -> Monitor Windows Service.

 

Fill out the simple window and include a display name as you see fit. I just used the service display name for consistency.

 

I’d recommend setting the collection interval to 5, which is the default interval at which vROps collects metrics from external systems anyhow. Once done, you should now have a new object in the tree that represents the Veeam Backup service. Give it a couple collection cycles and ensure the object is collecting and shows online. Check out the metrics that are reported, specifically AVAILABILITY/Resource Availability (indicates if the service is up or down) and those under UTILIZATION (indicating CPU usage and in-memory size). Now to test that the monitoring of the backup service is working, stop the Windows service (obviously don’t do this during a backup job). Check the Alerts section in vROps and we should see something after a few minutes.

 

This generic alert indicates the object, whatever that is, has an availability score of zero or is down. Now with this alert, you can forward it across to your email systems, ticketing systems, or anything else that you may choose. In addition, you can build your own custom alert and symptom definition if you would like. The only thing left at this point is to add the remainder of the Veeam services and configure custom alerts and notification plug-ins. Now you should have a good understanding of the health of your Veeam server through EpOps agents and vROps.

vRA and The Problem of the vCenter Folder

 

 

Between speaking with customers, architecting and deploying solutions, and helping others out with their issues on the VMware Communities forums, I encounter a lot of problems and get a lot of info on how customers do business with their CMPs. One of the requests that keeps coming up over and over again is how to deal with putting vRA’s workloads into the correct folder structure that has already been established in an enterprise. Very often—especially in large vCenter servers—companies have an established folder structure which could be quite complex. It is not unheard of to see a tree structure ten levels deep. These folders not only serve the purpose of providing visual VM organization, but also in defining permissions, monitoring, and backup boundaries. Permissions can be assigned at those folders granting access to the responsible parties. Folders, for example with vROps, can segregate monitoring policies. And just about every backup application on the market can now use vCenter folders as their organizational concept when setting up jobs. So they tend to be very important to certain organizations in a variety of ways. When implementing a CMP it is therefore important that it be able to fit into the established folder schema.

 

vRealize Automation (vRA) is certainly capable of putting its VMs into folders. In fact, if you changed nothing out of the box, it deposits its deployed VMs into a default vCenter folder called “VRM”.

For smaller shops that don’t really worry about folders, maybe this is fine. Through use of custom properties, vRA can stash those VMs into a folder of your choosing. By specifying the property VMware.VirtualCenter.Folder, the value of which determines the folder, any system deployed from vRA will land in that folder.

 

Use of a forward slash (“/”) indicates a separate level in the folder tree. The above would yield the following result below.

 

Anywhere this custom property is present within vRA that a deployment “touches” will be inherited and applied. For example, if this property were applied at the vCenter endpoint level, a VM that is deployed to that endpoint will be affected by the custom property. Likewise, if this property is present on a business group, and a user from that business group requests a machine, that machine will go into the folder specified by the custom property. What happens if this property is present twice at different locations? In that case, an order of precedence applies where the property which occurs later overrides one earlier. For example, if the property were on both a business group and specified at time of request, the value at time of request would win. While this provides some level of customization above what the VRM default folder affords, it’s still not enough for many customers. The ultimate desire is to have vRA deposit systems into the exact folder given a number of variables that might include things like department, environment, geographic location, and application. That level of flexibility is just not possible in vRA today without taking to vRealize Orchestrator (vRO) and writing custom JavaScript code. Until recently, that last sentence was true, but now SovLabs has an answer that changes everything and gives you the power you need.

 

            With the 2017.3 release of SovLabs Extensibility Modules for vRA comes a new module called the SovLabs Property Toolkit for vRealize Automation. This module has two main abilities. The first is to change and manage custom properties on systems after they have been deployed.  This is extremely powerful, as this kind of functionality does not exist natively in vRA and requires custom vRO workflows and code today.  The second capability, the subject of this blog and the answer to our prayers, is to create dynamic property sets or groups of properties. With dynamic property sets, each property within the set can derive its name and value based on the value of other vRA custom properties or any custom logic utilizing the SovLabs Template Engine. With a dynamic property set, you can now derive a value for VMware.VirtualCenter.Folder from VM properties, including user-input values at request time. Let’s dig in and see this in action.

 

            Imagine the following scenario, which I’ve taken from a real customer setup but made my own alterations:  Your organization is very strict in its vCenter configurations. With over 4,000 virtual machines, you have to be organized. Further, with multiple locations, multiple lines of business, multiple environments, and even more applications, things get downright dirty if they aren’t carefully organized. This is the folder organization we must achieve.

Backup jobs are organized by these folders with specific retention policies applied due to regulatory compliance requirements. And your operations team allows access to these applications from the app teams themselves, which means you can’t give open access to everything in vCenter. This scenario is very real for many organizations. Let’s see how the SovLabs Property Toolkit module can easily handle this type of structure.

 

            As shown above, we have a 5-level folder hierarchy in which we want vRA to stash machines that result from deployments. There are multiple options at each of these levels. For example, let’s start and assume we have the following specific structure:  Southeast -> Inside -> Sales -> Production -> Oracle. We need to create labels for each of these levels and put them around vRA for the module to consume. The reason is simple:  vRA services multiple regions, groups, departments, environments, and applications, and we must be able to distinguish between, say, Oracle and Sharepoint for each combination thereof. We’ll create custom properties for each of these levels in the hierarchy and apply them to the relevant constructs within vRA. This is a table of the custom properties I’ve created, their value, and where they reside.

 

Property Name

Value

Location in vRA

VCFolder.Region

Southeast

Cluster Compute Resource

VCFolder.BusGroupDept

Inside/Sales

Business Group

VCFolder.Environment

Production

Cluster Compute Resource

VCFolder.Application

Oracle

Blueprint (show in request)

 

The cluster (of which there is only one since this is a lab) has the Region and Environment properties, but imagine that we had multiple clusters, one for production and another for non-production. Those properties would then be split up. We’ve created a business group called “Inside – Sales”. There are others that begin with “Inside” but might be “Engineering” or something similar. The names are effectively two components of our folder naming plan. Lastly, the application is defined in our property dictionary and presents as a drop-down list of application names. This property has been added to our blueprint and is being shown to users in the request form. The reason being that, in this scenario, we’re only deploying IaaS and not PaaS, so applications have yet to be installed.

 

            Now comes the magic. We have our property names established and configured. It’s time to configure the integration. For this, we must create a new custom property beginning with SovLabs_CreateProperties_. This prefix is required and is how the module knows to invoke the action defined in its value. In this case, I’ve created SovLabs_CreateProperties_VCFolder. I’ve done this directly on a test blueprint we’ll deploy in just a minute. The value is the most important part as it is JSON describing how the VMware.VirtualCenter.Folder property is to be constructed.

 

{ "name": "VMware.VirtualCenter.Folder", "value":"{{VCFolder.Region}}/{{VCFolder.BusGroupDept}}/{{VCFolder.Environment}}/{{VCFolder.Application}}" }

 

With this value, we are telling it to templatize the VMware.VirtualCenter.Folder property from all the constituent parts it comprises. The forward slashes denote the hierarchy.

 

If we go to our catalog and select this item, we then are asked to tell it what application we want.

We submit the request and stand back for a second. Actually, let’s check vCenter and see what vRA does.

 

Ah, so we see the individual folder create tasks hit vCenter. By default, vRA will look for the value of VMware.VirtualCenter.Folder, and if it doesn’t exist it will create every level that is present. This shows we’re on the right track with a result.

Alright, that’s what we want to see! Is that not cool?! Once the machine is built, we can check the properties tab (presuming you have business group manager role assigned) and see the resulting custom properties from all over that applied to the deployment.

 

So as to be expected, the module gathered our custom properties from all around vRA and, based on the template we defined as the value of SovLabs_CreateProperties_VCFolder, patched together the VMware.VirtualCenter.Folder property dynamically at build time all without having to write so much as a single line of JavaScript. That’s pretty amazing and powerful functionality right there, especially keeping in mind that if another user were deploying from a separate business group, the resulting folder would be different, so entirely dynamic.

 

            I hope this small tutorial has piqued your interest in the possibilities this module unlocks. The sky is the limit. Next time, we’ll look at another common use case for this SovLabs Property Toolkit module: Multiple property background assignment.