I'm looking to develop a structured approach to capacity planning. I'd like to get some input from others who have attempted to formally approach capacity planning in a large cloud environment. The plan has to work itself out to accomplish certain goals.
The various stakeholders in the environment need to meet on some regular basis to review capacity in the environment. Based on current trends they need to determine when there will be a resource shortage in each area, and plan to make the appropriate purchases before those resources run out.
Every part of the datacenter needs to be considered. Storage, networking, compute, physical space in the datacenter. They need to be considered both from a sizing and from a resource utilization perspective.
For example, we need to categorize the following:
Storage:
- How much space is being used and when will it run out
- How much IOPs and throughput capacity is being used and when will it run out
Network:
- How many IPs are being used and when will they run out
- How much bandwidth is being used and when will it run out
Compute:
- How full are compute clusters. If we continue adding VMs at the current rate when will we need more servers.
- How fast are processor and memory resources running out on clusters. Which will run out first and when.
Datacenter
- When does physical space run out in racks and in chassis
- When do power supply resources run out
This is not an inclusive list, but you get the idea. Keeping track of everything is an effort requiring a broad vision to many areas of the datacenter across many teams and siloed areas. For each of these considerations, someone has to be responsible and we have to have a way of tracking the current state of each of these areas. It all has to be transparent to the cloud layer where people need to be able to be able to assume that the capacity has been planned for underneath them in every respect. Given that these issues are common to any growing cloud environment, what are your stragies, programs, methods for accomplishing these goals?
1. Do you use a single piece of software to track all this and make sure you keep tabs on everything from a single pane of glass? For example, to track when you need to make purchases, who is responsible, what the current usage is, etc.? Do you use Excel? Is there a template out there somewhere with the different areas listed such as above, with a link to it designed to help you think of everything?
2. Do you have weekly, bi-weekly, monthly meetings with all teams to review each area and see the state of each area?
Software like vCOps is great to estimate when your cluster resources run out, but it can't do those "what-if" scenarios on every aspect of the datacenter. It can't tell me when I will run out of rack space, physical switch ports, Fibre Channel switch ports. It can't tell me when my physical switch backplane runs out of available throughput. Yet if I am viewing the cloud environment holistically, and planning for rapid growth of the cloud environment, I need to keep tabs on how growth will affect every area of the datacenter, and make sure all teams are on board and I have make proper accomodations. I cannot just track only when my physical server runs out of RAM.
3. Do you have a formalized document that defines everything you track and makes all concerned parties accountable for their part in monitoring, reporting and capacity planning? Is there a template that you use or have found helpful?
Thanks!
