Now my period of non-compete is over - I'm free to now blog over at mikelaverick.com
Content here has been copied over there...
Now my period of non-compete is over - I'm free to now blog over at mikelaverick.com
Content here has been copied over there...
[As I was working through my "unpublished" blogposts I found this - which should have been posted AGES ago...]
Remember for sysprep to join a VM to the domain you will need to have DHCP on your network where your vApp resides - alternatively you will need to use Sysprep to change the SID, and then use a script and something like netdom.exe or PowerShell to add the VM to the domain. This is because ALL sysprep "clients" start their life as DHCP clients, before any static address is assigned to them. This is true in vCenter and vCD.
One of things that foxed me recently was why after importing an .OVF template into vCloud Director catalog are all the guest customization options greyed out? I first I thought it was a problem with my .OVF file, and then I thought perhaps it was user rights issue.
Joining a computer to the domain using vCloud Director Customization
It turned out I had problem with my expectations. Really these options open out once you have created a vApp FROM the catalog. Once the vApp has "deployed" its possible to edit these values on each VM within the vApp and modify them. That can include such things as forcing a complete guest customization or a reset of the administrators password. What I do like about this customization is the way vCloud Director can pick up on the Organization's OU location as well. My plan was to have VMs within a vApp join the domain automagically without having to do by hand - and to also ensure they were dropped in the right OU.
With this configuration (Use Organization's domain) above vCD use the LDAP configuration set for the Organization (held in OrganizationName, Administration, Settings, LDAP) and uses the account specified in the System LDAP configuration (System, Administration, System Settings, LDAP)
In my case I wanted to drop the computer object into a sub-OU called "Servers" at this path ou=servers,ou=CorpHQ-Organization,DC=corp,DC=com rather than the computer accounts being located in the CorpHQ-Organization OU. So I overrided the Organization's domain with specific path, and I also did a spot of delegation in Active Directory to allow the account called "email@example.com" to have the right's to join computers to the corp.com domain, but specifically only to the path ou=servers,ou=CorpHQ-Organization,DC=corp,DC=com
However, for this to work with my vApps on the Organization Network, I need to configure the NAT on the Edge Gateway to allow communications to the domain controllers (something I did a while back). Additionally, I needed to enable DHCP on the networks where my vApps reside. This can be in one of two places - either on the vCNS Edge Gateway that services the Organization Network or the vCNS Edge Gateway that services the vApp Network.
For the Organization Network (OrgName, Administration, Cloud Resources, Virtual Datacenter, vDCName, Edge Gateways, right-click Edge Gateway, Edge Gateway Services)
For the vApp Network (OrgName, My Cloud, vApps, select vAppName, Networking tab, right-click vApp Network, Configure Services)
Join the domain by script - without requiring DHCP:
Now, admittedly it could be the case that your "not allowed" to run DHCP on your network where your virtual servers reside. For me that's a bit funny for a couple of reasons. Firstly, the management of the DHCP lies within the scope of the person who manages the Organization or the vApp - not the network guys. Who are these network guys to tell me what I can do in my own vApp? Secondly, by using vCloud Director were all ready handing out network IDs in the shape of VLANs, vCD-NI, Portgroup or VXLAN enabled networks - how's a little DHCP going to harm things. Thirdly, I regularly use PXE to either install ESX (UDA or EDA are good virtual appliances for this!) or for AutoDeploy. PXE requires DHCP - so what gives...?
But seriously, I know that some shops see DHCP on server network as some cardinal sin. For that reason the other option is to use scripted methods.
Firstly, you need create this directory structure on the source VM that will become your vApp Template: C:\Windows\Setup\Scripts. On a Windows 2008 R2 server I found that the scripts directory did not exist and I had to create it.
Next, create .cmd file called: SetupComplete.cmd
Finally, edit this cmd file with notepad and add the required entries for the netdom.exe command:
netdom.exe join %COMPUTERNAME% /DOMAIN:corp.com /OU:ou=servers,ou=CorpHQ-Organization,DC=corp,DC=com /userd:corp\joindom /passwordd:Password1 /reboot
Note: Userd and passwordd are not typos. The extra D represents that this is a domain account used for joining computers to the domain/ou.
Finally my other tip for handling customization comes not from me but from Rick Vanover (and fellow vExpert) who works over at Veeam. He's got a good blogpost on TechRepublic which shows how to use the tzutil to set the correct time zone. I thought I'd fixed this issue in my template in vCenter, but seems like the Sysprep process as triggered by vCloud Director resets the TZ to be Pacific Time. Although my bogus company Corp.com is based in New York, as I'm in the UK and the lab is in the UK, I set my TZ to be GMT. I found Rick's article fixed my issue just fine:
So what's the best way? Well, I actually like the script method. As it doesn't require me to configure DHCP on every Organization Network or vApp Network. Enabling DHCP just once on the Organization Network is one time configuration that I don't mind, but having it remember to do it for each and every vApp Network I create - and remembering to do that BEFORE the first power on seems to be a bit of administrative burden to me. But that's just my opinion. I'm tempted to have Windows 2008 R2 vApp Template - one that does the join and one that doesn't - which seems a bit excessive but the reality is not every VM I created in my lab needs to be joined to the domain. Although it can be handy because I can use the GPOs to turn off the Windows Firewall and enabled Remote Desktop Services. One thing I've noticed is if you take a vApp that has domain membership already, and copy that into the catalog - even when it gets customized it doesn't seem to join the domain properly - it seems to think its existing domain membership is good enough - when actually a new computer account needs to be created in the OU. I've experimented with taking VMs within the vApp out of the domain before importing them into the catalog - and this seems to work much better. It means they are sysprep, renamed, and joined to the domain as net-new entities.
[As I was working through my "unpublished" blogposts I found this - which should have been posted AGES ago...]
Once vCD has imported and the welcome options dealt with (EULA, Licensing and so on), you will be able to login to vCloud Director proper. The default user name is admin and default with the virtual appliance version, but you can easily change this under the "Administration" tab, and the Users node.
As with a great many products from VMware now there's a step-by-step wizard that will take you through the main configuration, and these "light" up as each step is completed.
So my first task was to make vCloud Director aware of my vSphere environment, by adding in my vCenter system. That's where I hit my first change in vCloud Director 5.1. With the release of vSphere5.1 we now have new "single sign on" (SSO) service. That enables the ability to use your directory service credentials to logon via the SSO service to another service that is registered with it. That was something I hadn't done with vCloud Director yet.
As you can see vCD 5.1 is fully-aware of new web client that shipped as part of vSphere5.1. You have two options on how to make vCD aware of the Web Client URL either by using the lookup service (which lists all the service registered with SSO) or by just manually typing the URL. Quick google of the phrase "Configure vCloud Director to use vCenter Single Sign On" took me to the online documentation. The configuration to register vCD with the Lookup Service for SSO is located under the "Administration" tab and "Federation". There's a small button that allows you to register vCD with SSO. In my simple configuration using the vCenter Server Appliance - the vCenter/SSO/Web Client all reside on the same instance, but remember it is possible to split the services out into a more dedicated configuration.
You need to be a little bit careful at this point. It's important not to log out of vCD, before first including a "domain" account from the directory service that you configured with SSO - in my case this is the "corp" domain. The next step is to "import" accounts from the directory service (in my case the Corp domain hosted with Active Directory) into vCD and to make sure that account has rights.
The alternative is not to use the SSO service, and instead manually specify the vCenter/Web Client in the UI - of course whatever method you use to communicate to vCenter - you will also need to supply the IP/FQDN of the vShield Manager that is part of the vCenter instance.
Once you've completed this task you will find that "Attach vCenter" on the "Home" page becomes "Attach another vCenter". Additionally, you should also see it in the "Manage & Monitor" tab, under > vSphere Resources, >vCenters. Of course, you can add additional vCenter here as well. I noticed two things here - first the right-click option to open the Web Client on the vCenter5.1 appliance, and also that the vShield Manager IP address is exposed. In fact despite specifying the vShield by FQDN, the edit dialog box shows not the name but the FQDN.
Once you have vCenter listed in vCloud Director you should see it appears as a solution in >vCenter Solutions Manager >Extension Types:
The new version of vCloud Connector (vCC) was released just on the cusp of Christmas, which was very timely for me - as I've got to the point in the vCloud courseware. In case you haven't come across vCC before - its main job is broker access to public cloud providers who are based on vSphere/vCloud Director - but it also allows you to connect multiple private clouds together. There's really 4 different types of connection you can configure:
It's this configuration in bold I'm aiming for first as I own and control all the parts of the process... that's connecting a private vSphere installation to my private cloud... and vice-versa. In later posts I want to allow a private vSphere installation & my private vCloud to connect to a public provider. With a private vSphere to private vCloud it will allow my vSphere users to create VMs/vApps in vCloud Director from the vSphere Client. Previously, I would have manually exported the VM to an .OVF and then manually imported it into the vCloud DIrector catalog, and then performed a deployment - what I'm looking for vCC to do is automate that process substainitial...
Here's a quick summary of vCC function with the new 2.0 release
|View, copy, move VMs and templates||Yes||Yes|
|User interface improvements||Yes||Yes|
|Transfer speed and reliability improvements||Yes||Yes|
|Cross-cloud search for VM or template by name||Yes||Yes|
|Automatic catalog synchronization across clouds||No||Yes|
|Migrate VM while maintaining IP and MAC addresses [VXLAN required]||No||Yes|
For me the big new features are really "Content Sync" and "Stretched Deploy". The first keeps your catalog your private vCloud Director build in synch with your catalog public vCloud Director - the second allows you deploy VMs in the public cloud but have them communicate to VMs in your private cloud.
You can download the latest version of vCC here: DOWNLOAD and documentation both online and in a offline format is available HERE. There's three parts to the vCC - The UI, the Connector Server and Connector Node(s). The admin guide has this handy graphic that outlines the relationships:
As my private configuration calls for the easy ability to deploy to/from vSphere/vCloud I will deploy to vCC "Nodes"...
There's a tendency to gloss over these sorts of diagrams because, lets face it there are a bit dull sometimes. But it is useful to KNOW this especially if you embarking on vCC configuration with a 3rd party. Life is pretty much simple is you own all the pieces in the game, but as soon as you working with a 3rd party you need to know what you need from them, and what requirements are ideally suited to offer you (the consumer) the smoothest of experiences.
Firstly, encryption. All comms to the vCloud Connector Server on clear-text http UNTIL you enable SSL and either use a self-signed cert or replace with a fully-trusted. Secondly, all comms to the vCC Node are SSL by default, and use an untrusted comms. If your a Service Provider... you need one vCC Server and one vCC node - that's the same if your a private consumer. What you need as the consumer to register your Service Provider is - the URL for the nodes - like https://vccnode.cloudsRUS.com - your Organization Name, UserID and Password. So where its public or private you need precisely ONE vCC node per resource location (vCloud Director and/or vSphere).
Finally, quiz your selected partner over versions used. If your on vCloud 5.1/vSphere5.1 using VMs that have a hardware-level of 9. You vCloud Provider could be running vCloud 1.5/vSphere4.1 using VMs that have a hardware-level 8. This could scupper your planned use of vCloud Director (because you might be running 2.0, and they might be running 1.5). Even if your chosen provider is on vSphere5.1/vCloud5.1 given that vCloud Connector was released on the 21st Dec. Be careful they might not support it yet...
The UI of vCC can either be accessed by a web-browser (covered in the next post) or via the vSphere Client (covered in this process). The Connector Server is the central point of management, with the "node" being deployed either in either vSphere/vCloud Director - which helps with the transferring of VMs from location to another. As those transfers can become larger, there is a network resume feature in case there is an outrage during the transfer - it will be resumed once the network is available again. You only need one node per vCloud/vSphere instance as it is multi-tenant aware - that means you do not need a "node" instance for each and every Organization in vCD.
Double-check the supported web-browsers wtih vCC. I found my version of FireFox and IE were incompatible, making some webpages blank in the main admin pages of vCC Server and Connector. I found Google Chrome worked without an error
The vCloud Connector Setup Routine...
I chose to do the import directly into vSphere using the all-new vSphere Web-Client - rather than importing into vCloud Director (which is also a possibility - you add in a template to the catalog, deploy it - and make it accessible within an Organization using a destination NAT rule). After power on you can connect to its service page by the URL of http://vCC.corp.com:5480 - the default username/password for both appliances are admin/vmware.
1. Typically the first thing I do with any VMware Appliance is:
Set the Timezone (System, Time Zone)
Set the hostname (Network, Address)
Reset the admin password (Server, General)
and in the case of the vCC Connection Server input the vCloud license key.
2. I then repeat the same configuration (sans the licensing) part on the vCC Connector Node
It's worth mentioning you might not need to do this - for example if your using vCC to transfer VMs from vSphere to public cloud provider - what you will need is the node details of the Service Provider. In my case I'm setting up a node to access my own private vCD instance from vSphere. What's a bit wierd about this (this the inception bit that comes from just having a lab environment) is that very same vSphere environment is hosting the vCloud Director lab.
3. Then I needed to register the Connector Node with the location I was going to transfer stuff too - in my case I registered my connector with my vCloud Director instance using the URL https://mycloud.corp.com/cloud - I select to Ignore SSL certificates because none of my appliance use fully-trusted certificates. You can specify the name/IP address of your vCenter server (by selecting vSphere from the pull-down list). You might find that a little bit confusing the idea of a "Cloud Type" being "vSphere" after all vSphere on its own can't be construed as a "cloud" in its own right.
Really the important thing here is not where the vCC Node runs, but the cloud type and URL - as this controls the resources it will be able to access.
4. Once a node is "registered" with a cloud type, then you can configure the vCC Server to access the node. This means logging into the vCC Server, and selecting the nodes tab, and clicking the "Register Node" tab.
Note: The "public" tick of box is used if the cloud provider is external to your company. In hindsight the name "CorpHQ Organization" was a poor name - and I changed it to Corp Private Cloud instead...
5. The vCC Server can register itself with vCenter - to allow easy access to the vCloud Resources - this option is held under the vCC 'server' tab, and the vSphere Client...
This registration process adds vCloud Director icon to the vSphere C# Client like so:
The icon is merely a pointer to a web-page, and as such is dependent on the web-browser security settings. If you your using Internet Explorer, the recommendation is to lower your security settings to "medium" for it to work without error messages.
6. Under "clouds" its possible to add in a cloud provider - using the green + symbol, and specifying the username/password for the Organization:
Important: The account you use here is significant - as its privileges will dictate what actions are possible. In my first configuration I used a vApp Author account (firstname.lastname@example.org). This was a mistake because when I tried to copy a vSphere Template into my private cloud I received an access denied - that's because a vApp Author doesn't have rights to add stuff to the vCloud Catalog...
Once completed this will allow you to see the Private Cloud, the Organization and the Virtual Datacenters and beyond...
After my rename - the UI was a bit neater...
7. Once configured - a vSphere user can browse the vCloud Director catalog, and deploy a vApp from there... In this case I select the cloud, the vApp Template, and the Deploy button:
8. Deploying and registering a vCC node proved to be so easy I decided to create anoter vCC node, and register it with my vSphere installation.
and I then registered the vCC Node with the vCC Server:
Once you have vCC Node for both vCloud and vSphere - then you can do things like selecting a template from vSphere and deploy into the vCloud environment....
10. Once the vCC is aware of the private vSphere implementation, you can add into the vCC plug-in the vSphere Client as we did previously with the private vCloud instance.
11. In the vCloud Connector plug-in select the vSphere template (in my case I selected the Oracle Linux template I have which has Oracle eXpress Edition pre-installed - we can then copy the template into vCloud Director's catalog, and then deploy it once it has been uploaded - you also have the option to delete the template from the vCD catalog.
12. This allows you to select which cloud, catalog you want to deploy to... One thing you will notice when you do this deployment - is that you cannot control the storage location in this case vCC uses the default storage profile selected when the Virtual Datacenter was defined.
The vCloud Connector Install/Configuration guide has this useful graphic that shows you step-by-step what the relationships are:
1. Customer requests transfer using vCloud Connector UI.
2. vCloud Connector Server tells vCloud Connector Node to transfer vApp.
3. Node tells vCenter Server to "export" using VIM API.
4. Content is moved from datastores to source Node cache.
5. Content is transferred from source to destination Node using checkpoint-restart
6. Destination Node calls the VCD API to "import".
7. Content transfers from destination Node cache to VCD transfer server storage.
8. VCD sends the command for the appropriate vCenter import.
9. Content transfers from VCD transfer server storage to destination datastore network and is made available through the VCD catalog.
[Alternative Blogpost Title: Follow your personal unicorn...]
One thing I've been investigating is whether I can get ESX, vSphere, vCloud Director all working WITHIN a vCloud environment. The answer is yes, this is of course possible. After all its principle behind our hands-on-labs at VMware. I'm pretty sure I could get this working within the domain of my own lab environment, but what I would really like to do is try to this out on a public cloud environment. There's a couple of agendas here really. Firstly, the hardware resources required to get a REALISTIC vCloud environment up and running are not insignificant - not beyond the reach of the common-man but still a bit of an ask. You could go down the route of buying an array of inexpensive PCs or tower-servers which is popular amongst homelabbers - or you could go for the vTARDIS approach - where you have one jumbo PC with truckload of Memory/SSD to make it useable. Alternatively.... you could try taking the nested ESX concept to the next level and have it hosted in within a vCloud Server Provider environment - you call this by a number of terms - cloud nesting (where use one cloud to nest another cloud within it), nested clouds or clouds within clouds sounds rather nice too. Anyway, I'm going to play in my own lab to get this thing working (mainly so I can learn what the requirements are), some of my buddies from the HoL are going to give me an idea of what's required - and I'm also going to try and persuade a vCloud Service Provider I know if they'd be prepared to support a PoC. I'm thinking VMUG Advantage+VMTN Subscription+LabCloud might make an interesting proposition - heck, chuck in some credits for VMware Education you'd have a good package of resources for learning everything VMware.
Personally, I know that I'm starting from a low-base. I've never done Nested ESX before - and I'm interested to see how some of the automation that vCNS Edge Gateway does, but.....
As I see it there are number of challenges. Firstly (HARDWARE), as you may or may not know nested ESX does require particular hardware requirements, as well as options available within the web-client. These are precisely the sort of controls that you might not have access in a vCloud environment. To be honest I personally think the hardware requirements whilst VERY IMPORTANT in homelabs can by and large be taken for granted in public cloud environment - where modern CPU sets should be in use, with the attributes turned on in the BIOS.
As for enabling "nested ESX" Life is much simpler in ESX 5.1 because of the ability to add the creation of ESX VMs in the /etc/vmware/config file - a good account of this sort of nesting where you have "ownership" of the physical ESX instance can be found - a good account of this process can be found on William Lam's blog - How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5
Similiarly, vCloud Director 5.1 (assuming that's what your provider is using vCD5.1) has an option to expose "hardware-assisted CPU virtualization" like so...
Secondly (VM Hardware Levels), you have little or no control over what hardware-level the vCloud Service Provider has enabled. As you may or may not know - when a Provider vDC is created in a vCloud Director the SysAdmin sets the hardware-level.
That can cause problems later on down the track when it comes to importing OVFs. OVFs make a virtual machine portable, but even it cannot buck the VM Hardware Levels. So VM created on HW Level9 in vSphere 5.1 and then exported as an OVF, and then imported to a Provider vDC that support HW Level 8/7 will create this error:
That can cause challenges with .OVFs and OVAs that have been distributed via 3rd parties....
So I think the moral of the story is... if you going to go for this configuration - you need to confirm with your Service Provider what the hardware-level is for their Provider vDC. Remember we don't support downgrading the HW Level once a VM has been upgraded...
Thirdly (Networking) - the ESX host's management network needs to be set to "Promiscuous" under the Portgroup settings - and so do another NICs used by nested-VMs (VMs running inside nested-ESX hosts - keep up! it's like "Inception" this!). By default when vCloud creates Organization Network the security settings are configured to Reject, Reject, Reject...
The same is true of vApp Networks as well, and although from vCloud Director you can control IF you want a vApp Network - you cannot control the portgroup settings backing the vApp... without access to the vSphere layer... So that would mean having a vCloud Service Provider willing to make that customization for you... I'm also thinking of the networking requirements inside the inner-walls of your virtualized lab - how would VLAN tagging, VCNI, and VXLAN work within this virtual virtualization layer - this cloud within a cloud... That's a bridge I will cross when I get there....
There's a couple of things to consider about networking I think from a vCloud Director perspective. If you put the Cloud Nesting vApp on vApp Network, when you power it down the vCNS Edge Gateway is powered off, deleted along with its accompanying portgroup - unless you set the option to "Retain IP/MAC". If you put the Cloud Nesting vApp on an Organization Network you have to make the change to the portgroup once. Of course "promicious mode" is not for "free" from a networking perspective. By defintiion there would be a number duplicate packets on your network and as every VM recieves every packet, the physical switch has to process it and discard it.
All in all if you want your HomeLAb in the cloud with full rights to it - using a pay-as-you-go model to fund your homelab - I see many technical challenges, and you might find it winds up being no cheaper than the one off cost of some entry-level PCs stuffed with memory. That might change if a Service Provider perceived a market (in which they could make money) for selling VMware labs to people in the vCommunity OR alternatively.... if they saw an indirect value in doing so. Help a guy setup his homelab in a cloud... perhaps he might recommend that SP when it came to hosting in the cloud proper?
Anyway, I'm quite taken by the idea of trying emulate the VMworld HoL in my lab. Despite covering ground others far more able have trode before - its new to me, and I think I will learn loads along the way. So I might take a detour in my journal look at how this thing is setup. Plus I'm also think about fun ways to describe the layers upon layers of virtualization/cloud it entails. "Nesting" is nice, but I'm a fan of the "vINCEPTION" joke that circulates on twitter. So how about vINCEPTION0 to describe the physical layer with anncillary infrastructure VMs and your vCloud Environment "proper". vINCEPTION1 to describe Cloud Suite running in a vApp (ESX, vCSA, vCNS Manager, vCD) and vIINCEPTION2 to describe vApps running in the vINCEPTION1 layer...
All I need then is some kind of totem to remind me of which layer I was in. I was thinking of silver foil unicorn...
In my previous blogpost I was looking at DNAT rules with a vApp/VM on the "Organization Network". For peace of mind I wanted to look at the similar scenario if the vApp was behind a vApp Network. As you might recall DNAT rules allow you to "publish" VM's within a vApp and make them available to outside the organization. That could be to other organizations - or it could be to the Internet. In my case its to the Internet. Publishing a web-server for example behind a vApp Network is easy to do as another configuration. That's because all VMs within a vApp network get an "internal" address that's valid for the vApp Network as well as an "external" address that's valid for the Organization.
So in the screen grab above the web-server called "corphqweb01" internal address is 192.168.15.101 for the vAppNetwork - Web, and it's external IP address is 188.8.131.52. That means all I need to do is create DNAT rule at the organization that redirects one of my "external IPs" to the 184.108.40.206 address.
In my case that 192.168.3.11 address has been mapped from my physical Juniper Firewall. In fact I asked my colo to make the rule ANY:ANY so that makes the Edge Gateway my affective firewall. I dare say in the "real world" I would open port80 for one my "Internet IPs" (220.127.116.11) to the 192.168.3.11 address. That would give me three layers of firewall - the Juniper FW, The Organization Firewall, and the vApp Firewall. Such fun as Miranda might say, I'm sure Edward Haletky would approve!
As it is all I needed to do was review the Firewall settings on my vApp Network to make sure that port80 would be passed through it:
Anyway, the more I look at these "external networks" (mine is ridiculious based on the 192.168.3.x range as if I was on homelab with a WiFi router!) the more I'm thinking how different they might be say from a private cloud and public cloud. For a private cloud it seems "fair" to me to say that Organizations can use their external network to get to other entities within the Corp Holdings, Inc and the Internet. In this scenario it seems reasonable that the number of these external networks would be pretty small, and shared amongst the trusted members (the different organizations). However, when it comes to public cloud. I could easily imagine EACH Organization/Tenant wanting demanding their own EXCLUSIVE external network - and the IP associated with each of those external networks would be the standard 8-16 IP's assigned from an Internet range of address (18.104.22.168-22.214.171.124). Yes, perhaps this "external" network would be shared by multiple Organizations in vCD (Org1-Test/Dev, Org1-Staging, Org1-Production) but configured in such away that Org2/3/4 could not access them....
This configuration I'm doing between vCloud Director and View is largely unsupported. There's a couple of things that kind of breaks the rules here. Firstly, I had to give my "CorpHQ - EUC Solution" vApp access to a vCenter - which just happens to be where my Organizations are.
That's hardly the definition of multi-tennacy, where the Org's have no ideas what the underlying vCenter/vSphere environment is like! I guess I could have setup a dedicated vCenter environment just for running desktops, and done some delegation to keep one tennant seperate from another. I'm just using a service I know well as example, and as something to practise with. The DNAT configuration works equally well for something like a web-server on port 80....
One aspect of vCloud Director and the vCNS Edge Gateway I've not really looked at is Destination NAT rules. What are they and what are they for? Well, whereas "Source NAT" rules (SNAT) allow for outbound NATing for VMs/vApps behind an Edge Gateway to get to a wider network (the corporate network or internet) for example. Destination NAT (DNAT) rules allow for "Inbound" NATing. An excellent example of doing that is allowing services provided by the vApp to be accessible from the internet......................................
I have specific example. For years now I've have a remote lab environment which is reside in some kind of colocation facility. My remote access system of choice was Citrix MetaFrame XP F3. Yes, I have a really old and geriatric Windows 2003 32-bit host that I connect to get a desktop to then manage my system remotely. It's been around in some shape or form ever since I quite being a Citrix Certified Instructor (VCI) in 2003, and embarked on the path that would lead to becoming a freelance VMware Certified Instructor (VCI). Of course Citrix's ICA was far superior to Microsoft RDP which you know your remote desktop protocol his - was actually co-developed for Microsoft by Citrix. Anyway, for some years I felt it was time to move over to VMware View. The trigger was in the View 4.6 released which allowed for the first time the View Security Server to pass PCoIP traffic without the need for a VPN. [Those who know me well, will know how much I despise and loath VPNs!!!). So my use case was building a VMware View environment - and then allowing that to be accessible across the Edge Gateway using a combo of DNAT rules to handle the IP translation from the public-to-private, together with some firewall rules. One word of warning. Technically, what I'm doing isn't really supported as such. Allow VMware View is just another VM to me, it does need access to some kind of vCenter instance for the provisioning of desktops - and also the VMs that VMware View creates don't appear in vApps in vCloud Director, they appear in vSphere. Of course with a little bit of rule bending you can make it work in an unsupported way... The other thing I would say is given I increasingly don't live in the "real world" I'm keen to deploy technologies I know well into vCloud Director as way of forcing some kind of realism into what I'm doing - so there is some kind of real application that I'm using...
So I create a Windows 7 VM, and installed the View Agent to it, and then dumped into a manually create resource pool and folder on my Gold Cluster, calling it "CorpHQ - Desktop"
Then I created vApp which will contain my VMware View environment - I slight cheated into two ways - firstly by putting my Security Servers and Connection Servers on the same network as well as patching my virtual desktop manual to the Organization Network. That's naughty because the whole point of the Security Server is that should reside on separate "DMZ" style network. To be honest I wanted to keep it simple. So I would complete vApp that would eventually contain our entire EUC solution (View, Horizon, ThinApp Factory, and so on) that I could power up and down when I ever wanted to do any EUC work that technically falls outside my new role as its in different business unit. I believe in a world without walls - so intend to keep my hand in with our EUC stuff. On a more holistic level I see our cloud vision as encompassing ALL aspect of IT delivery - infrastructure, apps, and users.
Of course my "corphqparentvm" is configured for the CorpHQ Organization Network (CorpOrgNetwork), and its a DHCP Client, so I had to make sure that when it booted it would get a IP address and join the domain correctly. That meant enabling the DHCP service on my Organization Network for virtual desktops that get cloned either via the standard "template" method or using "linked clones".
After putting the bits in place - I installed a View Connector Server and View Security Server into the vApp, and configured them together. I appreciate you might know NOTHING about View. But basically when you install these services they get paired together and there's a configuration on the View Connection Server that allows you to configure what the EXTERNAL IP and FQDN values they are. This makes sure that when an external View session is established the client when it gets info back from the View Server(s) knows to query an external/Internet facing DNS/IP range rather than an internal LAN based one. Without this configuration when the initial hand-shake process begins an a computer in an Internet cafe on 126.96.36.199 would be told to connect to a View Connection Server on 192.168.3.20, rather than the external Internet IP at the datacenter (such as 188.8.131.52). Typically, that configuration would look like this when configured in the View Manager under View Configuration, Servers, Connection Servers, right-click and edit. This really has nothing to do with DNAT rules or vCloud Director, and everything to do View. But it does kind of illustrate that more multi-tiered your application is. The more you might have to be concerned about the configuration of the application layer - and this would be true even if you did something barmy like installing View on a physical box...
Okay, that's the preamble dealt with. Lets get down to the whole DNAT/Firewall configuration. Firstly, I have a couple of layers of firewall in my environment. At the colo I have Juniper Firewall, and then of course there's firewall layer at the vCNS Edge level. I nothing about Juniper, and if I make a request for a change I have to pay my colo provider to open ports and so on. That can be quite pricey. So I asked my colo to take two of my spare IP address (for example 184.108.40.206 and 220.127.116.11) and open them for ANY:ANY to 192.168.3.10 and 192.168.3.11. These are actually interfaces on my vCNS Edge Gateway in the CorpHQ Organization. That configuration affectively makes my vCNS Edge Gateway the affect firewall/NAT for that Organization. It means my Organization Admin has the rights to expose stuff to the Internet. In the REAL world I would take time out to learn Juniper so I could have two layers of firewalling.
So the first question is where do these 192.168.3.10/11 address come from? Well, when you configure the vCNS Edge Gateway for the Organization you assign a "sub-allocation pool" of IP address from External Network to the vCNS. Remember with a NAT-Routed Organization Network the Edge Gateway has at least two interfaces - one connected to the External Network, and other "internal" interface for the Organization - so my Edge Gateway for the CorpHQ "Production" Virtual Datacenter is 192.168.3.10 for the external network, and 18.104.22.168 for the internal network. [A little bit of consistency in my IP allocations would have made this neater - 192.168.3.10 and 22.214.171.124 would have been a bit cleaner. But hey, IP is IP and this is just lab!]. You don't REALLY need to know this "internal" interface IP BUT, you do need to know the external interface IP. Especially, if you doing what I did with my colo, and getting my external IP 126.96.36.199 redirected to 192.168.3.10. The sub-allocation pool is visible/configurable on the "Edge Gateway" tab (Select the Organization Virtual Datacenter, Edge Gateways tab, Right-click the Edge Gateway).
What's not commonly understood is the IP address that's used as the 1st connection to the external network (192.168.3.10) is valid for use, along side the other IPs in the range of 192.168.3.10-192.168.3.20. What's is also needed (and this pretty critically) is the IP address of the system your trying to DNAT to. Now in my first attempt I'm doing something VERY simple. I've got one View Security Server (188.8.131.52) connected to another View Connection Server (184.108.40.206). I do plan to eventually setup two View Security Servers and two Connection Servers - and then attempt to enable the Edge Gateway's load-balancing feature across them - something I did in the eucbook.com using F5's BIG-IP. All VMs on vApp get an IP valid for the Organizational Network - and even (and this is the important bit) do VMs on vApp Network connected to an External Network...
This means the vCNS Edge Gateway on the Organization Network could do a DNAT (call it reverse or inbound NAT if that works better for you) from 192.168.3.11 to 220.127.116.11, and the vApp Network would handle the translation to 192.168.15.101. In my case my View services are just on the Organization Network, but I do intend to look at the same configuration for vApp Network too...
The first connection that happens with any connection server is on 443. At that point after authentication - after that the View Client is given a list of available desktops. So the first thing I need is a DNAT rule to allow redirection to the Security Server for 443.
1. Navigate to the Edge Gateway for the Organization (Organization, Cloud Resources, Virtual Datacenters, Select your datacenter, Org VDC Networks Tab, Configure Services
2. Next under the NAT tab, click the Add DNAT button
So here, the traffic is coming from the Corporate Network - one of my external networks - that just so happens to be connected to the Juniper Firewall. The rest I think is self-explanatory the traffic is TCP/443 and being redirected to 18.104.22.168 using 443. So the DNAT allows for both IP translation as well as Port Translation. I tested this configuration by just cranking up a web-browser - this brought up the standard web-page for View which allows the user to download the client:
At the very least this configuration would allow for an RDP style connection to a desktop. In View a connection to Security/Connection server happens on 443, and Microsoft RDP packets can be tunneled in the SSL session. PCoIP sessions are also encrypted but the connection is made on 4172 for both TCP/UDP. Knowing this can be quite useful. For example if you can connect to View with 443/RDP but not with PCoIP/4172 it tells you two things - either your configuration is wrong OR the location where the user is at is blocking PCoIP/4172. Most locations I've been at allow outbound ANY/ANY, but some cheerless places only allow specific ports outbound such as 80/443...
So here the Mac View client successfully connected using 443/RDP (it actually cranks up the native RDP client for Mac)
Whereas when I try to connect with PCoIP the connection fails because the Edge Gateway hasn't got a DNAT rule for that protocol yet.
3. So clearer the next step is configure a DNAT rule to allow the PCoIP mapping.
That means I now have two DNAT rules for 22.214.171.124 - one for 443 and another for 4172.
I initially setup these rules with the firewall turned off (and if I'm honest with you - I was using ANY/ANY that would ALLOW absolute any TCP connection of any type to come in to 126.96.36.199). So remember all a DNAT could be very permissive, as it is my connection allows only for specific DNAT rule to happen to a particular node on a particular TCP Port, but I feel the proper configuration should also have a firewall rule as well.
4. So re-enabled the Firewall, and configured it to allow - and added my first rule to enable inbound 443 traffic:
and the second to allow 4172....
So here I'm allowing an traffic coming externally to connect to 188.8.131.52 on 1472/UDP/TCP...
... for what its worth this is really just a temporary configuration to make sure I can get this thing work. What I REALLY want to do next is setup my 2nd set of View Connection/Security Servers - and then enable the vCNS Edge Gateway for load-balancing. That's going to require a different set of IP configuration - so clients come through on different 172.168.5.x and load-balanced across two of my security servers.
This weeks chinwaggie clearly needs no introduction as the No1 Independent Blogger on the vCommunity scene - Eric Sloof. This week I was fortunate to sit in on the VMware vCloud Design Best Practises course that Eric was running in London. Although this course is still based on the older 1.5 version of vCloud Director it was still a fantasic course - because despite the obivious improvements in vCloud Director 5.1 (some of which may intail changes to those very same best practises) the principles and design challenges remain largely the same. The course is one of those which is very dependent on having a very capable instructor (that's sorted!) and very good students (that was sorted too). There was a good mix of folks on the course including a former student of mine who taught vSphere4, as well people who had done their fair share of implementations as well. For me it was an excellent opportunity to both learn from others, as well as putting forward my own concerns about whether I had "done it the right way". So look forward to a "debrief" post of mine in the next week as I run through the manual and my copious notes about how the course changed my views on the design. As ever with these courses you partly looking to have your own tentive ideas confirmed, or having them questioned...
After the course was over I was able to do an impromptu chinwag from the back of class with the man and legend - talking about vCloud Director, the challenges of design and the stuff that Eric would like to see VMware do next with vCloud Director.
In previous posts (the one's that all have vApp (184.108.40.206.5) in the title I played about with static routes - one of those examples was adding a static route to enabled communication from one Organization Network to another via the External Network. I did that because I wanted to know more and configuring things (even if they may not be optimal is one way of learning). It strikes me that a SSL VPN connection between Organizations would be more appropriate given that the data stream is encrypted as it moves from one Organization to another. I'm resisting the term site-to-site because in my case the two Organizations (CorpHQ and COIG) actually reside in the same physical location. But I guess the principle of VPN could be used to link to Orgs within and outside of existing vCD cell configuration as well to external providers - in the elastic-burst-on-demand way that everyone speaks of nowadays.
The vCND Edge Gateway supports VPN connectivity via the Organization Network...
The SSL VPN Configuration allows for:
It's worth saying that all that work I did in the vApp Networking posts (1-5) isn't wasted. Just because there is a VPN connection from one Organization to another - it doesn't mean that the vApp Network is automagically exposed. There are couple of requirements to be met along the way - in some respects a lot of this configuration has already been done by me over the last couple of months:
IF there is a firewall present between the tunnel endpoints then UDP Ports 50,51,500 and 4500 must be opened (these corresponds to the ESP, AH, IKE protocols). Enabling the VPN configuration requires access rights from one Organization to another as it create bi-directional VPN configuration on both sides. You can either use the credentials at the destination Organization setup for the purpose or use the SysAdmin credentials if you have them. This initial configuration documented below would initially allow only vApps on the respective Organization Network to speak to each other... Although vCD push the configuration to both vCNS Edge Gateways in both Organizations. It doesn't enable it at the "Peer Network" or the destination location...
1. Configure the Services of your Organization Network - in my case I select the CorpHQ Organization Network
2. Select the VPN tab, and enable the VPN option - then click the Add button to create VPN Configuration
3. Type in a name and description - and from the pull down list select "a network in another organization"
4. This should cause the dialog box to refresh to show the "Login into remote vCD" button:
5. Click the "Login into remote vCD" button - this produces a form that allows you to specify the URL, Organization Name, Username and Password to access the other organization. In my case I used the admin credentials, ticking off the option to "Login as System Admin"
Note: Notice how its just the base URL of the vCD instance... nothing more!
6. Click Login - this should (if your authenticated correctly) cause the page to refresh again. It should show the Local Organization Network (the Organization where you started the configuration) and the remote or "Peer Networks" of the Organization you just authenticated to. You use the pull-down list to select which Organization VDC you want to use (COIG - Production Virtual Datacenter in my case) and then which Edge Gateway will be used (in my case this is the single Organization Network called COIG-CorpOrgNetwork). It's entirely possible for any Organization to have MANY externally NAT-connected Organization Network - so you must select which Organization Network you wish to use as the endpoint for each side of the VPN connection. When you click at the Organization Network's they are highlighted in dark blue.
7. Next, under this section review your VPN Settings. I found I didn't have to change anything here to make my configuration work.
8. Click OK.
9. Next login to the Peer Network Organization (in my case that's the COIG Organization) and enable the VPN link it is side.
10. Once enabled you should find you get nice green ticks next to enabled and its status
Once the VPN is up then you should be able ping from one VM to another VM on different organization network (assuming firewalls are turned off, or the appropriate rules are in place.) So here a ping when from the OrgNetwork 172.168.5.x to the OrgNetwork 172.168.8.x...
You should also a find a ping from vApp behind a vApp Network to another VM on different Organization Network works as well... So here I did a ping/tracert from one of CorpHQweb servers - it crossed its own gateway (192.168.5.1) and on to it own Organization Network (172.168.5.x) and then across the VPN link (* * * Request timed out) and then arrived at VM (220.127.116.11) on the other Organization Network (COIG)
Note: Sadly the VMTN blogging format has mangled all my formats - its just to hard to try fix this now. The next edition will somewhere else - so perhaps that problem will go away.
This is issue 1, and its likely subsequent issues will be vmware.com. We working on it...
Hello and welcome to the very first vCloud Suite Digest, a compilation of common technical questions and answers on vCloud Suite architecture and implementation. Co-authoring this Digest is my colleague, Pang Chen, a Principal Consultant in VMware’s Global Technology Solutions. Pang keeps tabs on current questions being raised by VMware customers and partners through SEs, PSO, Trainers, TAMs, and TSEs and helps track down their answers. Behind Pang is a legion of people- a group of people far too numerous to mention (for me these are the unsung heroes of VMware)- who provide definitive answers and guidance. People fixing problems on a daily basis, that don't get half the spotlight that jumped up evangelists like I do...;-)
I’ve partnered with Pang to add more context and detail, including occasional screen grabs, to the Q&A to showcase here in my own blog posts. Ever the self-promoter you see? :-D
But joking a part I’ve always thought before joining VMware that there was likely to be a mine of useful information that could be shared with the community. This digest is a classic example. So I see it as part of my “Senior Cloud Infrastructure Social Media Competitive Evangelist” role to find ways of getting this stuff to the wider world. Yes, that’s right I work with our competitive team and I spend all day bad-mouthing Microsoft and saying how awful they are… :-O So if next your boss is asking some loony-toon questions such as why you don’t run Oracle RAC on Oracle VM, you can contact me for help and advice…
If you like this Digest, we’ll compile more. And here it goes:
Q. Is SSO mandatory for vCloud Director functionality? When would I want to enable SSO for vCloud Director?
A. You have to distinguish between the vCenter SSO and SAML SSO. Neither is mandatory. The former is just for vCloud Administrators and you would enable it if you want to manage their access from vCenter SSO. You can still use LDAP authentication for vCloud Director admin purposes without SSO. When considering SSO, it depends on if you are talking about the System scope or Organizations. vCloud Director v5.1 supports an external IP for Organizations only, and does not support vSphere SSO for Organizations. vCloud Director 5.1's System scope, as well as vCenter/Next Generation Client 5.1 only support vSphere SSO, but not external IdPs.
Q. In vCloud Networking and Security v5.1.x, do we support SSO group-based role assignment?
Q. Resetting the SSO Administrator password automatically expires 365 days later by default. How do I change this expiration policy?
Q. I cannot mount a media from a public catalog without first copying the media to a private catalog. Am I doing something wrong?
Q. When sharing an Org VDC network to multiple VDCs, is there a way to determine the "owning" VDC for that network? Or does the network become detached from the original VDC? When I look at the vCloud Director web-pages I can't see where I would find that out?
Q. How can I determine what the instance name specified in the initial configuration wizard is for a vCloud Director deployment?
A. You can query the vCloud Database using a SQL statement: select value as "Installation Name" from config where name='SystemName';
Or look in vCenter > VM and Templates > the root folder holding all the vCloud Director
workloads should be the name of the vCloud Director instance.
Q. How can I find the Installation ID?
Q. I have a multi-cell vCD configuration with a load balancer in front. I have created a single keystore file with 2 certificates http and console proxy and the 2 FQDN of the load balancer (public http and console proxy address). During the vCD’s configuration of each vCD cell, will I use this unique keystore file? So in this case, what will be the impact if the load balancer is not available? Can I still connect to the cloud directly to one vCD cell (private network)?
Q. Is there any way to have with vCNS Load Balancer configured in Active/Passive mode?
Q. I am trying to set up a pVDC with vCDNI network pool but I am getting an error that I am not licensed for VXLAN (which is obviously not included by default in vCloud Director). How can I configure my pVDC with vCDNI network pool?
Q. Does vCNS v5.1.x provide any way of checking what ports are OPEN?
Q. Is it possible to downgrade a vCNS Edge Gateway from full to compact?
I've been at VMware since August occupying a relative new role within the Cloud Infrastructure business unit as the Senior Cloud Infrastructure Evangelist. Yes, I know I sometimes wince at the job title myself. Over here in the UK/Europe "evangelism" has these religious connotations that make some of us feel uneasy (or even quesy). I've been thinking that if anyone asks me what my job title means - I will say that involves staring off into the middle-distance as drone on about how you need get VMware into your life, like I've just drunk the Kool-Aid.
What many people don't know is what team I'm in here at VMware. It might surprise you. I'm actually in the "Competition Team" at VMware. Most organizations the size of VMware have team of folks who's job is help customers, partners, the field to counter the Great Wall of FUD that emanates from companies fighting for the same customers. In my team is Eric Gray (@eric_gray) who is the dude behind the vCritical blog, and my colleagues are the people behind such things as the "Virtual Reality" blog and the Get The Facts portal. Given my previous life as trusted independent I've askewed using this blog to openly make competitive style statements. It's my personal belief that you (my loyal and lovely followers) would find that a turn off. Rightly so.
For the record I believe competition to be a good thing - its keeps us lean, mean and efficient - and stops any company from becoming a loose, baggy and complacent - assuming it has the right to your business and your money. Competition clarifies, cuts through, and captures the essence of the evolutionary spirit - to misquote Gordon Gekko from Wall Street. Competition is good for customers and its good for the industry generally.
In the last couple of months I've intermittently contacted by member of the vCommunity for help on competitive matters. For example there was this partner who had meeting where the "Oracle DBA Guys" thought that Oracle DB/RAC should run on Oracle VMM. I helped him out with the reasons why that might not be smartest move in the book. Anyway, what I want to say is if there is anyone internally (a VMware employee) or externally (VMware Partner) who wants to use me informally as conduit for getting the right information - I'm here for you. Sadly, I can't extend this informal offer to customers. Why? Well, a lot of that information is business sensitive and contains IP, and in the wrong hands could leak out to our competition - and when you in competition the first rule is not give the ball to the opposing team...
If you think I can help you, feel free to contact me via twitter or by "community email" - mike AT mikelaverick DOT com and I will do my best....
I'm not really much of "gadgetman" to be honest... But there's been something I've been lusting after for a while - in-ear style bluetooth headphones. I opted for the TechReview: Plantronics BackBeat 903+ Stereo Bluetooth Headphones. They are pretty good - although you do have to fiddle with the "telescope" (up and down) movement to get them to fit on your ears properly (I've got little "pasta shell" ears!) They paired perfectly fine with iPhone, iPAD and MacBookPro - and they work with ordinary calls, and Skype admirably. My main reason for wanting pair of in-ears like this - is one of the 1st-World problems I have is getting tangled up in cables, and I also wanted something neater to wear when recording chinwags/podcasts.
The pairing process was painful - and the device appears like an AppleTV or any other "Airplay Device"
Do they work. Yes. How I want them too. Not really. Firstly those plastic lugs that are meant to push into earholes - don't really fit properly - especially with my pasta shell ears. Having previously owned some Bose in-ear headphones with the ergo-nomic lugs, this feels like a step back:
I was given these by a sponsor as thank-you gift for being a panel session at VMwold 2011. Sadly, I left them on a plane last year. :-(
I still have different shapped plugs from Bose (you get backup set because not everyone ears are the shape and size...)
The other thing that doesn't really work are over-the-ear buttons on the Plantronics... so you do wind-up hunting for the device (the iphone) to adjust volumes and skip tracks and change volume.... But that's made easier by having no cables...
The other thing I would say is how the "feel" on the ears. A bit weird. Especially if you walking around town in coat & scarf. You go turn your head to check you left-side for on coming traffic and it feels like they will come of your ears. So there's tendency to want to turn your entire body to look around. So long as you forward facing all the time, your okay, but in a urban environment where your looking around a lot less so. That said when your indoors and don't have scarf or coat-collar in the way there's more movement.
Finally, the built-in mic. Works perfectly fine for calls and Skype - but I'm not convinced by their use for podcast recording. Quality seem "choppy" via the MBP. So in the end I used a hybrid model. Platronics as headphones, with the microphone being my Blue Snowball Podcasting mic. With the benefit of hindsight perhaps the Plantronics Backbeat GO In-Ear Wireless Bluetooth Headphones instead...
In a previous blogpost I skirted around the issue of the firewall with the vCNS Edge Gateway, as used by vCloud Director. It got snuck in during a blogpost about monitoring - because monitoring includes syslogging, and syslogging is used by the firewall rules is was introduced in a rather Trogen Horse kind of way. In case you have realized all those post about static routing we done with the firewall switched OFF, as that aided troubleshooting - that of course, is neither best practice or realistic. Even if you do allow traffic from NetworkA to NetworkB within or without of an Organization its unlikely to be unfiltered. So I decided to play around with my configuration of the firewall on my "web" vApp Network and see if I could tighten the screws a little more.
The configuration of the firewall component can get quite involved - the more vCNS Edge Gateway's you deploy - each one will have it own firewall - which is turned on by default. It's tempting to keep the number of vApps and vApp Networks to minimum, since each vApp Network created generates a new vCNS Edge Gateway, and potential firewall management. I guess much depends on what your trying to achieve (as ever!) if you essentially want to protect those VMs and give them outbound access to the Internet that's a relatively trivial affair.
To keep things simple in my lab environment I create quite a permissive rule that allows ANY network to ping the VMs in my "Web" vApp Network. To do that I use the key words "external" and "internal". It means in my lab I can still use ping/tracert and so on to do tests. It lets me verify that the network path is good, and validate that its the firewall blocking the traffic and not some other source - such a bad route or NAT entry. I think there maybe practical use of this rule to allow for diagnostics. So the rule could be disabled during ordinary hours, and enabled for diagnostic purposes. This allows any inbound traffic coming in "external" to the vApp Network pass traffic across the vApp Networks Edge Gateway into the "internal" network using ICMP...
Additionally, I keep the default vApp Network firewall rule that allows the VMs speak outside of the vApp Network, onto the Organization Network to which they have been connected. This allows all "internal" traffic out across the "external" network (on to the Organization Network in my case) using any TCP/UDP port they need. That's quite handy for external services like querying the DNS service that either resides on the Organization Network or the External Network.
My next task was to grant wide access to the web-services residing on the VMs on the vApp. For testing purposes I installed Microsoft IIS to Windows 2008 R2, and replaced the default.htm file will basic file that said hello world - and name of the web-server in question. If you've followed my vApp Network series of post, you will know I setup static routing between vApp in OrgA to vApp in OrgB - in my case between the Web-Servers in the CorpHQ Organization to the QuarkReporting vApp in the COIG Organization - this allows traffic to flow from/to 192.168.30.x network (the QuarkReporting App) to the Web-Servers vApp (192.168.15.0). As a quick test I did pathping from one of the QuarkReporting VMs to one of the CorpHQ Web-servers just to validate the connection was good. I wish in my other posts I'd used pathping rather just ping/tracert as it seems to show more about the true direction of the packets.
So here you can see the coigqurkrepts01 VM speaking via its default gateway (192.168.30.1) on to the COIG Organization vCNS Edge Gateway (18.104.22.168) which then speaks across the External Network (192.168.3.x) to external interface of the CorpHQ Organization vCNS Edge Gateway, and then on to the vApp Network vCNS Edge Gateway (22.214.171.124) where web-servers residing on the 192.168.15.x network - in this case the packets arrive at the corphqweb01 web-server on 192.168.15.100.
So I wanted a more specific firewall rule than just internal/external allow - so I created a firewall rule on the vApp Network of the web-servers just to the VMs within the QuarkReporting vApp.
This rule blocks any access from VMs within the CorpHQ Organization - so I added a second file wall rule to allow the vApps on the Corp HQ Organization Network.
Here's a summary of my rules:
Once I had this configuration in place I just cranked up a web-browser on the 192.168.30.x network to see if I could reach http://corphqweb01
Ed Grigson is a vExpert and based here in the UK, I know him well from the London User Group - and I'd forgotten that I'd been his instructor in the previous decade when I was VMware Certified Instructor. Of course like every vExpert he runs a blog where he shares what he learns and his views called vexperienced.co.uk and when he's not doing real work he also tweets @egrigson . Of course, he's too busy doing real work to be social media butterfly like yours truely...
Here's a cut & paste job from Ed "about" page:
I love that line about "just to annoy @stevie_chambers". Stevie is known to have a hate-hate relationship with ITIL. I see both sides...
In this weeks chinwag we didn't so much have "questions" but more talking-points - so in the episode you will hear Ed give his views on the way Oracle are packaging the DB, OracleVM and hardware together in a single stack - is that a good thing? Ed's not so sure... He also talks about his views on using vCloud Director as "lab management" system, and some of the challenges he's faced in finding a good vCloud Service Provider... finally we talk more light-heartedly about his experience of being on Stephen Foskets Storage Field Day in Silcon Valley.
Now I'm done with vApps and their networking (for the moment - there's still VPN/Load-Balancing/DNAT to think about!!!) - I'm working my way gradually through all the labs in the "VMware vCloud: Deploy and Manage VMware Cloud" course. My plan is to finish these labs, and blog my experiences along the way. Once I'm done with these labs, I think I will be ready to attach the vCAT documentation - I'm hoping also to find time to RTFM the official admin guides that ship with vCD, in the hope to pick up tidbits that aren't covered in the official course. That's something I think students often forget. Most vendor based courseware take you through the top most common configurations and questions - they rarely take you through every-single-little-setting otherwise you'd never complete a training course!
One of the chapters in the course deals with monitoring. There's a number of "status areas" to check out. That takes the shape of number views:
Note: SysAdmin View...
Note: OrgAdmin View... It was bit scary looking at this tasks/events log - not because of the error messages - but looking at the number of events which was north of 5,000 I realized how much work I'd been putting into vCloud Director!
Of course there are system logs that help troubleshoot the vCloud Director software itself. These logs are accessible directly from the vCD cell itself located in /opt/vmware/cloud-director/logs. There's six core logs (which rotate) called:
From what I can gather the most valuable logs for admin troubleshooting are probably the vcloud-container-info and vmware-vcd-watchdog.log...
Note: You'll notice a number of JMX log files. JMX stands for Java Management Extensions. This an internal system used to managing and monitoring applications - each of the objects monitored are called MBeans (Managed Beans). Is it me is this referrence to coffee in Java no longer cool, but deeply irrating and unfunny...?
Cell.log if you open it with vi general shows the start-up information of the vCD cell itself:
vcloud-container-debug.log shows internal communications and process happening in the cell:
vcloud-contain-info.log shows warnings and error messages. Doing a cat or tail on the file with | grep WARN will retrieve all the warning messages:
vmware-vcd-watch.log shows attempts by the watch dog service to restart the core vmware-vcd service
Status of vCloud Cell & Connection To vCenter:
Monitoring Organization Resource Consumption:
Monitoring Organization Virtual Datacenters Consumption:
Syslog and vCD
Before I get started I wanted to name check a rather good blogpost by fellow vExpert, Gabrie Van Zanten. He documents the process of using SysLog to monitor Firewall activity/changes on the Organizational and vApp Networks using a Splunk based Syslog Service. I didn't use Gabrie's post to write this - but it sounds like this "splunk" thing is increasingly popular, and might be of interest to folks:
For me the only problem I have with "splunk" is the name, it sounds stomach-churningly familiar to some other word which will remain unmentionable.
As for me - I'm using the Syslog service running the vCenter Server Appliance - in fact many of my other VMware Appliance that offer configuration to Syslog (such as the vCNS Manager aka vShield Manager) are configured for it. To tell you the truth I've never bothered trying to access these logs, so I thought I might give it try and see what gives.
Before you begin, you need enable vCloud Director for Syslogging. This is done as the SysAdmin under System, Administration, General:
If these settings change - then you can use the "Synchronize Syslog Settings" option which appears on the right-click of each Organizations vCNS Edge Gateway or vApp Network vCNS Edge Gateway. As far as I can see in vCloud Director 5.1 so long as you set the Syslog IP address before you create any Organization/vApp Networks then the option for syslogging should be there...
Note: Synchronize Syslog Server Settings on an Organization Network
Note: Synchronize syslog server settings on the properties of vApp. Mmm, not sure why capitalization applies on one menu, and not another! But lets not split hairs or get too pandantic shall we?
You should be able to confirm the correct syslog settings are applied by right-clicking the Organization Network or vApp Network and checking the Syslog Settings tab:
Throughout all my work with networking in previous post I've actually been disabling the Firewall both in the GoS, vApp Network and Organization Network. That's very unrealistic given the firewalls of the vCNS Edge Gateway appliances are always turn on. I'm going to turn on the vApp Network's Firewall and also enabled logging...
Once applied it immediately stops any inbound traffic to the vApp - because the default firewall rule on a vApp allow ALL traffic OUTBOUND, but NOTHING, INBOUND. In this case I was making the change on the Web vApp which works on the 192.168.15.x network - after clicking OK and Applying the configuration the inbound ping I was doing stopped responding - of course, the outbound ping I was doing kept on working.
If I wanted to allow inbound traffic to passed through the vApp Firewall, I could create rule that the opposite of this default rule - one that said the any source traffic coming in externally, was allow to access the internal destination on any protocol. This is the same as turning the firewall off I guess... The important thing to note from a Syslog perspective if the logging is enabled on per rule basis:
and we could easily modify this allow inbound pings from the Organization Network - by changing the pull downlist for Protocol from being ANY to ICMP. This would allow ping traffic to pass but other traffic like Microsoft Remote Desktop Protocol (TCP 3389) or Web Services (TCP 80) and so on would be blocked.
Then I went looking for the Syslog files on the vCenter Server. This was a little bit harder than first thought - but I did eventually locate them. I was attempting TCP 3389 connection to the 192.168.15.101. I expected to see a syslog entry for the vApp Network, but instead I found the entries in the syslog on the Organization Network. That I think makes sense because the traffic came from the External Network >>> Organization Network >>> vApp Network.
The syslog files for the vCenter Server Appliance are located on:
/var/log/remote/ and then there is a directory for the IP that represents my Edge Gateway on the Organization Network - ::ffff:192.168.3.10/ within this directory was a folder for the month 2012-12 - with log file for each day - the most recent being the day I was writing this blogpost: messages-2012-12-20
By using the cat command I was able search for any entry with the reference to the word "DROP", indicating the packet had been dropped by the firewall. Before attempting this - I had go at using Microsoft RDP to gain access to the VM, something that failed through a lack of access via the vApp Firewall.
cat messages-2012-12-20 | grep DROP
SRC=192.168.3.198 was the machine I was making the request to, and DST=192.168.15.101 was where I was trying to get to. The destination port attempted was 3389 the port number used for the connection. After adding a rule to allow inbound RDP like so...
The second attempt to access the VM via RDP was successful like so:
2012-12-20T19:02:51.000+00:00 firewall [759194f4-bb9e-4262-be20-5d247c821722]: ACCEPT_2IN= OUT=vNic_1 SRC=192.168.3.198 DST=192.168.15.101 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=25500 DF PROTO=TCP SPT=1520 DPT=3389 WINDOW=64512 RES=0x00 SYN URGP=0 MARK=0x1
I could tell which Firewall Rule was responsible for the access by the line "ACCEPT_2IN=" this indicated it was the 2nd Rule in the vApp Network firewall rule that was responsible for allowing the access.
The only REALLY odd thing about picking up SysLogs from vCenter is curiosity over the name of the SysLog directories - which are named after hosts. I found entries for things I wasn't expecting such as "SALESDESKTOP05". It turn out I had stale and out of date records in my DNS. I once had desktop called "SALESDESKTOP05" with an IP of 192.168.3.21. It turned out my vCNS Edge Gateway for the COIG Organization had the same IP as well. SysLog must have done some sort of lookup on the name.
Clearing out these bum records, enabling automatic scavenging and setting some names for the Edge Gateway interfaces cleaned this up over time... I would hope in a Production environment DNS would be handled more appropriately than in a lab environment where you tend to re-use and blow IP/Arecords frequently. I've since cleared out a lot my static records, and enabled scavenging of stale records on the DNS...