Blog Posts

On Monday, January 21, at approximately 9:31am Pacific, one of the two nodes in the cluster restarted. Normally, this is imperceptible to almost all site visitors; however, in this case the node did not rejoin the cluster correctly and the node, while appearing to be up, was not serving content fully. This put additional load on the other node and degraded overall site performance. It was not apparent until midnight on Monday what had happened, at which point we figured out the restart script for the down node had a bug in it that we had not encountered before. We manually restarted the second node just after 1am Pacific on January 22 (today, as I write this), at which point performance rebounded significantly but not fully. We spent the working day on January 22 investigating, and figured out that a system parameter was unexpectedly reset on Monday, which caused point levels to be calculated 4 times more frequently and added more load to the system. At approximately 4:30pm Pacific, we set the parameter back to its intended level, and performance is now back to last week's levels. We are fixing the script bug, and we are investigating why the parameter was inadvertently reset.

Happy New Year!

The VMware Communities application is up and running relatively smoothly, and community members are posting questions, answering questions, and reading threads, documents, and blogs. The overall situation looks good.

I am still working on getting the test and stage environment in place, which is required before we can roll out the 2-column design that removes some u/i clutter and reduces JavaScript. My IT team has the VMs configured (that part was easy of course), but like organizations everywhere, it is taking longer than expected to get processes in place to ensure continued stability. I will write another blog post by next week with specifics on when that will roll out.

Speaking of stability, it has worked out well to cluster the VMware Communities application. One of the nodes restarted on Sunday, January 6, but because of the cluster, users did not see downtime (only those connected to the node that restarted got reconnected to the other node, which would not affect people browsing the site), and no one posted that there was any interruption. The previous time a node restarted was on December 21, so that is more than 2 weeks between node restarts.

As for performance, the application itself is performing quite well. Most users are seeing acceptable page load times but would be happier if the site were faster; only a relatively small number of users are still posting about unacceptably slow page load times. Performance data indicates the application is not the culprit, which is why we are focusing on reducing and optimizing JavaScript. As noted above, the 2-column design is a first step in this direction, and once that is in place, we will investigate additional JavaScript tuning.

The next step is to get to the latest release of Jive Software's ClearspaceX product. This will help improve maintainability of the application and get a lot of fixes that are now in the base product. We are working with Jive to scope out the work required, since the version we are running now has some customizations and also has a number of backports of fixes from subsequent releases.

Also in plan are fixes for some u/i bugs, browser compatibility (especially for IE6), and on adding some context-sensitive links (e.g. adding links to the VI3 documentation into the VI3 communities).

And finally, we plan to create some administrative scripts to streamline certain backend processes.

Now you see the development roadmap for VMware Communities for the rest of the first quarter. I'll keep you updated as we make progress on the list. Meanwhile, please continue to use the VMware Communities Feedback to report issues and post feedback.

I'm looking forward to a productive 2008!