Blog Posts

We've fixed the last open issues since Friday's system upgrade:

1. Inbound emails are now getting posted. So if you receive an email for a community or thread, you can now once again reply to that email, and your reply will be posted.

2. New account registration (the "register" link if you're not logged in) is now working. So newbies can now create accounts and start to participate.

Also, we've done some additional tuning on the system, and we have been seeing overall better performance. There is still more we need to do on site performance, but I believe we have made improvements in the last week. Tell us know what you're seeing: Has Communities Site Performance Improved?.

Performance appears to be back at standard levels. We are continuing to monitor the system closely and assess tuning opportunities, as we believe faster performance should be achievable shortly.

Outbound email notifications (e.g. when you click 'Receive email notifications' in the Actions box) are now working again. However, if you reply to these notifications, the replies are currently not getting posted. I expect this to be fixed by Monday at the latest.

Private messages may be disabled for a couple of days. Sorry if this causes any inconvenience.

Tonight we added a third node to the cluster running the VMware Communities application. Performance is good as I write this, but we are investigating an error we are seeing on one of the three nodes. I will update before Monday with the status on resolving that error as well as the overall performance we see this weekend. Thank you for your patience with the downtime required for this change.

We continue to work to improve VMware Communities performance and ensure consistently acceptable page load times. We are tuning the system in the following key areas:

1. Load Balancer Configuration: Today, we updated the F5 load balancer configuration to direct requests directly to the two clustered Tomcat instances running the communities application, bypassing Apache. This will remove the cause of the Gateway Timeout messages many of you have reported. If one of the nodes in the application cluster is down, the load balancer will now redirect all traffic to the functioning node.

2. Redirecting Google traffic: The performance slowdowns of the last several weeks correlate directly to increased load caused by automated services like Google or RSS readers indexing / accessing the VMware Communities site. We had been aware of this issue since December when we updated the robots.txt file on the servers to disallow crawling by Google and reconfigured Apache to redirect Google crawls to a mirror of the site. This fix appeared to work; however, in the last couple of weeks, we have seen that when the system is under load, Google traffic affects the application anyway. The load balancer configuration changes described in #1 above should help resolve this: instead of having Google redirected from Apache running on the same server as the application, it is now redirected from the load balancer. This should remove the cause of the recent performance slowdowns while ensuring that Google continues to index the VMware Communities content. As I write this, we are still in the process of investigating whether there are other changes required to isolate Google and other automated service traffic, so I will update later then week when I have confirmed this fix.

3. Adding a third node: This will add 50% more processing capacity and should allow the application to handle traffic peaks better. We are taking advantage of this change to review all system configuration settings across the cluster. We will implement the third node in the next two weeks.

In addition, we have made two application setting changes to temporarily increase performance:

a. Query caching: Many of you have noticed the "Your message was posted successfully, but there will be a short delay before it is viewable in the thread" message when you post. We turned on query caching, which reduces system load about 20% by not requiring the application to rebuild the thread when a new message is posted. The query cache was originally set to 10 seconds, but we reduced it to 5 seconds, which should reduce how often community participants see the message. Our current thinking is to remove query caching when we stabilize performance.

b. Status Level Calculator: The status level calculator refresh rate has been set to 12 hours, which reduces system load. We will reset the status level calculation refresh rate to a shorter interval when we determine that doing so won’t negatively affect system performance.

I wanted to give an update on VMware Communities performance. Overall, we have been seeing some intermittent slowness again this week, and we are working on a couple of areas to address it:

1. Networking errors: We have identified and are in the process of debugging some networking issues between the two nodes in the cluster.

2. Private network: We are currently testing and plan to roll out a private network between the two nodes, which will increase the transfer rate of cluster information between them.

3. Automatic restarts: From early December until mid-January, the communities application ran with consistent performance, but then started slowing down. This problem was resolved when we rebooted at the operating system level, which makes us think there is a resource leak. We are therefore implementing an automated restart at the operating system level on a periodic basis.

4. Resourcing: We are putting additional Java experts on these problems to get them fixed quickly. Performance and system stability is the top priority.

We've had system monitoring in place since early December, which includes automatic restarts of the application when performance thresholds are reached. And the root cause analysis that we do at every application restart consistently is pointing us at the issues listed above.