by GL_Support, 11 hours agoGreetings,
We implemented a fix for the intermittent database connection issues that we have been seeing recently. We chose to implement this change immediately this morning instead of waiting. The changes took significantly longer than we had planned due to a stupid error made on the part of an engineer in our datacenter. No damage was done, but the screw up meant I had to put the site in maintenance mode instead of using our load balancer. This engineer *will* be flogged.
As for the intermittent connection issues, we are going to monitor the situation and keep our eye on these intermittent DB connections. We should know within a few days if the changes made today are a workable solution.
Thanks for your patience while me work to keep things running in tip top shape.
-Stephen
by GL_Support, 5 days agoWe will be releasing Version 7.2.3 at 6:30am tomorrow on 7/28/2010. There will be about 30 minutes of downtime for this release. More info on the specifics of this release are here:
http://www.guildlaunch.com/blog/?p=255
Enjoy!
Stephen
by GL_Support, 9 days agoUPDATE 3:52pm EST
We believe that code in the recent Template Editor customization was causing a TCP connection issue due to the number of extra DB connections created. Effecitvely with the new CSS file creation most page loads now represent one additional connection. Multiply this over the number of pages we serve and it appears to have crossed a threshold where an intermittent error would occur. This is an error that is difficult to predict, but now that we've seen it in action we can put a solution in place.
We've lengthened the cache for the editor customizations and it appears at this time to have alleviated the issue. We're now researching a more permanent solution to ensure that the extra caching doesn't cause issues where a user can't see their edits and to optimize the cache length.
-Stephen
UPDATE 2:57pm EST
We believe we found the source of these errors and we're verifying a temporary fix. Once we're sure the problem is identified we'll come up with a permanent fix.
-Stephen
Original Message
We are researching the database connection errors that are happening today. The errors are very intermittent and were working on isolating the cause.
This may be related to upgrades we made this morning, but we are not sure at this time. Our network engineers are researching the issue.
We will update this post as we have news.
-Stephen
by GL_Support, 10 days agoDuring the weekly downtime today we performed a partial upgrade of a portion of the server software for Guild Launch. We will be finishing those upgrades during a short downtime tomorrow morning. The downtime will being right around 5am Eastern and will last approximately 30 minutes.
These upgrades are to improve performance and stability and also to lay the groundwork for functionality we will be releasing next week.
-Stephen
