PLO Diaries

Every now and then I get PMs, posts, or messages from members who are wondering what we’re up to behind the scenes at DC, in terms of design, development, and general improvements to the site. I’ve promised to start blogging about this when I get opportunities, so here’s the first of what will hopefully be a semi-regular column.

First: a little about us. While we update DC very frequently (on average 5+ updates go out to the website and codebase per week), we’re a very small team at the moment. Behind the scenes we have one developer/designer and one pure developer, who both work on Rails. Aaron, our dev/designer hybrid, also works a lot in Javascript (we use Prototype) and in Flex/AS3 for the Flash Player. I’m currently learning Ruby+Rails and can help out with some minor tweaks here and there, and DeathDonkey contributes to some of the codebase as well (including fully maintaining the handconverter). But for the most part, the majority of the developments and improvements you see to the site are done by two people. Props to them.

So that’s a start. In the past few months we’ve rolled out features like Advanced Playlist Integration and a considerable improvement to DC’s dashboard which allows users to keep track of threads and videos they are interested in. But as Fred Wilson says, “Speed is the most important feature.” So in the past few weeks, we’ve set off on a number of tasks to help improve the performance of DC as we continue to grow. When we started out, we were running on one small but dedicated webserver with a few gigs of RAM and two processors. We’re still able to operate on a single server, but now we’re running on 6 cores and 20GB of RAM hosted in the RailsMachine Cloud. We primarily use two apps, Scout and New Relic to keep a close eye on server performance, so that we know things are slowing down before you get a chance to notice it.

As I’ve posted in the past, the vast majority of our files are hosted by Amazon AWS’s S3 filehosting service. Early in our site’s history we wrote a geographical locator that figured out where you were downloading from, and routed you to either our S3 servers in the US (on the East Coast) or in Europe (Ireland). That service has taken us a long way, but as we’ve moved more into streaming media (thanks to our Interacitve Flash Player ), we’ve outgrown the abilities for S3 to be able to stream files quickly to all of our members, and have started making the move over to Amazon’s CloudFront Services. This will automate the process of picking between one of 15 different geographical locations based on which should be fastest (and closest) to you, and will also allow that same flexibility for our streaming videos, which should make our flash player accessible to a lot more people. Holla to speed.

After a fair amount of work and testing, that project is nearing completion — I’ll make a post about it on the forums when it launches so I can keep an eye on how it’s working for everyone. Our next performance improvement projects are a lot more experimental and we’re in the early phases of working on them. Here’s a sneak peek at the first: the DC dashboard.

To be brief, the dashboard is one of the most important areas of DeucesCracked, but unfortunately it’s also one of the slowest — and the most complex. As users track more and more data (and as we add more types of data that people are interested in), we have to be constantly aware of the balance between speed and abilities of the application. Add too much at any one point in time, and we run the risk of it slowing down to a crawl. In the past week or two we’ve started looking into Redis for tracking different types of data that are used on the dashboard. As it stands right now, we track a little over five million video views and 18 million forum post views; while MySQL is great, we’re looking into Redis (and other nosql solutions) as potential ways to speed up the ability to track some of those larger datasets. As you can tell, things are still pretty damned fast on DC, and we intend to keep it that way.

Hope this was interesting to some of you out there — if it was, please post in the comments so I know to do it again in the future. I’m interested in hearing your feedback.