Tim Morgan, Senior Developer at Scribd.com, or better known as the “rails guru”, has been working with Ruby on Rails since version 0.8. He is also known as the “walking Rails API” at the Scribd office, although sometimes tells people to read the manual. Morgan has worked at companies such as Tickle (Division of Monster) doing Rails and Java, and Amazon.com working on C and Perl.

Tim discussed with me about working at Scribd, his upcoming presentation at the second LA Ruby Conference, and production outages. Tim was kind enough to give this  preview into his upcoming talk on his experience with a high-volume revenue generating, production Rails Web site, and will reveal more at the second LA Ruby Conference on February 19-20, 2010 in Burbank, CA.

Working at Scribd is a wake-to-sleep job, but on the weekends I’m a private pilot, and I have a Rails site at http://www.terminalprocedures.com for other pilots.  My most popular open-source project is Autumn, the Rails-like framework for writing IRC bots.

When I started working at Scribd I knew a lot about Rails but not a lot about scalability.  I had only the basic MySQL knowledge most non-expert engineers probably have.  Unfortunately, as a start-up without classes and training, it was a lot of “learn by doing,” or in my case, “learn by breaking.”  I would sometimes make what would seem like the most innocuous changes, only to have the website go down in a ball of flame.

My talk focuses on three of the most notorious and tricky of my mistakes – changes that, if you ran a Rails site of 3,000 daily visitors or so, you wouldn’t think twice before committing.  It’s only when you understand the peculiar quirks of both Rails and MySQL (or PostgreSQL or etc.) that you see why these otherwise simple changes can utterly destroy your site when you get millions of visitors per day.

I provide my viewers with each change, how I drilled down and figured out what was wrong, and what I learned for my next commits.

Q: Most people don’t present on what they did wrong, why did you choose to share what not to do rather than what to do?

Tim Morgan: How to do Rails “right” is so often covered that it’s almost common knowledge: Lots of people know The Rails Way, and they know the tricks of the trade from screencasts and blogs that we’ve all seen.  The problem is some of these techniques can cause serious issues if you don’t understand what’s going on under the hood.  It’s the case for me, and I think for many other engineers, that I almost never try to understand what’s going on underneath the hood until I make a big mistake, and I find myself knee-deep in rdebug, wading through Rails’s internals.  Showing what I did wrong will allow me to recreate the thought process that engineers go through when their site breaks, which is when we often do our most low-level learning.

Q: Were most of the outages you experienced at Scribd attributed to Rails and MySQL caveats specifically or more high-level topics such as scalable design?

Tim Morgan: The two are very related, I think.  If you choose to use MySQL, that’s a high-level design decision, and it will (at least, it should) affect how you design your schema, at a high level.  MySQL has advantages and disadvantages next to, say, PostgreSQL, that ought to guide how you set up your relations.  Likewise, choosing to use Rails is a high-level design decision.  Rails and MySQL are scalable: Scribd is living proof of that.  When people say, “Rails doesn’t scale!” it’s usually because they just don’t know HOW Rails scales.

That being said, I suppose it’s “fortunate” that Scribd’s outages have mostly been marked PEBKAC (Problem Exists Between Keyboard And Chair).  We’re lucky to have a good DBA who made smart decisions early in our growth, eliminating a lot of the growing pains other Rails sites experience.

Q: Are the design considerations different for building a high volume Rails Web site different that say a Web site built on Java technology, for example?

Tim Morgan: From an application standpoint, absolutely.  Rails hides a lot with its “magic” … named scopes and validations can obscure critically slow database queries, and even the JavaScript helpers can result in inefficient front-end code that increases load time.  The magic is fine (it makes your code very self-documenting), but it’s ripe for “cargo culting” where less experienced engineers just copy-paste code and tweak it a bit, which often results in disaster.  Part of designing a Rails site is understanding what the magic is doing.

I did some J2EE/Spring before I did Rails, and the pitfalls are definitely very different in that world.  I saw a lot of similar problems with schema design (though at the larger Java shops I’ve worked at you couldn’t just churn out your own migrations without layers of approval), but in the ORM and view layer the problems are different.  I often hand-wrote my SQL, eliminating the “magic” issue, but creating entirely new problems for code reuse and abstraction – a different kind of growing pain