A post written for Social Agency’s corporate blog, Socialsama.

One of the interesting parts of developing social applications is that the platforms and technologies we interact with are periodically changing, and we must keep pace both with our platform and with our application marketing strategies.

Facebook recently completed a fundamental redesign of much of its site.  Transitions on such a big platform are always painful, and a portion of the Facebook user base was vocal in their opposition (and were covered widely in the media).  Nevertheless, the new Facebook is now fully implemented, and we are starting to see the application ecosystem adapt to the new restrictions and features.

The biggest change from the application point of view is the fact that most of the profile boxes that were the prime source of application virality have been pushed onto a rarely-visited tab of the profile. Previously, going to a friend’s profile displayed all the boxes from their various applications, and users tended to discover new applications by seeing cool things on their friends’ profiles.  This meant that the most successful applications were those that made the user say “I want that” when seeing the profile box on their friend’s profile.  Of course, this led to a competition over profile real estate from applications, and a top user complaint about the old Facebook was that the profile pages were crowded and difficult to use. The new Facebook has made the profile box irrelevant for most apps.

The new profile page is centered around a feed of user activities on Facebook and its applications.  This means that the way a user discovers a new application is by seeing an action by one of their friends and saying “I want to do that”.  The feed is the central interface for Facebook, and much of the social interaction a user has now on Facebook is to consume new feed entries created by their friends.

So, what does a current successful Facebook app look like?  It invites meaningful and frequent user interaction, and posts these interactions to the user’s feed.  Users can discuss feed entries in the feed itself, so a successful app’s feed entries invite conversations between users.  Additionally, the new Publisher interface lets applications integrate completely into the feed interface, letting users compose entries from their home page with an application-defined interface, lowering the barrier to creating a new entry.

In a future post, we’ll look at what makes a successful MySpace app and what an app needs to do in order to succeed in both worlds.

v.

To attempt to answer a question for which the speaker knows no answer by saying random phrases related to the question’s domain in the hopes that the phrases will spontaneously create meaning.

Palining is almost always obvious to an audience with knowledge of the question’s domain, but palining sometimes works when the audience is less knowledgeable than the speaker. Example: “I tried to palin my way through that essay exam, but the professor saw right through it.”

Palining in real life:

COURIC: Why isn’t it better, Governor Palin, to spend $700 billion helping middle-class families who are struggling with health care, housing, gas and groceries; allow them to spend more and put more money into the economy instead of helping these big financial institutions that played a role in creating this mess?

PALIN: That’s why I say I, like every American I’m speaking with, were ill about this position that we have been put in where it is the taxpayers looking to bail out. But ultimately, what the bailout does is help those who are concerned about the health-care reform that is needed to help shore up our economy, helping the—it’s got to be all about job creation, too, shoring up our economy and putting it back on the right track. So health-care reform and reducing taxes and reining in spending has got to accompany tax reductions and tax relief for Americans. And trade, we’ve got to see trade as opportunity, not as a competitive, scary thing. But one in five jobs being created in the trade sector today, we’ve got to look at that as more opportunity. All those things under the umbrella of job creation. This bailout is a part of that.

Your new database-backed website is starting to become popular, and load times at peak hours are starting to rise.  You add a slave server or two to spread out the reads, and put a lot of the reads in a cache.  That helps, and load times fall.   You can repeat this a few times, but eventually you hit a wall where you have too many writes to handle, no matter how much hardware you have.  Then you’re stuck.  You have to start partitioning your data into separate clusters, but your application was written with a lot of joins, and a lot of stuff no longer works if everything isn’t in the same database.  So, you’ve got a big development effort that has a high possibility of failure.

I suspect that this story would happen to the majority of startups out there if they had dramatic success.  It happened to Friendster, and it cost them global success as the social networking space which they basically invented and completely owned slipped through their fingers.

Relational databases suck for another reason.  If you want any kind of user-defined data, where the user needs to customize the fields gathered, you are faced with either dynamically altering tables (maintenance nightmare, not scalable), or entity-attribute-value tables (hard to query, not scalable).   Scalability is great, but this is the real flaw that’s been hitting me lately.  There are no good solutions to this problem in the relational world.  Google had all these problems years ago with their web indexing database, so they made BigTable, which currently holds their entire 800TB index of the web with metadata in a single table.  That technology now holds almost everything they do.

BigTable is Google-proprietary, unfortunately.  But as of the last year or so, there are reasonably-mature implementations of BigTable and the Google File System that it runs on.  Hadoop is the GFS equivalent, it takes care of managing an arbitrarily-large set of constantly-changing data.  HBase is the BigTable equivalent, providing a REST interface to a simple query language.  These are both very new technologies, but Yahoo is using Hadoop to manage its crawled web data, and HBase is running Powerset, one of the most database-intensive web ideas I can think of.

These are column-oriented databases.  You don’t define columns, you define column families, and a row can have zero to many of each family, each uniquely tagged.  This lets you do away with many-to-many joins, and have infinite capability for user-defined fields that are stored with the main data.  Empty columns are never even stored.

We’re planning to run Hadoop and HBase on a cluster of Amazon EC2 machines.  We’ll use S3 as the disk that the Hadoop cluster reads and writes from.  What this means is that since every system (web, DB, storage) scales with the number of machines in operation, all more traffic will ever mean to us is a bigger bill from Amazon.  And I bet that bill will be lower than the total cost of ownership of the equivalent traditional setup, because we can run fewer servers at night and save money.

So, the next bit of my life will be spent writing a solid database abstration layer between CakePHP and HBase, and an EC2 monitor and dynamic machine allocator.  I hope to open-source these so we can make this a realistic choice for more people out there.

Highlights:

• The creator of Slideshare showing off the really slick Ajax he did to absolutely maximize usability.  SWFUpload is a technology I will definitely be using in the future.
• Michael Lopp, a product manager at Apple whose opinions I really respect showing how the creative design process works there, which is something I’ve wondered about.
• A panel composed of old-school marketing dinosaurs completely collapsing under the weight of the Twitter-enabled audience in a mini version of the Zuckerberg debacle.  Beautiful.  I missed the Zuckerberg thing in person, but these kind of audience revolts were happening all over the place.  If most of your audience is actively talking about you behind your back, they’re not going to put up with anything.
• Architects from Twitter, Six Apart, Meebo and more discussing their architecture struggles, which was great.
• Several of the people behind OpenID discussing the various implications of it.  I am now confident in saying that I don’t know how people will authenticate themselves on the web in twenty years, but I know they’ll be doing it over the OpenID protocol.  If you’re a small company today that wants to maximize user registrations (and who doesn’t), you should be accepting OpenIDs with the Yahoo button.  You can make it more generic after the public is better-educated.
• The creators of every major Javascript library (jQuery, Protoype, Scriptaculous, Dojo) on one panel discussing highly technical Javascript techniques.  I vote for SXSW to have more panels like this.
• Developers from Digg, Flickr, and Wordpress discussing their insane scalability.  I’ve been doing a lot of scalability research lately, and now I’m fairly confident I can build systems that can be scaled in the future.
• I stood next to Jeff Bezos for a couple of minutes, and had conversations with other important people.

All in all, it was a great experience.   I hope I get to solve my own scalability problems soon.

So, what’s important from the very start? Most of these items are applicable to any large software project, but some of them are specific to large web sites run by PHP. In no particular order:

• Internationalization. If your project is successful at all, someone will eventually want to use it in a language other than English. This will probably happen sooner than you think, because even though you’ve never met them, there are a billion Chinese people with internet access. This means that every bit of English text you put into your application needs to have hooks built-in to translate it. Thankfully, the foundation to do this right is there, with GNU gettext and the locale setting in the environment, but you have to be aware of this from the beginning, even if you have no plans to internationalize. It costs almost nothing to put those hooks in now, just in case.
• UTF-8. If you don’t use UTF-8, you make it so that no one can post content to your site in any language other than one that uses Latin characters, and you’ll eventually be forced into a messy, annoying, never-quite-done transition from ISO-8859 to UTF-8. If you use UTF-8 from the beginning, you’ll never really think about the fact that user 375209 posts his blogs mostly in Korean but sometimes in Thai. The fact that pretty much everything out there defaults to 8859 is like a bad joke. Do you not understand what I’m talking about? Read this.
• HTML filtering (XSS prevention). If you are making a site that allows non-authenticated users to post text that other users will see (which is just about every dynamic web site ever), not doing complex, difficult, slow filtering on every last bit of textual user input means that a malicious user can redirect anyone who would have seen their text to a site of their choosing. And yes, in real life, people do this, usually with redirects to porn. It happened to one of my clients this week (not my fault). This is so difficult to get right that even huge sites like Myspace get this wrong regularly. Again, there are some great open source libraries out there, such as the HTML Purifier. And don’t think you can get away with filtering out all HTML; it’s the future, people want to post rich text. When the first user posts the first porn redirect on your front page, you need a solution you can implement in a few hours, and if you haven’t done filtering at all, that’s just not possible. Don’t use a BBCode-style solution; it’s not good for a variety of reasons.
• Model-view-controller. We don’t use a particular framework for this, but it’s the philosophy that’s important. In any request, there should be a controller that reads user input, a model that does data transformation/processing, and a view that controls the format of the output sent to the user. We use Smarty for the view. Don’t make your own templating system or use simple PHP files, use Smarty. With Smarty, you aren’t tempted to do complex stuff.
• Code standards. Decide on spaces or tabs (important for source control). Use PHPDoc religiously. Decide on camel case or underscores. Don’t use functions (well, sometimes), use objects. Write these decisions down.
• Scalability. Maybe you’ll be successful and you’ll get to the point where it all can’t run on a single rented box, and you’ll start to have to worry about performance. At that point, you need to start caching your models (this is one of the reasons you need MVC). We use memcache. One of the things you can do now is keep all methods that change data in the same place. Later you can put code there to kill the cache entry for this object so it can be regenerated with new data.
• Don’t worry about performance. The number of CPU cycles you take up processing your text is no longer important up to a pretty absurd point in today’s world of cheap fast parallel web servers. Your DB server is what’s important, since it’s pretty difficult to have more than one of them, and extrordinarily difficult to have more than one that you can write to. And if you notice someday that one particular page is really slow, well, then cache it.
• Database abstraction. Do not write SQL queries in your code. This is probably the most controversial item in this list, but I stand by it. Never write SQL queries. Generate them programmatically. We use DB_DataObject. Today, I’d probably use MDB_QueryTool, but they’re basically equivalent. It can be a pain, especially when you’re talking multi-join, but this is also where it’s most useful. A big query with a bunch of variables which could be filled with where clauses and joins (but maybe not, make sure and put an and at the beginning of your additional condition unless you’re the first condition and spaces at the beginning and end of all variables, just to be sure!) is impossible to maintain, and the business rules in it are easy to overlook. SQL is a 30-year-old language and sucks hard in some pretty basic ways; let a library deal with the pain for you.
• No stored procedures. No triggers. No views. No foreign keys. The world has moved on from the days of the all-powerful DBMS whose status at any moment can be relied upon by a court of law. Your database should just be tables that you can add and delete from. Simple. There’s nothing you can do in a stored procedure that you can’t do in your main code, but there are plenty of things you can’t do from a stored procedure.
• Security. Security is important. Write every piece of code with an eye toward who the user is and whether they can do this. This probably means roles, etc. It’s a pain in the ass. It’s important. Do periodic security audits.
• SQL injection. SQL injection prevention is important and hard to get right every time, so you need to do it systematically. Here, more than any other security concern, is where one bad line will totally screw you.
• Standardized HTML/CSS. This is our next big project, since we didn’t do this one right at all. I don’t know the right answer to this one yet, but I will soon. YUI?
• URL rewriting. As strange as this seems, people really do care about what the URLs of pages look like, even if they’re never going to use them. We use Net_URL_Mapper, but I’m not sure what the best answer is here either.
• No code generation. Also a controversial statement. My experience with code generation has been universally negative. My primary argument is that changes in the code that transforms your configuration into your implementation should have real-time effects, not effects that show up months later when you want to change the configuration but the regeneration you just did broke something else because configurations like this were phased out six months ago, except for this one… you get the idea. If the performance of this transformation is too slow for real-time, then just cache it forever in memcache, but don’t keep it around on the disk.
• All warnings on in development, all fatal errors logged and examined. If you do this, you will increase your software reliability significantly.
• No configuration in the database. There are many reasons for this, but the best one is that it’s pretty much impossible to usefully version control the contents of database tables. Configuration should go in files that are version controlled. We use JSON. Simple, human editable, machine editable.
• Open source. Use all open source software. Do you really trust your business to the whims of some other company? What if there’s some bug in their software that’s really screwing you up, but you can’t fix it? There’s plenty about web development you can’t control without voluntarily using closed software in your architecture, especially when there’s such good stuff out there.
• Database layer isolation. That’s not a good phrase for what I’m talking about, but I’m not sure what the industry word is for this. To put it simply, don’t have queries for a specific table all over your code, call one piece of code that queries that table. What if you want to add a hidden flag to that table? Then you’ve got to go add a where clause to all those queries, and you’ll miss one, and then the hidden thing will show up on the front page, and people will be pissed.

This is just what’s important to do right from the beginning. There’s a whole other set of processes that are just as important, but it’s the kind of thing you can do as you grow. That’s for next time.

So for a start, I’ve moved away from using Livejournal to read RSS feeds and wrote my own reader. It’ll work better on my phone anyway. SimplePie makes RSS reading laughably easy. Why are the best libraries for actually doing stuff always hard to find? I was looking for work and all I could find was Magpie, which is GPLed and therefore useless (does no one understand that you might as well put your library under the You Can’t Use This Unless You’re a Hippie license), and some other libraries that care what format an RSS feed is in. Why are building the future of the internet on a standard with at least three incompatable versions that do the same thing? And that store dates using some weird text format? It takes SimplePie 10K lines of code to sufficiently abstract all the nonsense.

Another hidden but great library: Net_URL_Mapper, a Pear library that neatly does the Ruby URL mapping thing but that has almost nothing written about it in English. It works perfectly, by the way. This page will probably show up on the first Google result page for it, and that’s pathetic. (Google searcher: look at the test cases, they are all the documentation you need.)