While I was at Yahoo, we purchased several amazing properties and hired some of the best product minds in the world. One by one, an organization run by Hollywood execs and then Wall Street execs broke the products and broke the spirits of the brilliant entrepreneurs that they aqui-hired by forcing them to justify their design decisions to a panel of executives who had no clue about product.

After years of stagnation, what should Yahoo do now? Everyone notes that the company’s vision is unclear and focus is necessary but where does it go? Yahoo has two extraordinary assets, a distribution engine and a monetization engine. It should use these assets to re-build the world’s greatest directory of online content.

Yahoo can monetize better than any individual website because (1) ad revenue goes up when you have a name brand and buy in bulk and (2) ad revenue goes way up when you can do smart targeting across many different properties. Yahoo also has a massive global distribution engine in the form of their homepage and user base.

Why does it point that awesome user-base, that incredible resource base at second-tier properties just because Yahoo owns them? Why not make the “Yahoo Travel” button go to Hipmunk? Why not make the “Yahoo News” go to Prismatic? There’s no need to buy all of these companies, just setup revenue sharing – the incentive structure works much better!

What if web developers saw Yahoo as an enormous distribution and monetization platform? More importantly, what if users knew that the Yahoo homepage would send them to best of breed in every category? Everyone would have a reason to use the Yahoo homepage again. What are the best sites for buying cars, playing online games, checking the weather? I don’t do any of these things enough to know, and today Google just sends me to the sites with the most advanced SEO.

Yahoo can learn from and emulate the success of Apple and Facebook’s app stores. Curate and promote high quality applications to give users a great experience and take a share of the revenue from the featured applications.

Facebook, Linkedin and Twitter all offer identity platforms but without an integrated advertising/monetization solution and without the ability to strongly promote high quality apps. Developers on the Facebook platform often spend their time bolting on unwanted viral features to their applications in order to acquire users. Yahoo could use its homepage to become the internet’s largest driver of user traffic and help promote the highest quality applications while making a ton of money in the process.

I’ve been swapping cute math tricks with my coworker Paul Pedersen, and recently he showed me this beautiful method for sketching polynomials:

As an example let’s pick a random polynomial:

y⁴ + x³ – y² + x – 1 = 0.

Plot on the x axis of a graph a point for every x power. For example for x³ plot the point (3,0). For x¹ plot the point (1,0).

Now label “+” the positive coefficient’s points and “-” the negative coefficient’s points. Do the same for the y axis.

Now connect the lines.

Now reflect across both axis. If you have an even exponent flip the sign.

Now draw a line between every positive and negative sign.

Now connect the lines. My freehand splines are a little sloppy, but you get the idea…

And there it is. Compare with an actual graph:

Let’s try with some simpler curves where we know the answer: take x² + y² = 1 or x²
+ y² – 1 = 0:

Connect the lines:

Circle!

Just for fun, try x² – y² – 1 = 0:

Connect the lines:

Wow. Why don’t they teach this as part of the high school math cannon? These kinds of simple inexplicable patterns remind me why I loved math in the first place.

Note – If you want to plot a function like y=x^3, transform it into 0=x^3-y and follow the above steps and it should work.

Rahul points out an interesting counterexample in x^2+y^2-4x+3=0, but I might argue that in some sense it does work. You end up with a diamond and the x axis has a + at (0,0), a – at (1,0) and a + at (2,0). No other edges should have lines through them. So you would draw a circle. I think it’s clear that this method can’t make any claims about x intercepts and y intercepts, just the overall shape. Same comment about x^4-3x^3+x^2-x+20.

Can anyone find a counterexample where the shape is completely “wrong” (for some definition of wrong that doesn’t include intercepts or stretching)?

I’m going to start by reposting a few of my favorite posts from the 2007-2008 era.

I checked the site and it gave me a 500 error — only 1 in 10 requests seemed to get me an actual page — so I logged into our app server and saw the disk was full. The log file had grown to 20 GB! I deleted it and asked my friend Zuzka to check and see if we’d been Slashdotted. She found a NY Daily News article that had just gone up talking about FaceStat, but we’d survived the traffic from a Wall Street Journal Buzzwatch post and it didn’t seem like that would be enough to make the log file blow up. I checked the file again and it was back up to 20GB.

I wondered if we were under some kind of denial of service attack, so I called up Brendan to see if he could check it out. He found that our box was getting thousands of hits per second. Searching online, he found a whole bunch of Yahoo Answers questions asking why the site was down.

I’ve been hit by Digg and Slashdot before, but this spike in traffic was like nothing I’d ever seen. Then Zuzka figured out what was going on:

That’s right, we were on the front page of Yahoo.com, the most trafficked site on the web!

I looked in my inbox and it was full of thousands of angry emails like this:

Subject: facestat site would be interesting….

….if it worked. unfortunately, site crashes like this in conjunction with a press release don’t do much for credibility. after all, if site design and construction are poor, why would anyone think that the underlying concept and software have any validity?

Subject: quite the blunder!

Wow, the day you have your site profiled on Yahoo is the same day your site is down. What a stupid blunder!

It turned out that the reason Chris got a phone call was someone had looked up his phone number and posted it in the newspaper article’s forum:

Want to contact owner of the site?!? Chris Van Pelt in Spencer, Iowa is the registered owner and his phone number is: (712)262-8863. The IP address for the site is: 67.207.137.234 and is in St. Louis MO. The ISP is Slicehost LLC out of St. Louis, MO. which explains the IP. His email address is vanpelt@fluent-design.com Ahhh… the internet a wonderful thing when you decide not to hide personal information. Check out www.ip-adress.com This is all public information no gathering of information has been gathered in any illeagal manner.

I think they were meaning to obnoxious, but the early warning was a huge help . I had been working like crazy and had resolved to take Sunday off; Brendan was about to go out with his friends and Chris was on vacation.

I called up Chris, who was an hour away from a computer, and then ran to Brendan’s house, where we whipped up an error page:

Sorry! Turns out Yahoo put us on their front page without any giving us any warning… (We appreciate the traffic but weren’t ready for a 1000x increase in load ) We’re working on getting the site back up in the next couple hours. Send an email to request@doloreslabs.com, and we’ll send you an invite when we’re back up, or check back in once we’re off the front page.

Instantly emails started pouring in to request@doloreslabs.com asking to be notified when they could use our site.

After working so hard to get users to come to your site, it’s amazingly frustrating to see hundreds of thousands of people suddenly locked out. Unbelievably, our webserver (nginx) couldn’t even reliably show that static page… Brendan discovered that we were exceeding the system’s open file limit — set at 100,000 — because connections were counting as open files.

There was an awesome article in Thaindian news that commented:

Facestat.com is a service where anyone can upload their picture and get it judged by the public.
…
Weather the website has been hacked or it is just some server issues seems to be unknown, we will post any updates as we try to gather more information.

We went to our hosting company Slicehost and started buying up more machines and bumping up the RAM on the database server. Before we were hit we had one app server and one database server, and no automated way of setting up machines. In the previous few weeks, we had been adding requested features as quickly as possible, not worrying at all about performance. While Brendan worked on setting up boxes I started ripping out every database intensive feature of our system and Chris added more caching… Around 1 AM we were back online and looking pretty stable. We thought about moving our database server to a bigger box, but the email system was really unstable: we were dropping invites that our users were sending, so that seemed like a bigger priority to fix. I was pretty sure that the load would be way down since we were off of Yahoo’s front page, so we eventually got to bed by 5 AM…

The next day I woke up and found we were being hit with a load similar to Sunday. I’m still not sure why so many people are still coming or how they’re finding the site, but because our site was actually functioning we were successfully serving up a much higher percentage of page views. Google analytics is telling me almost all the traffic comes from people without a referral URL.

The latency was still making the site almost unusable, at least from my perspective. We rely on the fact that people who upload a photo stick around to judge other ones, but with the high latency it seemed unlikely that they would do that. I was worried we’d be stuck with tens of thousands of photos uploaded and no way to get them judged. We moved the database to a new machine and added memcached. Brendan hacked together some amazing tools to monitor our boxes, which were a random hodge-podge of whatever size slices Slicehost could give us.

So now it’s Tuesday night and the site seems to be cranking along under 50x the load that used to work on one box. We have 6 app servers and a big database machine. I’m really impressed what awesome hackers Chris and Brendan are and what amazing tools are available these days. Slicehost has scaled up as fast as we’ve needed them to. Amazon’s S3 serves all the images, and while the latency isn’t great, we never could have dealt with the bandwidth issues on our own. Capistrano lets us deploy and rollback everywhere; git with github lets us all hack frantically on the same codebase then merge and deploy. God keeps all the servers running, and memcached has given us great caching with very little pain (mostly… ). Brendan would also give a shoutout to ITerm and its crazy multitab input mode, but he can write his own blog post about that… It’s one thing to code scalably and grow slowly under increasing load, but it’s been a blast to crazily rearchitect a live site like FaceStat in a day or two.

I figure at this point we’ve been on the number 1 (or 2) page on the internet, so there’s no bigger instant spike in traffic that could happen to us…

Some lessons I learned for the next time this happens:

(1) Monitor the site better. We had exception handling emailing us, but there were so many exceptions that I didn’t really look at them and I wasn’t online. It wouldn’t have made sense to scale our site to handle this kind of load in advance, but it’s unfortunate we had to rely on random people deciding to lookup Chris’s email address to call his home phone number to yell at him…

(2) Don’t be afraid to put up an error page. We had lots of excited users emailing us when we had a page up saying we were down and explaining why. We had lots angry users emailing us when the site was up but with intolerable lag or crashing intermittently. I think wishful thinking caused us to put up the site an hour or two before it was ready.

(3) A statically generated homepage is a very good thing and memcached is awesome.

My favorite comment from one of the judges, “It’s amazing how difficult it is to judge people just from looks, and yet we do it every day. “  I’m amazed by the agreement rates were seeing on things like liberal vs conservative.  We’ll end up with a fascinating data set for sociologists.