I never thought I’d see this: http://www.nytimes.com/2009/01/07/technology/business-computing/07program.html
I’m not sure I can articluate why but R makes me both happier and angrier than any other programming language. You have to appreciate a language that yells at you when you try to make a pie chart.
UPDATE: Amazon S3 has been extremely reliable since I wrote this post. At this point they are almost certainly more reliable than us hosting ourselves.
FaceStat would have been much harder to make without Amazon’s S3. We’ve had over 100k images uploaded in the last month or two, and those images are requested constantly. We’re not big enough to justify using a CDN, but handling the image bandwidth ourselves would have introduced problems that I’m really glad we never had to deal with.
But I guess we’re going to have to deal with hosting our own images now. Since 8:43 this morning our site has been rendered non-functional. That’s 3 hours so far. I assumed Amazon would be more reliable than us at serving content.
All I can do is look at the status messages:9:05 AM PDT We are currently experiencing elevated error rates with S3. We are investigating.9:26 AM PDT We’re investigating an issue affecting requests. We’ll continue to post updates here.9:48 AM PDT Just wanted to provide an update that we are currently pursuing several paths of corrective action.10:12 AM PDT We are continuing to pursue corrective action.10:32 AM PDT A quick update that we believe this is an issue with the communication between several Amazon S3 internal components. We do not have an ETA at this time but will continue to keep you updated.11:01 AM PDT We’re currently in the process of testing a potential solution.11:22 AM PDT Testing is still in progress. We’re working very hard to restore service to our customers.
I especially love this bullshit about “elevated error rates”. Requests for at least 99% of the images we have cause s3 to hang. If our site failed for 99% of requests, I would not call that “elevated error rates”, I would call that being completely fucking down.
So what can we learn from this? Since many commentors mocked us for not building our own homemade monitoring system for FaceStat when we got hit with the Yahoo traffic, I’m sure people will mock us for using S3 in our runtime. And in this case the criticism might be fair. Should we have known better than to rely on Amazon? How many sites are there out there like us? Do they fail over to their own servers? Seems like that defeats the point of using S3 or EC2 at all.
I guess I’m spending my Sunday building a “poor mans cdn“.
It made me really happy today to see FaceStat and Dolores Labs in Slate. It brought in less traffic than the wall street journal, or even ny daily news, but it’s not every day your site is featured in your favorite magazine :). And they even talked a little about the fun research we’ve been doing.Some of the better quotes:Slate: “It’s market research for the individual! Not some shallow exercise for the insecure!”news.com.au: “Facestat site perfects public humiliation.”CNet: “What happens when Hot or Not hooks up with Google Analytics” What does that mean?Thrillist: “You look at yourself in the mirror every morning and think, “what a handsome, intelligent, athletic, manly man I am!”, but how does the rest of the world really interpret your neckbeard? Find out the awesome/horrible truth, with FaceStat.”Buzzfeed: “Reinforcing Shallow Stereotypes. Now Online!”Miss Bimbo: “FaceStat is sad.”
I thought this page was pretty funny. Took me a while before I realized that msn doesn’t actually have a comparison page for LSD prices:
Sunday morning I was sitting in my kitchen reading the newspaper and thinking about brunch, when Chris called me, saying he had a bunch of voicemails on his home phone in Iowa claiming something about our web app FaceStat being down. FaceStat is a site we made to show off our Dolores Labs crowdsourcing technology, and has had a small loyal following since we made it live about a month ago.
I checked the site and it gave me a 500 error — only 1 in 10 requests seemed to get me an actual page — so I logged into our app server and saw the disk was full. The log file had grown to 20 GB! I deleted it and asked my friend Zuzka to check and see if we’d been Slashdotted. She found a NY Daily News article that had just gone up talking about FaceStat, but we’d survived the traffic from a Wall Street Journal Buzzwatch post and it didn’t seem like that would be enough to make the log file blow up. I checked the file again and it was back up to 20GB.
I wondered if we were under some kind of denial of service attack, so I called up Brendan to see if he could check it out. He found that our box was getting thousands of hits per second. Searching online, he found a whole bunch of Yahoo Answers questions asking why the site was down.
I’ve been hit by Digg and Slashdot before, but this spike in traffic was like nothing I’d ever seen. Then Zuzka figured out what was going on:
That’s right, we were on the front page of Yahoo.com, the most trafficked site on the web!
I looked in my inbox and it was full of thousands of angry emails like this:
Subject: facestat site would be interesting….
….if it worked. unfortunately, site crashes like this in conjunction with a press release don’t do much for credibility. after all, if site design and construction are poor, why would anyone think that the underlying concept and software have any validity?
Subject: quite the blunder!
Wow, the day you have your site profiled on Yahoo is the same day your site is down. What a stupid blunder!
It turned out that the reason Chris got a phone call was someone had looked up his phone number and posted it in the newspaper article’s forum:
Want to contact owner of the site?!? Chris Van Pelt in Spencer, Iowa is the registered owner and his phone number is: (712)262-8863. The IP address for the site is: 67.207.137.234 and is in St. Louis MO. The ISP is Slicehost LLC out of St. Louis, MO. which explains the IP. His email address is vanpelt@fluent-design.com Ahhh… the internet a wonderful thing when you decide not to hide personal information. Check out www.ip-adress.com This is all public information no gathering of information has been gathered in any illeagal manner.
I think they were meaning to obnoxious, but the early warning was a huge help :). I had been working like crazy and had resolved to take Sunday off; Brendan was about to go out with his friends and Chris was on vacation.
I called up Chris, who was an hour away from a computer, and then ran to Brendan’s house, where we whipped up an error page:
Sorry! Turns out Yahoo put us on their front page without any giving us any warning… (We appreciate the traffic but weren’t ready for a 1000x increase in load :)) We’re working on getting the site back up in the next couple hours. Send an email to request@doloreslabs.com, and we’ll send you an invite when we’re back up, or check back in once we’re off the front page.
Instantly emails started pouring in to request@doloreslabs.com asking to be notified when they could use our site.
After working so hard to get users to come to your site, it’s amazingly frustrating to see hundreds of thousands of people suddenly locked out. Unbelievably, our webserver (nginx) couldn’t even reliably show that static page… Brendan discovered that we were exceeding the system’s open file limit — set at 100,000 — because connections were counting as open files.
There was an awesome article in Thaindian news that commented:
Facestat.com is a service where anyone can upload their picture and get it judged by the public.
…
Weather the website has been hacked or it is just some server issues seems to be unknown, we will post any updates as we try to gather more information.
We went to our hosting company Slicehost and started buying up more machines and bumping up the RAM on the database server. Before we were hit we had one app server and one database server, and no automated way of setting up machines. In the previous few weeks, we had been adding requested features as quickly as possible, not worrying at all about performance. While Brendan worked on setting up boxes I started ripping out every database intensive feature of our system and Chris added more caching… Around 1 AM we were back online and looking pretty stable. We thought about moving our database server to a bigger box, but the email system was really unstable: we were dropping invites that our users were sending, so that seemed like a bigger priority to fix. I was pretty sure that the load would be way down since we were off of Yahoo’s front page, so we eventually got to bed by 5 AM…
The next day I woke up and found we were being hit with a load similar to Sunday. I’m still not sure why so many people are still coming or how they’re finding the site, but because our site was actually functioning we were successfully serving up a much higher percentage of page views. Google analytics is telling me almost all the traffic comes from people without a referral URL.
The latency was still making the site almost unusable, at least from my perspective. We rely on the fact that people who upload a photo stick around to judge other ones, but with the high latency it seemed unlikely that they would do that. I was worried we’d be stuck with tens of thousands of photos uploaded and no way to get them judged. We moved the database to a new machine and added memcached. Brendan hacked together some amazing tools to monitor our boxes, which were a random hodge-podge of whatever size slices Slicehost could give us.
So now it’s Tuesday night and the site seems to be cranking along under 50x the load that used to work on one box. We have 6 app servers and a big database machine. I’m really impressed what awesome hackers Chris and Brendan are and what amazing tools are available these days. Slicehost has scaled up as fast as we’ve needed them to. Amazon’s S3 serves all the images, and while the latency isn’t great, we never could have dealt with the bandwidth issues on our own. Capistrano lets us deploy and rollback everywhere; git with github lets us all hack frantically on the same codebase then merge and deploy. God keeps all the servers running, and memcached has given us great caching with very little pain (mostly… 
). Brendan would also give a shoutout to ITerm and its crazy multitab input mode, but he can write his own blog post about that… It’s one thing to code scalably and grow slowly under increasing load, but it’s been a blast to crazily rearchitect a live site like FaceStat in a day or two.
I figure at this point we’ve been on the number 1 (or 2) page on the internet, so there’s no bigger instant spike in traffic that could happen to us…
Some lessons I learned for the next time this happens:
(1) Monitor the site better. We had exception handling emailing us, but there were so many exceptions that I didn’t really look at them and I wasn’t online. It wouldn’t have made sense to scale our site to handle this kind of load in advance, but it’s unfortunate we had to rely on random people deciding to lookup Chris’s email address to call his home phone number to yell at him…
(2) Don’t be afraid to put up an error page. We had lots of excited users emailing us when we had a page up saying we were down and explaining why. We had lots angry users emailing us when the site was up but with intolerable lag or crashing intermittently. I think wishful thinking caused us to put up the site an hour or two before it was ready.
(3) A statically generated homepage is a very good thing and memcached is awesome.
There’s a lot of interesting projects to build WordNet like relationships among words with games. Human Brain Cloud is different in that it shows a random word and just asks the user to pick the first word that comes to mind. Sometimes the relationships are rhymes or other surface form relationships, and sometimes there’s deep semantics involved. Either way the data set is fascinating to play with.
We recently posted about a product we made at Dolores Labs called FaceStat. You can upload your photo (or a friends photo) and Turkers will judge it on a number of different dimensions. It started as part of an experiment we blogged about here. It’s kind of like hotornot, but we’re able to ask more interesting questions. I’ve been thinking we might make a version with company logos.
My favorite comment from one of the judges, “It’s amazing how difficult it is to judge people just from looks, and yet we do it every day. “ I’m amazed by the agreement rates were seeing on things like liberal vs conservative. We’ll end up with a fascinating data set for sociologists.
Look at this result for the query “amazon” in firefox. The stylesheets look messed up and there’s a serious deduping issue.
There was a huge fire across from my house. Within an hour there were hundereds of pictures on flickr including this one that shows the fire starting: http://www.flickr.com/photos/ymbiont/2341396397/.
I’ve been working on a tool that makes interfacing with Mechanical Turk easier. We were talking about how to market it, thinking about buying Google adsense and so on, and finally we decided why not just take that money we would have spent and use it to run fun data collection experiments through our system. I think this human color wheel that my friend Brendan made is really cool.
