Wednesday, August 25, 2010

Channel API

We've been having server troubles since Lifehacker covered the application. A rush of over 5,000 users have brought the server costs skyrocketing. Throughout the dilemma, I've been slapping band-aids on the problem, raising server quotas and getting creative with memcache to try and stay afloat, waiting for the Channel API from Google.

Today, that API landed. Sort of.

Moishe Lettvin just emailed me, letting me know that our application had been selected (largely thanks to the recommendation from Michael Davidson) for extremely early access to the API. Unfortunately, Google wants to load-test the API, and wants to roll it out to smaller apps before larger ones.

Fortunately, we already have a separate server for this kind of thing-- the development server, used by our brave beta-testers. I've been given the go-ahead to expand that group up to 50 people to test the CPU usage and get a reliable idea of the effect this will have on our quotas. I'm expecting a dramatic decrease, but could be wrong.

So here's what's going to happen. We have a group of beta-testers right now, but there's still about 40 slots in the beta available for people to use. Send an email to paddy at android 2 cloud dot org, and if there's still a spot left, I'll invite you to the beta group, and you can help test the new server software, and will hopefully take some strain off the main server. As a perk, delivery from this test server will be almost instantaneous. No more 15 second delay because you sent the link at just the wrong time. The new API makes everything lightning fast. To quote Google, "the expected latency of delivering messages is on the order of 100ms". So a tenth of a second after you send a link, it opens on your browser.

Moishe has assured me that as the API gets its legs under it, and as Google is able to handle the load, the main server will get consideration for early access, as well. Which is really exciting; server woes may soon be coming to a close, and features that have been left out as I struggled to downsize the load on the server can be added.

Thursday, August 19, 2010

Servers and Money

As many of you know, we've had trouble since the start keeping servers running. A lot of trouble. The free App Engine quotas served us well when we had a hundred or two hundred users, but ever since Lifehacker covered the application, the free quotas simply haven't been close to enough. As of this blog post, we've passed the 5,000 users mark. That's a lot of users.

The Chrome extension works by asking the server every 15 seconds if there's a new link. If there is, it opens the link. This approach is called "polling" the server. When 5,000 people are polling the same server every 15 seconds, it eats through quite a bit of CPU. Right now, our server can (and often does) handle 110 hours of CPU time a day.

Unfortunately, that can cost up to $11 a day to keep the server running. I just got a bill for roughly $50 from Google, this last week's server expenses. And that's partially before I upgraded the server quotas to their current marks. We're staring down the barrel of $70 a week bills for this service.

I simply don't have the money to support that. I'm a college student who just returned to campus, and is buying books, working three jobs, and arguing with Financial Aid. As much as I would love to, I can't just eat a $70/week expense.

Many of you, kind, generous souls that you are, donated money to the project. I have been keeping careful track of your donations, and want to assure you that they are all being put towards keeping the server running. I will continue to pay for the server, at the very least until those funds run out on me. Maybe I'll keep it up for a few days after that point.

But my question to you is, what should I do after that? What should I do after the money runs out? Should I simply let the server go down every day whenever it surpasses the free quota? Should I gradually reduce the quota we are currently using, to try and extend the length of time we're operating with a raised quota? Should I just shut down the public server? That would definitely lose users. What should I do? This is as much your app as it is mine.

There is a light at the end of the tunnel. Google will be releasing a Channels API for App Engine, and I have a promise from a Google Engineer that I'll be one of the first to get access to it. That will reduce the load on the server considerably, as it changes the polling model to be more of a push model. But something needs to be done while waiting for that.

I'm sorry. I wish I could just pay the bill every week, but I can't afford to. I also apologise for my response time and lack of updates over the last two weeks. I am, as I said, just getting back on campus, and haven't had the time to devote to this project that I'd like. I promise an update is coming soon, one that should solve a lot of problems people are having with not understanding how to set up the app.

Monday, August 16, 2010

Downtime

It looks like we may run into some downtime in the next hour and a half or so. Everything should be fine after 3 a.m., Eastern Standard Time, on Tuesday, August 17. My apologies for this downtime-- we've burnt through over 80 hours of CPU time in under 24 hours!

Sorry about this. I'm working with Google to solve these outages, and will have something to show for it soon, I hope.

--Paddy

Friday, August 6, 2010

Authentication Issues

You know, it feels like 90% of the development and support on this application goes authentication. The OAuth just doesn't want to work for me, does it?

It looks like uninstalling and reinstalling the extension fixes most, if not all, of these issues. I've released version 1.0.1.1 of the extension, which will detect if you're having auth issues and will attempt to reauthenticate you. Meaning it will open a new browser window, and will ask you to log in and grant access again.

I'm sorry for all these issues. Really, I want to make them stop as much as you all want to see them stop. I appreciate everyone who is filing bug reports and is helping to track down the issue. You all are amazing.

In related news, I've decided to create a group of trusted Beta Testers for the application. If you're interested in testing the new features of the application before they roll out to all 2000+ people, and are ok with buggy releases (you must be, if you're still using the application... sorry!) please send an email to paddy at android 2 cloud dot org. I'll invite you to the beta tester Google Group, and you'll get instructions from there.

Thanks everyone for your support. You all rock.