Imagine an application that acts as a super proxy for your Twitter account. The server acts as the client, pulling updates from the API. Updates are relayed to your iPhone and desktop clients in near real-time1 with XMPP. No more multi-device API limitations getting in your way. Add in analytics, trends (specific to your follow list), link harvesting, groups, searches, filtering and more automatically sync’d to all your devices.
Why doesn’t this exist? This should be what Twitter itself does, but I can’t see them suddenly going in this direction. Build it and you can have my money.
[1] As in, as fast as it can be delivered once it is retrieved from the API. Not real-time to the initial update.
I’ve got this piece of code and I’d like to get rid of the boolean parameter. How would you do it? I have some ideas, but I’m looking for some alternate perspectives.
In Part 1 I showed how the server-side tweet retrieval and caching mechanism works for the .NET Twitter Stream on Managed Assembly. In this post, I’ll show you how the UI pulls the items and updates the list live.
Twitter streams tend to show the newest items at the top of the page which I don’t think is ideal for reading a live stream of tweets. I decided to build an interface that showed the newest tweets at the bottom of a list so that it would read more like a chat window. I wanted the list to automatically scroll to keep up with the new items, but if you scrolled up to start reading older items autoscoll would be disabled so you wouldn’t be constantly losing your place.
When a visitor hits the page a call is made to the /Refresh method described in Part 1. On the first request the last 100 cached tweets on the server are returned along with the ID of the most recent tweet, which is stored in a hidden field. On subsequent requests the value from the hidden field is included in the request to /Refresh so that only new tweets are returned and appended to the list. This minimizes the amount of traffic sent back and forth and the amount of work needed to append items to the list (just loop through the returned items and append the generated HTML).
There’s a couple inline styles but bare with me. There’s a certain amount of “get-it-done”-ness to the code (particularly the HTML generation later) but it’s not important, this is just an example that happens to be running great live :-)
There’s a lot of CSS involved, but the important style definitions to make the autoscrolling work are as follows:
#autoScrollOff { color: #f00; }
#autoScrollOn { color: #0f0; }
#statusContainer
{
height: 525px;
overflow: auto;
border: 1pxsolid#ccc;
margin: 5px0;
}
#statuses
{
width: 100%;
border: none;
border-collapse: collapse;
}
There are two functions that make it happen. The first calls for the JSON and schedules itself to be called again in 60 seconds:
The other function is called when the JSON is retrieved (again, this should use client-side templating or something better than just string concats, but it doesn’t so get over it):
html += result.Text + "</p><p class=\"meta\">" + result.RelativeTime + " from " + result.ClientLink;
if (result.IsReply) {
html += " <a href=\"http://twitter.com/" + result.InReplyToUser + "/statuses/" + result.InReplyToId + "\">in reply to " + result.InReplyToUser + "</a>"
}
html += "</p></td></tr>";
$("#statuses").append(html);
});
if (autoScroll) $("#statusContainer").scrollTo(99999);
}
We store the max ID returned to use for the next call. The tweet list is reversed to put new ones at the bottom (this could be handled server-side too), the HTML is built and appended to the table. At the end of the function we check to see if autoScroll is set and if so, scroll to a big number using the jQuery ScrollTo plugin.
To start things in motion, we call the getUpdates() function in document.ready() and attach the handlers to manage autoScroll state:
autoScroll is defined outside of document.ready() so that’s available in all the calls to getUpdates().
There’s a similar method for retrieving the list of users currently being followed for the stream. Since the time I originally wrote this code I’ve started reading JavaScript: The Good Parts by Douglas Crockford so I’m a little ashamed of the JavaScript, but it does work well and that’s all I need for now.
Back when I launched ManagedAssembly.com around MIX I put together a page to show real-time Twitter search results for ‘MIX09’. On the heels of the dvplrs.com launch this week and the lack of .NET representation, I decided to update my live Twitter page to show live updates of notable people in the .NET community. This is how I built it. You can see the finished product over at ManagedAssembly.com/Twitter.
Since I decided to use the standard Twitter API instead of the Search API, I had to build a server-side cache of updates so that I wasn’t hitting the API on every view of the page which would quickly exhaust the API rate limits. Using a database was overkill so I decided to use ASP.NET’s built-in cache mechanism to store new updates. When the client makes a request for the latest updates, the items will be served out of the cache. The cache is updated every so often with new items.
The workflow when the client makes a request for updates (via jQuery) is like so:
Grab the date of the last cache update from the cache. If the value doesn’t exist or it’s older than the cache expiration threshold, it’s time to get more statuses from Twitter.
When new updates are pulled from Twitter, store the highest ID in the cache as well so that for future requests we can get only the new tweets.
Pull the currently cached items and add the new items from the Twitter API call to it.
When updates are sent to the browser, the client-side code stores the latest ID. For future requests this ID is passed back to the server which returns items from the cached list that are greater than the locally-stored ID.
What this gives us is the ability to only call Twitter once a minute no matter how many times the site is hit. It also allows for each visitor to only get back tweets that are new to them (which is different for every visitor depending on when the first hit the page) whenever they make a request. Since the tweets are being pulled from memory on almost every request there’s no delay while the API call is being made. If the application restarts and we lose the cache it won’t really matter because we just grab all the updates it will send and repopulate the cache. It also means that the amount of data being sent to the client via JSON is the bare minimum needed. That makes updating the UI simple (just append the new ones without having to process the full list) and keeps the bandwidth requirements low.
That was a lot of words, so time for some code. We start by determining if we need to get new items from Twitter and if so, grabbing them. This uses the CacheManager from a previous post.
The first time this is hit lastFetch doesn’t exist in cache and the cache manager returns the default for DateTime (DateTime.MinValue) which I’m pretty sure is always more than a minute ago. We’ll update the fetch time in cache later. Once we’ve determined that the cache is empty or expired, we grab the last_id value from cache (which is 0 if it doesn’t exist). We then use TweetSharp to get the latest updates and store them in a list.
We also make sure that statuses isn’t null because if you’ve exceed the Twitter API rate limit, TweetSharp returns a null collection. Once we have the updates we have to process them since the text doesn’t include links for @mentions, #hashtags or links. When deciding how to approach this part I contacted Jon Galloway and he pointed me to the code in Witty which handles creating links, which I borrowed heavily from (thanks Jon!). MA also uses SubSonic 2.2 for the DAL and I used one of the methods provided in that library for determining if a word is a URL.
if (statuses != null) {
foreach (var status in statuses) {
string raw = status.Text;
string[] words = Regex.Split(raw, @"([ \(\)\{\}\[\]])");
StringBuilder output = newStringBuilder();
foreach (string word in words) {
if (word.StartsWith("#")) {
string hashtag = String.Empty;
Match foundHashtag = Regex.Match(word, @"#(\w+)(?<suffix>.*)");
OK now that we’ve got the latest updates from the API and added the links, it’s time to store them in the cache.
var cachedStatuses = cache.Get<List<TwitterStatus>>("cachedStatuses");
if (cachedStatuses == null) {
cachedStatuses = newList<TwitterStatus>();
}
if (statuses != null) {
cachedStatuses.AddRange(statuses);
}
if (cachedStatuses.Count > 0) {
maxId = cachedStatuses.Max(s => s.Id);
cache.Store("last_id", maxId);
}
cache.Store("cachedStatuses", cachedStatuses);
Last but not least we build a JSON return value for all the tweets in the cache with an ID higher than the one sent by the client (lid is a parameter passed from the client).
I figured since Visual Studio 2010 was just around the corner that it was about time to update my Visual Studio Built-in Code Snippets Cheat Sheet for 2008. You can find the updated cheat sheet with the others.
In Part 1, I demonstrated getting started with Twilio and ASP.NET MVC. In this post, I’ll go over the remaining controller action implementations.
So far we’ve got up to the point where we play the current greeting for a caller. Coaches and umpires use the hotline to report any issues the League Director needs to be made aware of. In Part 1 we handle generating the response needed to play the greeting and then beep to let the caller know they can leave a message. When a message is left by a caller, a POST request is made to our /RecordVoicemail action:
Log(CallGuid, "Could not send voicemail notification email: " + ex.Message);
}
returnnewEmptyResult();
}
This method downloads the message from Twilio’s servers, attaches it to an email and sends it off to the Director. He can listen to these messages on his BlackBerry and take care of any issues reported.
If there’s a problem we log the issue to a file. The other methods have logging as well, but I’ve excluded them from my examples for brevity.
Lastly, the League Director needs to be able to record a new greeting from his phone in case he’s out at the fields and the weather forces a cancelation. While the greeting is being played, the Director can enter the PIN followed by # to record a new greeting. Once the PIN has been entered, we send the required Twilio response to prompt for the recording:
response.Add(Verb("Say", "Record your greeting after the tone. Hang up to save the greeting or press a key to start over."));
response.Add(Verb("Record", "",
new { maxLength = 120,
action = ActionUrl.RecordGreeting,
method = "POST"
}));
}
doc.Add(response);
returnnewXmlResult(doc);
}
When the PIN has been entered, a POST request is sent to this method. We use parameter binding again to get the digits entered. These are checked against the settings file. If the PIN is invalid, we notify the caller and ask for the PIN again. This will loop until the right PIN is entered (or the caller hangs up).
Once the correct PIN is entered, we return some short instructions and set up the Record verb. Once the new greeting is recorded, a POST is sent to our RecordGreeting controller action:
response.Add(Verb("Say", "Record your greeting after the tone. Hang up to save the greeting or press a key to start over."));
response.Add(Verb("Record", "",
new { maxLength = 120,
action = ActionUrl.RecordGreeting,
method = "POST"
}));
doc.Add(response);
returnnewXmlResult(doc);
}
Settings.GreetingUrl = RecordingUrl;
returnnewEmptyResult();
}
Because it can sometimes take a few tries to record the greeting properly, pressing a number during recording will restart the recording. Once the Director is satisfied with the updated greeting, he hangs up and the new greeting URL is saved to the settings.
My first implementation of the recording restart was to redirect back to /Greeting and including the correct PIN as a query string parameter which was bound to the Digits method parameter. While this worked, I didn’t like passing the PIN in a request so I just rebuild the response XML and return it instead.
That’s all there is too it. Twilio offers a lot more options including a complete REST API for accessing recordings, provisioning phone numbers, etc. So far I’m very impressed with the service and didn’t run into any major hangups (ba-dum-dum).
In an effort to give myself more things to write about I’m going to start writing about code I’ve written recently.
I’ve been interested in writing something using Twilio since it became available late last year. Twilio is a service that provides an API for building telephony applications. I’ve had a lot of ideas over the years for integrating phone-related features into applications and Twilio makes it really easy.
I used to run a softball league and the past few years I was using GrandCentral (now Google Voice) for a weather hotline. On days when the weather wasn’t conducive to playing softball the players and coaches would call into the hotline to see if we were going to play or not. I no longer run the league and didn’t want to donate my precious Google Voice number to the league so I started looking for an alternative. Most of the services out there are too expensive if you don’t use them a lot and there’s only a couple weeks a year when the league needs the hotline.
If you’re not familiar with how Twilio works be sure to read up on it. In short, when your number gets an incoming call, the Twilio service makes a request to a specified URL, to which your app responds with a set of commands in XML.
The hotline has a few simple requirements:
When a call is received, play a greeting. This can either be text-to-speech (if there’s no recorded greeting) or the currently recorded greeting.
After the greeting is played, allow callers to leave a message (coaches use this to report issues to the league) which is emailed to the League Director.
Allow the League Director to record a new greeting by calling the number and entering a secret PIN.
I created a new MVC site, removed all the default cruft and created a simple route (/{action}) to allow me to use the following URLs:
/ (root) – Entry point for every incoming phone call.
/Greeting – Invoked when the PIN is entered and prompts caller to record new greeting
/RecordGreeting – Handles the completion of the recording. If a digit is pressed while recording a new greeting, it restarts recording. If a hang up is detected, it saves the URL of the greeting audio file to the settings file.
/RecordVoicemail – After a voicemail is left, this action method handles downloading it from the URL Twilio provides and emails the .mp3 recording to the League Director.
Because we’ll be returning a lot of XML, I created an XmlResult to take an XDocument and output it:
var gather = Verb("Gather", "", new { action = ActionUrl.Greeting,
method = "POST",
finishOnKey = "#" });
// say current greeting
if (string.IsNullOrEmpty(Settings.GreetingUrl)) {
gather.Add(Verb("Say", "Thank you for calling the league hot line. Please leave a message."));
}
else {
gather.Add(Verb("Play", Settings.GreetingUrl));
}
response.Add(gather);
response.Add(Verb("Record", "", new { maxLength = 120,
action = ActionUrl.RecordVoicemail,
method = "POST" }));
doc.Add(response);
returnnewXmlResult(doc);
}
This method builds the required Twilio response XML. The first thing we add is a Gather verb which listens for digits being pressed. When the # key is pressed, a POST request is sent off to /Greeting (generated using the ActionUrl helper class I wrote to make sure the complete URL is returned). To keep listening for input while the greeting is being played, we nest the Say or Play verb inside the Gather verb. Lastly a Record verb is added which lets the caller leave a message which is posted to /RecordVoicemail.
I’ve written a helper method that makes it easy to generate XElements in the proper form. The only thing it doesn’t really support very well is nesting, but it’s easy enough to get around that and you could easily add params parameter to accept an array of child elements.
foreach (var item in paramObject.ToDictionary()) {
element.Add(newXAttribute(item.Key, item.Value));
}
return element;
}
The last parameter lets you pass in an anonymous object to set attributes on the XElement, similar to ASP.NET MVC’s use of anonymous objects. ToDictionary() is an extension method which has been posted in numerous places.
Since this is getting long, I think I’ll stop here and do a Part 2 with the remaining method implementations.
If you don’t care about Twitter, see you next week. It seems like every Twitter blog post I come across is immediately barraged with anti-Twitter sentiment so if that’s what you’re planning on doing, save your keystrokes. I promise I won’t write about it often. That’s what Twitter is for.
Twitter is still relatively new and as such there isn’t a lot of precedent for good and bad behavior, especially for businesses. I have some recent experiences about the right way to use Twitter for your company, and the wrong way.
First the wrong way. If you’re a Mac enthusiast you’re probably familiar with MacHeist. They bundle a bunch of software together, sell it for a big discount and then give a portion of it to charity. The more people buy, the more software that’s included in the bundle. Neat system. For the recent MacHeist 3 there was an additional software package you could get if you sent out a tweet to your followers promoting MacHeist. I had probably a dozen of those come through my Twitter stream before I added a filter in TweetDeck to hide any more of them.
There’s a lot wrong with this situation. Those tweets were essentially paid advertisements that MacHeist was extorting out of people in exchange for a popular piece of software. This is tantamount to spam. Imagine Twitter didn’t exist and you got an email from a half dozen of your friends from an automated system with a standard message provided by MacHeist. There is no doubt you would consider this spam. Why is it different on Twitter?
This is the problem with how many companies are approaching Twitter. They see it as another broadcast medium when in actuality, it’s a distributed conversation platform. So if you’re a business you should use Twitter to engage in conversations with your current or potential customers.
So here’s the right way. A few months ago I complained about Telerik controls ruining my day. Shortly there after I received an email from one of the cofounders of Telerik asking me if I would care to elaborate and that he was sorry his product has ruined my day. I wrote back outlining my issues with the r.a.d. controls and after a little back and forth we were able to work out the issues we were having. The latest version is really not so bad and is markedly better than the version we had.
Ideally, the Telerik conversation would have taken place out in the open on Twitter, but despite that not being the case, there was a happy ending.
So if you’re a company looking to use Twitter to your advantage you have two options. You can become a slimy spammer and get the resulting negative PR, or you can directly converse with your customers and leave them with a favorable view of you (so much so I was compelled to blog about the experience). Which would you prefer?
I’ll be going to my first Twin Cities Code Camp this Saturday so if you’ll be there, keep an eye out for me. Hopefully I’ll learn a few things I can blog about.
