Back when I launched ManagedAssembly.com around MIX I put together a page to show real-time Twitter search results for ‘MIX09’. On the heels of the dvplrs.com launch this week and the lack of .NET representation, I decided to update my live Twitter page to show live updates of notable people in the .NET community. This is how I built it. You can see the finished product over at ManagedAssembly.com/Twitter.
Since I decided to use the standard Twitter API instead of the Search API, I had to build a server-side cache of updates so that I wasn’t hitting the API on every view of the page which would quickly exhaust the API rate limits. Using a database was overkill so I decided to use ASP.NET’s built-in cache mechanism to store new updates. When the client makes a request for the latest updates, the items will be served out of the cache. The cache is updated every so often with new items.
The workflow when the client makes a request for updates (via jQuery) is like so:
- Grab the date of the last cache update from the cache. If the value doesn’t exist or it’s older than the cache expiration threshold, it’s time to get more statuses from Twitter.
- When new updates are pulled from Twitter, store the highest ID in the cache as well so that for future requests we can get only the new tweets.
- Pull the currently cached items and add the new items from the Twitter API call to it.
- When updates are sent to the browser, the client-side code stores the latest ID. For future requests this ID is passed back to the server which returns items from the cached list that are greater than the locally-stored ID.
What this gives us is the ability to only call Twitter once a minute no matter how many times the site is hit. It also allows for each visitor to only get back tweets that are new to them (which is different for every visitor depending on when the first hit the page) whenever they make a request. Since the tweets are being pulled from memory on almost every request there’s no delay while the API call is being made. If the application restarts and we lose the cache it won’t really matter because we just grab all the updates it will send and repopulate the cache. It also means that the amount of data being sent to the client via JSON is the bare minimum needed. That makes updating the UI simple (just append the new ones without having to process the full list) and keeps the bandwidth requirements low.
That was a lot of words, so time for some code. We start by determining if we need to get new items from Twitter and if so, grabbing them. This uses the CacheManager from a previous post.
CacheManager cache = new CacheManager(new ShortTermProvider());
IEnumerable<TwitterStatus> statuses = new List<TwitterStatus>();
long maxId = 0;
DateTime lastFetch = cache.Get<DateTime>("last_fetch");
if (lastFetch < DateTime.Now.AddMinutes(-1)) {
long lastCacheId = cache.Get<long>("last_id");
if (lastCacheId == 0)
lastCacheId = 1; // TweetSharp/twitter returns null if you use Since(0)
var request = FluentTwitter.CreateRequest().AuthenticateAs("UserGoesHere", "passwordgoeshere");
request.Configuration.UseGzipCompression();
request.Statuses().OnFriendsTimeline().Since(lastCacheId).AsJson();
statuses = request.Request().AsStatuses();
}
The first time this is hit lastFetch doesn’t exist in cache and the cache manager returns the default for DateTime (DateTime.MinValue) which I’m pretty sure is always more than a minute ago. We’ll update the fetch time in cache later. Once we’ve determined that the cache is empty or expired, we grab the last_id value from cache (which is 0 if it doesn’t exist). We then use TweetSharp to get the latest updates and store them in a list.
We also make sure that statuses isn’t null because if you’ve exceed the Twitter API rate limit, TweetSharp returns a null collection. Once we have the updates we have to process them since the text doesn’t include links for @mentions, #hashtags or links. When deciding how to approach this part I contacted Jon Galloway and he pointed me to the code in Witty which handles creating links, which I borrowed heavily from (thanks Jon!). MA also uses SubSonic 2.2 for the DAL and I used one of the methods provided in that library for determining if a word is a URL.
if (statuses != null) {
foreach (var status in statuses) {
string raw = status.Text;
string[] words = Regex.Split(raw, @"([ \(\)\{\}\[\]])");
StringBuilder output = new StringBuilder();
foreach (string word in words) {
if (word.StartsWith("#")) {
string hashtag = String.Empty;
Match foundHashtag = Regex.Match(word, @"#(\w+)(?<suffix>.*)");
if (foundHashtag.Success) {
hashtag = foundHashtag.Groups[1].Captures[0].Value;
output.Append(string.Format(@"#<a href=""http://search.twitter.com/search?q=%23{0}"" target=""_blank"">{0}</a>", hashtag));
}
}
else if (word.StartsWith("@")) {
string userName = String.Empty;
Match foundUserName = Regex.Match(word, @"@(\w+)(?<suffix>.*)");
if (foundUserName.Success) {
userName = foundUserName.Groups[1].Captures[0].Value;
output.Append(string.Format(@"@<a href=""http://twitter.com/{0}"" target=""_blank"">{0}</a>", userName));
}
}
else if (SubSonic.Sugar.Validation.IsURL(word)) {
output.Append(string.Format(@"<a href=""{0}"" target=""_blank"">{0}</a>", word));
}
else {
output.Append(word);
}
}
status.Text = output.ToString();
}
cache.Store("last_fetch", DateTime.Now);
}
OK now that we’ve got the latest updates from the API and added the links, it’s time to store them in the cache.
var cachedStatuses = cache.Get<List<TwitterStatus>>("cachedStatuses");
if (cachedStatuses == null) {
cachedStatuses = new List<TwitterStatus>();
}
if (statuses != null) {
cachedStatuses.AddRange(statuses);
}
if (cachedStatuses.Count > 0) {
maxId = cachedStatuses.Max(s => s.Id);
cache.Store("last_id", maxId);
}
cache.Store("cachedStatuses", cachedStatuses);
Last but not least we build a JSON return value for all the tweets in the cache with an ID higher than the one sent by the client (lid is a parameter passed from the client).
long lastClientId = lid ?? 1;
var data = from s in cachedStatuses
where s.Id > lastClientId
orderby s.Id descending
select new {
Id = s.Id,
ProfileImageUrl = s.User.ProfileImageUrl,
Text = s.Text,
Source = s.Source,
UserName = s.User.ScreenName,
RelativeTime = s.CreatedDate.ToRelativeTime(false),
ClientLink = s.Source,
IsReply = s.InReplyToStatusId != 0,
InReplyToId = s.InReplyToStatusId,
InReplyToUser = s.InReplyToScreenName
};
return Json(new { results = data.Take(100), max_id = maxId });
In Part 2 I’ll cover building the UI.
In Part 1, I demonstrated getting started with Twilio and ASP.NET MVC. In this post, I’ll go over the remaining controller action implementations.
So far we’ve got up to the point where we play the current greeting for a caller. Coaches and umpires use the hotline to report any issues the League Director needs to be made aware of. In Part 1 we handle generating the response needed to play the greeting and then beep to let the caller know they can leave a message. When a message is left by a caller, a POST request is made to our /RecordVoicemail action:
public ActionResult RecordVoicemail(string CallGuid, string RecordingUrl, string Caller) {
try {
var msg = new MailMessage();
msg.To.Add(Settings.VoicemailEmailToAddress);
msg.From = new MailAddress(Settings.VoicemailEmailFromAddress);
msg.Subject = "New voicemail received from " + Caller;
var client = new WebClient();
string filename = string.Format("{0}-{1}.wav", Caller, DateTime.Now.ToString("MMddyyyhhmmss"));
msg.Attachments.Add(new Attachment(client.OpenRead(RecordingUrl), filename));
msg.Body = "Received " + DateTime.Now;
var smtp = new SmtpClient();
smtp.Send(msg);
}
catch (SmtpException ex) {
Log(CallGuid, "Could not send voicemail notification email: " + ex.Message);
}
return new EmptyResult();
}
This method downloads the message from Twilio’s servers, attaches it to an email and sends it off to the Director. He can listen to these messages on his BlackBerry and take care of any issues reported.
If there’s a problem we log the issue to a file. The other methods have logging as well, but I’ve excluded them from my examples for brevity.
Lastly, the League Director needs to be able to record a new greeting from his phone in case he’s out at the fields and the weather forces a cancelation. While the greeting is being played, the Director can enter the PIN followed by # to record a new greeting. Once the PIN has been entered, we send the required Twilio response to prompt for the recording:
public ActionResult Greeting(string CallGuid, string Digits) {
var doc = new XDocument();
var response = new XElement("Response");
if (Digits != Settings.PIN) {
response.Add(Verb("Say", "Invalid pin number. Please try again."));
response.Add(Verb("Gather", "",
new { action = ActionUrl.Greeting,
method = "POST",
finishOnKey = "#"
}));
}
else {
response.Add(Verb("Say", "Record your greeting after the tone. Hang up to save the greeting or press a key to start over."));
response.Add(Verb("Record", "",
new { maxLength = 120,
action = ActionUrl.RecordGreeting,
method = "POST"
}));
}
doc.Add(response);
return new XmlResult(doc);
}
When the PIN has been entered, a POST request is sent to this method. We use parameter binding again to get the digits entered. These are checked against the settings file. If the PIN is invalid, we notify the caller and ask for the PIN again. This will loop until the right PIN is entered (or the caller hangs up).
Once the correct PIN is entered, we return some short instructions and set up the Record verb. Once the new greeting is recorded, a POST is sent to our RecordGreeting controller action:
public ActionResult RecordGreeting(string CallGuid, string RecordingUrl, string Digits) {
if (Digits != "hangup") {
var doc = new XDocument();
var response = new XElement("Response");
response.Add(Verb("Say", "Record your greeting after the tone. Hang up to save the greeting or press a key to start over."));
response.Add(Verb("Record", "",
new { maxLength = 120,
action = ActionUrl.RecordGreeting,
method = "POST"
}));
doc.Add(response);
return new XmlResult(doc);
}
Settings.GreetingUrl = RecordingUrl;
return new EmptyResult();
}
Because it can sometimes take a few tries to record the greeting properly, pressing a number during recording will restart the recording. Once the Director is satisfied with the updated greeting, he hangs up and the new greeting URL is saved to the settings.
My first implementation of the recording restart was to redirect back to /Greeting and including the correct PIN as a query string parameter which was bound to the Digits method parameter. While this worked, I didn’t like passing the PIN in a request so I just rebuild the response XML and return it instead.
That’s all there is too it. Twilio offers a lot more options including a complete REST API for accessing recordings, provisioning phone numbers, etc. So far I’m very impressed with the service and didn’t run into any major hangups (ba-dum-dum).
Download the Sample MVC Project
In an effort to give myself more things to write about I’m going to start writing about code I’ve written recently.
I’ve been interested in writing something using Twilio since it became available late last year. Twilio is a service that provides an API for building telephony applications. I’ve had a lot of ideas over the years for integrating phone-related features into applications and Twilio makes it really easy.
I used to run a softball league and the past few years I was using GrandCentral (now Google Voice) for a weather hotline. On days when the weather wasn’t conducive to playing softball the players and coaches would call into the hotline to see if we were going to play or not. I no longer run the league and didn’t want to donate my precious Google Voice number to the league so I started looking for an alternative. Most of the services out there are too expensive if you don’t use them a lot and there’s only a couple weeks a year when the league needs the hotline.
If you’re not familiar with how Twilio works be sure to read up on it. In short, when your number gets an incoming call, the Twilio service makes a request to a specified URL, to which your app responds with a set of commands in XML.
The hotline has a few simple requirements:
- When a call is received, play a greeting. This can either be text-to-speech (if there’s no recorded greeting) or the currently recorded greeting.
- After the greeting is played, allow callers to leave a message (coaches use this to report issues to the league) which is emailed to the League Director.
- Allow the League Director to record a new greeting by calling the number and entering a secret PIN.
I created a new MVC site, removed all the default cruft and created a simple route (/{action}) to allow me to use the following URLs:
- / (root) – Entry point for every incoming phone call.
- /Greeting – Invoked when the PIN is entered and prompts caller to record new greeting
- /RecordGreeting – Handles the completion of the recording. If a digit is pressed while recording a new greeting, it restarts recording. If a hang up is detected, it saves the URL of the greeting audio file to the settings file.
- /RecordVoicemail – After a voicemail is left, this action method handles downloading it from the URL Twilio provides and emails the .mp3 recording to the League Director.
Because we’ll be returning a lot of XML, I created an XmlResult to take an XDocument and output it:
public class XmlResult : ActionResult
{
private XDocument _doc;
public XmlResult(XDocument doc) {
_doc = doc;
}
public override void ExecuteResult(ControllerContext context) {
context.HttpContext.Response.ContentType = "text/xml";
_doc.Save(context.HttpContext.Response.Output);
}
}
I also have a Settings class for managing the application settings:
public class Settings
{
private static XDocument doc =
XDocument.Load(HttpContext.Current.Server.MapPath("~/App_Data/settings.xml"));
private static string Get(string key) {
return doc.Descendants(key).FirstOrDefault().Value;
}
private static void Set(string key, string value) {
doc.Root.SetElementValue(key, value);
doc.Save(HttpContext.Current.Server.MapPath("~/App_Data/settings.xml"));
}
public static string GreetingUrl {
get {
return Get("greetingUrl");
}
set {
Set("greetingUrl", value);
}
}
public static string PIN {
get {
return Get("pin");
}
}
public static string VoicemailEmailFromAddress {
get {
return Get("voicemailEmailFromAddress");
}
}
public static string VoicemailEmailToAddress {
get {
return Get("voicemailEmailToAddress");
}
}
}
The settings file looks like so (obviously this isn’t the most secure solution in the world, but this is only an example):
<?xml version="1.0" encoding="utf-8" ?>
<settings>
<greetingUrl></greetingUrl>
<pin>1234</pin>
<voicemailEmailToAddress>test@example.com</voicemailEmailToAddress>
<voicemailEmailFromAddress>voicemail@example.com</voicemailEmailFromAddress>
</settings>
We’re ready to start implementing our action methods. I’ll start with the first one hit when a call is received.
public ActionResult Index(string CallGuid, string Caller, string CallStatus) {
var doc = new XDocument();
var response = new XElement("Response");
var gather = Verb("Gather", "", new { action = ActionUrl.Greeting,
method = "POST",
finishOnKey = "#" });
// say current greeting
if (string.IsNullOrEmpty(Settings.GreetingUrl)) {
gather.Add(Verb("Say", "Thank you for calling the league hot line. Please leave a message."));
}
else {
gather.Add(Verb("Play", Settings.GreetingUrl));
}
response.Add(gather);
response.Add(Verb("Record", "", new { maxLength = 120,
action = ActionUrl.RecordVoicemail,
method = "POST" }));
doc.Add(response);
return new XmlResult(doc);
}
This method builds the required Twilio response XML. The first thing we add is a Gather verb which listens for digits being pressed. When the # key is pressed, a POST request is sent off to /Greeting (generated using the ActionUrl helper class I wrote to make sure the complete URL is returned). To keep listening for input while the greeting is being played, we nest the Say or Play verb inside the Gather verb. Lastly a Record verb is added which lets the caller leave a message which is posted to /RecordVoicemail.
Twilio passes some standard parameters (like CallGuid, Caller and CallStatus) with every request. ASP.NET MVC makes it really easy to get these values using parameter binding.
I’ve written a helper method that makes it easy to generate XElements in the proper form. The only thing it doesn’t really support very well is nesting, but it’s easy enough to get around that and you could easily add params parameter to accept an array of child elements.
private XElement Verb(string verb, string value) {
return Verb(verb, value, null);
}
private XElement Verb(string verb, string value, object paramObject) {
var element = new XElement(verb, value);
foreach (var item in paramObject.ToDictionary()) {
element.Add(new XAttribute(item.Key, item.Value));
}
return element;
}
The last parameter lets you pass in an anonymous object to set attributes on the XElement, similar to ASP.NET MVC’s use of anonymous objects. ToDictionary() is an extension method which has been posted in numerous places.
Since this is getting long, I think I’ll stop here and do a Part 2 with the remaining method implementations.
