Building a mostly real-time web-based Twitter client with ASP.NET MVC, jQuery and TweetSharp, Part 1

Back when I launched ManagedAssembly.com around MIX I put together a page to show real-time Twitter search results for ‘MIX09’. On the heels of the dvplrs.com launch this week and the lack of .NET representation, I decided to update my live Twitter page to show live updates of notable people in the .NET community. This is how I built it. You can see the finished product over at ManagedAssembly.com/Twitter.

Since I decided to use the standard Twitter API instead of the Search API, I had to build a server-side cache of updates so that I wasn’t hitting the API on every view of the page which would quickly exhaust the API rate limits. Using a database was overkill so I decided to use ASP.NET’s built-in cache mechanism to store new updates. When the client makes a request for the latest updates, the items will be served out of the cache. The cache is updated every so often with new items.

The workflow when the client makes a request for updates (via jQuery) is like so:

• Grab the date of the last cache update from the cache. If the value doesn’t exist or it’s older than the cache expiration threshold, it’s time to get more statuses from Twitter.
• When new updates are pulled from Twitter, store the highest ID in the cache as well so that for future requests we can get only the new tweets.
• Pull the currently cached items and add the new items from the Twitter API call to it.
• When updates are sent to the browser, the client-side code stores the latest ID. For future requests this ID is passed back to the server which returns items from the cached list that are greater than the locally-stored ID.

What this gives us is the ability to only call Twitter once a minute no matter how many times the site is hit. It also allows for each visitor to only get back tweets that are new to them (which is different for every visitor depending on when the first hit the page) whenever they make a request. Since the tweets are being pulled from memory on almost every request there’s no delay while the API call is being made. If the application restarts and we lose the cache it won’t really matter because we just grab all the updates it will send and repopulate the cache. It also means that the amount of data being sent to the client via JSON is the bare minimum needed. That makes updating the UI simple (just append the new ones without having to process the full list) and keeps the bandwidth requirements low.

That was a lot of words, so time for some code. We start by determining if we need to get new items from Twitter and if so, grabbing them. This uses the CacheManager from a previous post.

CacheManager cache = new CacheManager(new ShortTermProvider());

IEnumerable<TwitterStatus> statuses = new List<TwitterStatus>();

long maxId = 0;

DateTime lastFetch = cache.Get<DateTime>("last_fetch");

if (lastFetch < DateTime.Now.AddMinutes(-1)) {

    long lastCacheId = cache.Get<long>("last_id");

    if (lastCacheId == 0)

        lastCacheId = 1; // TweetSharp/twitter returns null if you use Since(0)

    var request = FluentTwitter.CreateRequest().AuthenticateAs("UserGoesHere", "passwordgoeshere");

    request.Configuration.UseGzipCompression();

    request.Statuses().OnFriendsTimeline().Since(lastCacheId).AsJson();

    statuses = request.Request().AsStatuses();

}

The first time this is hit lastFetch doesn’t exist in cache and the cache manager returns the default for DateTime (DateTime.MinValue) which I’m pretty sure is always more than a minute ago. We’ll update the fetch time in cache later. Once we’ve determined that the cache is empty or expired, we grab the last_id value from cache (which is 0 if it doesn’t exist). We then use TweetSharp to get the latest updates and store them in a list.

We also make sure that statuses isn’t null because if you’ve exceed the Twitter API rate limit, TweetSharp returns a null collection. Once we have the updates we have to process them since the text doesn’t include links for @mentions, #hashtags or links. When deciding how to approach this part I contacted Jon Galloway and he pointed me to the code in Witty which handles creating links, which I borrowed heavily from (thanks Jon!). MA also uses SubSonic 2.2 for the DAL and I used one of the methods provided in that library for determining if a word is a URL.

if (statuses != null) {

    foreach (var status in statuses) {

        string raw = status.Text;

        string[] words = Regex.Split(raw, @"([ \(\)\{\}\[\]])");

        StringBuilder output = new StringBuilder();

        foreach (string word in words) {

            if (word.StartsWith("#")) {

                string hashtag = String.Empty;

                Match foundHashtag = Regex.Match(word, @"#(\w+)(?<suffix>.*)");

                if (foundHashtag.Success) {

                    hashtag = foundHashtag.Groups[1].Captures[0].Value;

                    output.Append(string.Format(@"#<a href=""http://search.twitter.com/search?q=%23{0}"" target=""_blank"">{0}</a>", hashtag));

                }

            }

            else if (word.StartsWith("@")) {

                string userName = String.Empty;

                Match foundUserName = Regex.Match(word, @"@(\w+)(?<suffix>.*)");

                if (foundUserName.Success) {

                    userName = foundUserName.Groups[1].Captures[0].Value;

                    output.Append(string.Format(@"@<a href=""http://twitter.com/{0}"" target=""_blank"">{0}</a>", userName));

                }

            }

            else if (SubSonic.Sugar.Validation.IsURL(word)) {

                output.Append(string.Format(@"<a href=""{0}"" target=""_blank"">{0}</a>", word));

            }

            else {

                output.Append(word);

            }

        }

        status.Text = output.ToString();

    }

    cache.Store("last_fetch", DateTime.Now);

}

OK now that we’ve got the latest updates from the API and added the links, it’s time to store them in the cache.

var cachedStatuses = cache.Get<List<TwitterStatus>>("cachedStatuses");

if (cachedStatuses == null) {

    cachedStatuses = new List<TwitterStatus>();

}

if (statuses != null) {

    cachedStatuses.AddRange(statuses);

}

if (cachedStatuses.Count > 0) {

    maxId = cachedStatuses.Max(s => s.Id);

    cache.Store("last_id", maxId);

}

cache.Store("cachedStatuses", cachedStatuses);

Last but not least we build a JSON return value for all the tweets in the cache with an ID higher than the one sent by the client (lid is a parameter passed from the client).

long lastClientId = lid ?? 1;

var data = from s in cachedStatuses

           where s.Id > lastClientId

           orderby s.Id descending

           select new {

               Id = s.Id,

               ProfileImageUrl = s.User.ProfileImageUrl,

               Text = s.Text,

               Source = s.Source,

               UserName = s.User.ScreenName,

               RelativeTime = s.CreatedDate.ToRelativeTime(false),

               ClientLink = s.Source,

               IsReply = s.InReplyToStatusId != 0,

               InReplyToId = s.InReplyToStatusId,

               InReplyToUser = s.InReplyToScreenName

           };

return Json(new { results = data.Take(100), max_id = maxId });

In Part 2 I’ll cover building the UI.