On November 5th I gave a talk at the Twin Cities .NET User Group entitled ‘Using web service APIs in your applications’. I started out with a very brief overview of how REST APIs work and then talked about the current state of using REST APIs in .NET. I followed that up with a demonstration of a new OSS project I’m working on for making accessing REST APIs much easier. The project is called RestSharp but in the talk I refer to it as Stillwater as that was my working name for it prior to coming up with the final name.
Next I put RestSharp to use by writing a Twitter search bot that runs as an Azure worker role searching for mentions of a search term and then creating FogBugz tasks from the results. Then I demonstrated using Twilio to receive phone calls (utilizing ASP.NET MVC) and create FogBugz cases from those incoming phone calls. And lastly I demonstrated how to use RestSharp with Twilio’s REST API to initiate an outgoing phone call from a .NET app.
This was my first formal technical presentation (I gave one at jQuery Conference, but that was much more informal) and you can tell early on in the video. Stick with it however, things smooth out as they go along. Also, the audio is a little tinny with some minor background noise, but it’s not unbearable.
I’ll be posting more about RestSharp/Stillwater soon as I get closer to launching it. You can follow @RestSharp on Twitter, follow the project on GitHub or visit the official site (nothing there yet though).
Watch the Video (49:25)
Download (right click, save as): MP4 (103MB) | WMV (135MB)
Back when I launched ManagedAssembly.com around MIX I put together a page to show real-time Twitter search results for ‘MIX09’. On the heels of the dvplrs.com launch this week and the lack of .NET representation, I decided to update my live Twitter page to show live updates of notable people in the .NET community. This is how I built it. You can see the finished product over at ManagedAssembly.com/Twitter.
Since I decided to use the standard Twitter API instead of the Search API, I had to build a server-side cache of updates so that I wasn’t hitting the API on every view of the page which would quickly exhaust the API rate limits. Using a database was overkill so I decided to use ASP.NET’s built-in cache mechanism to store new updates. When the client makes a request for the latest updates, the items will be served out of the cache. The cache is updated every so often with new items.
The workflow when the client makes a request for updates (via jQuery) is like so:
- Grab the date of the last cache update from the cache. If the value doesn’t exist or it’s older than the cache expiration threshold, it’s time to get more statuses from Twitter.
- When new updates are pulled from Twitter, store the highest ID in the cache as well so that for future requests we can get only the new tweets.
- Pull the currently cached items and add the new items from the Twitter API call to it.
- When updates are sent to the browser, the client-side code stores the latest ID. For future requests this ID is passed back to the server which returns items from the cached list that are greater than the locally-stored ID.
What this gives us is the ability to only call Twitter once a minute no matter how many times the site is hit. It also allows for each visitor to only get back tweets that are new to them (which is different for every visitor depending on when the first hit the page) whenever they make a request. Since the tweets are being pulled from memory on almost every request there’s no delay while the API call is being made. If the application restarts and we lose the cache it won’t really matter because we just grab all the updates it will send and repopulate the cache. It also means that the amount of data being sent to the client via JSON is the bare minimum needed. That makes updating the UI simple (just append the new ones without having to process the full list) and keeps the bandwidth requirements low.
That was a lot of words, so time for some code. We start by determining if we need to get new items from Twitter and if so, grabbing them. This uses the CacheManager from a previous post.
CacheManager cache = new CacheManager(new ShortTermProvider());
IEnumerable<TwitterStatus> statuses = new List<TwitterStatus>();
long maxId = 0;
DateTime lastFetch = cache.Get<DateTime>("last_fetch");
if (lastFetch < DateTime.Now.AddMinutes(-1)) {
long lastCacheId = cache.Get<long>("last_id");
if (lastCacheId == 0)
lastCacheId = 1; // TweetSharp/twitter returns null if you use Since(0)
var request = FluentTwitter.CreateRequest().AuthenticateAs("UserGoesHere", "passwordgoeshere");
request.Configuration.UseGzipCompression();
request.Statuses().OnFriendsTimeline().Since(lastCacheId).AsJson();
statuses = request.Request().AsStatuses();
}
The first time this is hit lastFetch doesn’t exist in cache and the cache manager returns the default for DateTime (DateTime.MinValue) which I’m pretty sure is always more than a minute ago. We’ll update the fetch time in cache later. Once we’ve determined that the cache is empty or expired, we grab the last_id value from cache (which is 0 if it doesn’t exist). We then use TweetSharp to get the latest updates and store them in a list.
We also make sure that statuses isn’t null because if you’ve exceed the Twitter API rate limit, TweetSharp returns a null collection. Once we have the updates we have to process them since the text doesn’t include links for @mentions, #hashtags or links. When deciding how to approach this part I contacted Jon Galloway and he pointed me to the code in Witty which handles creating links, which I borrowed heavily from (thanks Jon!). MA also uses SubSonic 2.2 for the DAL and I used one of the methods provided in that library for determining if a word is a URL.
if (statuses != null) {
foreach (var status in statuses) {
string raw = status.Text;
string[] words = Regex.Split(raw, @"([ \(\)\{\}\[\]])");
StringBuilder output = new StringBuilder();
foreach (string word in words) {
if (word.StartsWith("#")) {
string hashtag = String.Empty;
Match foundHashtag = Regex.Match(word, @"#(\w+)(?<suffix>.*)");
if (foundHashtag.Success) {
hashtag = foundHashtag.Groups[1].Captures[0].Value;
output.Append(string.Format(@"#<a href=""http://search.twitter.com/search?q=%23{0}"" target=""_blank"">{0}</a>", hashtag));
}
}
else if (word.StartsWith("@")) {
string userName = String.Empty;
Match foundUserName = Regex.Match(word, @"@(\w+)(?<suffix>.*)");
if (foundUserName.Success) {
userName = foundUserName.Groups[1].Captures[0].Value;
output.Append(string.Format(@"@<a href=""http://twitter.com/{0}"" target=""_blank"">{0}</a>", userName));
}
}
else if (SubSonic.Sugar.Validation.IsURL(word)) {
output.Append(string.Format(@"<a href=""{0}"" target=""_blank"">{0}</a>", word));
}
else {
output.Append(word);
}
}
status.Text = output.ToString();
}
cache.Store("last_fetch", DateTime.Now);
}
OK now that we’ve got the latest updates from the API and added the links, it’s time to store them in the cache.
var cachedStatuses = cache.Get<List<TwitterStatus>>("cachedStatuses");
if (cachedStatuses == null) {
cachedStatuses = new List<TwitterStatus>();
}
if (statuses != null) {
cachedStatuses.AddRange(statuses);
}
if (cachedStatuses.Count > 0) {
maxId = cachedStatuses.Max(s => s.Id);
cache.Store("last_id", maxId);
}
cache.Store("cachedStatuses", cachedStatuses);
Last but not least we build a JSON return value for all the tweets in the cache with an ID higher than the one sent by the client (lid is a parameter passed from the client).
long lastClientId = lid ?? 1;
var data = from s in cachedStatuses
where s.Id > lastClientId
orderby s.Id descending
select new {
Id = s.Id,
ProfileImageUrl = s.User.ProfileImageUrl,
Text = s.Text,
Source = s.Source,
UserName = s.User.ScreenName,
RelativeTime = s.CreatedDate.ToRelativeTime(false),
ClientLink = s.Source,
IsReply = s.InReplyToStatusId != 0,
InReplyToId = s.InReplyToStatusId,
InReplyToUser = s.InReplyToScreenName
};
return Json(new { results = data.Take(100), max_id = maxId });
In Part 2 I’ll cover building the UI.
One of my favorite open source projects is the Argotic Syndication Framework. Argotic makes it really easy to generate RSS (and Atom and others) feeds with .NET. The library has a really clean API and has great extensibility support for RSS extensions like Media RSS.
I’ve been working almost exclusively with ASP.NET MVC the past five or six months and also been doing a lot of RSS feed generation. To make it easier to return RSS feeds from a controller action, I’ve created a new class that derives from ActionResult called RssResult. RssResult takes in a feed generated with Argotic and spits out the appropriate content type and content from the feed. Here’s what RssResult looks like:
public class RssResult : ActionResult
{
public RssFeed RssFeed { get; set; }
public RssResult(RssFeed feed) {
RssFeed = feed;
}
public override void ExecuteResult(ControllerContext context) {
context.HttpContext.Response.ContentType = "application/rss+xml";
SyndicationResourceSaveSettings settings = new SyndicationResourceSaveSettings();
settings.CharacterEncoding = new UTF8Encoding(false);
RssFeed.Save(context.HttpContext.Response.OutputStream, settings);
}
}
And here’s a sample controller action:
public ActionResult Index() {
PostCollection posts = PostService.ListPopular(1, 30);
RssFeed feed = new RssFeed();
feed.Channel.Title = "Managed Assembly : Popular";
feed.Channel.LastBuildDate = DateTime.Now;
foreach (var post in posts) {
feed.Channel.AddItem(new RssItem {
Author = post.User.DisplayName,
Description = post.FeedContents,
PublicationDate = post.CreateDate,
Title = post.FeedTitle,
Link = new Uri(post.FeedLink)
});
}
return Feed(feed);
}
My controller base class has the ‘Feed’ method like so:
public RssResult Feed(RssFeed feed) {
return new RssResult(feed);
}
In this example, the feed would be built up on every request. In some cases, I’ve added caching (using my CacheManager) so that the feed is only generated every 15 or 30 minutes.
The purpose of this post is to find a better way to write the code shown below. The examples below are not meant to be best practices or examples I would expect to be used in any system. I came up with this and am posting it to get a code review from anyone who stumbles upon it. I would love to find a better way to accomplish this, so if you have ideas or suggestions, please post them in the comments.
I have a side project I’m working on which requires users to be able to enter in comments. I wanted to support a very limited set of formatting so things like TinyMCE, FCKEditor, Markdown and the like are out. I have some basic requirements:
- Convert *italics* to <em>italics</em> and likewise for _bold_ to <strong>bold</strong>, but not in code blocks.
- For code blocks, convert lines starting with at least four spaces to be wrapped in <pre></pre> tags. Consecutive lines should be grouped together.
- Encode any HTML.
- Convert line breaks to HTML line breaks outside of code blocks.
I’ve created a text file you can look at that has some sample input.
For the first requirement, I’ve created a method to take a string input, a delimiter and an HTML tag to replace the delimiters with. I’ve also created a list of special characters used in regular expressions so I can assemble a proper regex if the delimiter is one of those characters.
private static List<string> SpecialRegexChars =
new List<string> { "$", "^", "{", "[", "(", "|", ")", "]", "}", "*", "+", "?", @"\" };
private static string ReplaceWithHtml(string input, string delimiter, string tag)
{
if (SpecialRegexChars.Contains(delimiter))
delimiter = @"\" + delimiter;
string regex = delimiter + "(.+)" + delimiter;
string output = input;
Regex r = new Regex(regex);
foreach (Match match in r.Matches(input))
{
if (match.Groups.Count > 1)
{
output = output.Replace(match.Groups[0].Value, string.Format("<{1}>{0}</{1}>", match.Groups[1].Value, tag));
}
}
return output;
}
Originally I ran this method using the entire sample input and it worked great, except for that if you had text surrounded by * or _ in a code block, it was replaced with a tag. So I set this aside and moved on to detecting code blocks so I could know where to avoid performing the replacement.
A code block for this scenario is any single or consecutive group a lines of text that start with four or more spaces. The beginning of every block needs to be prepended with a <pre> tag and appended with a </pre> closing tag. I considered using regex to handle this, but I couldn’t (and honestly didn’t want to) get my head around it. If there is a simple, easily-readable regex (there’s an oxymoron) that accomplishes what I’m about to demonstrate, please share it in the comments.
Short of an elegant regex, the next option I decided to try is splitting the input into an array of lines and looping through each line and determining whether or not it is a code block along with the lines before and after it. The vast majority of input will be relatively short so while a loop isn’t perfect, it actually works pretty well in this case. It also allows me to process the tag replacement on a per-line basis depending on whether or not that line is code.
Not too long after starting on my Codify method I had a morass of complicated and nearly unreadable ‘if’ statements. But, it worked! I was getting exactly the desired output. But I knew I could never come back and read the code later if the requirements changed. I worked at cleaning it up as much as possible focusing on improving readability, and this is the result:
private static string Codify(string input)
{
bool isInCodeBlock = false;
string[] lines = input.Split(new[] { Environment.NewLine }, StringSplitOptions.None);
for (int i = 0; i < lines.Length; i++)
{
bool isFirst = i == 0;
bool isLast = i == lines.Length – 1;
bool isNotFirst = i > 0;
bool isNotLast = i < lines.Length – 1;
string nextLine = (isNotLast ? lines[i + 1] : "").TrimEnd();
string prevLine = (isNotFirst ? lines[i - 1] : "").TrimEnd();
bool nextLineIsCode = isLast ? false : nextLine.StartsWith(" ");
bool prevLineIsCode = prevLine.StartsWith(" ");
string prefix = "";
string suffix = "";
string contents = HttpUtility.HtmlEncode(lines[i].TrimEnd());
bool thisLineIsCode = contents.StartsWith(" ");
if (((isFirst) || (isNotFirst && !isInCodeBlock)) && thisLineIsCode)
{
prefix = "<pre>" + Environment.NewLine;
isInCodeBlock = true;
}
if (!isInCodeBlock)
{
contents = ReplaceWithHtml(contents, "*", "em");
contents = ReplaceWithHtml(contents, "_", "strong");
}
if (isInCodeBlock && !nextLineIsCode)
{
suffix = Environment.NewLine + "</pre>";
isInCodeBlock = false;
}
if (!thisLineIsCode && !nextLineIsCode && !prevLineIsCode)
suffix = "<br />";
lines[i] = string.Concat(prefix, contents, suffix);
}
return string.Join(Environment.NewLine, lines);
}
While this isn’t the prettiest 45 lines of code I’ve ever written, it does work.
If you see something, anything, wrong with this code, tell me about it in the comments! Don’t hold back, I want to know everything you think is wrong with this code. I’m trying to work on not being attached to code I write so this will be good practice.
