Hacking journalstar.com RSS

December 10th, 2006
Lincoln, web

I’ve recently switched over my RSS subscriptions to Google Reader. Google Reader makes reading and managing a high volume of RSS feeds so effortless and intuitive, I was inspired to visit some of the sites I hoped to add to my feed reading list. As a recent expatriate of Lincoln, Nebraska, naturally I wanted to add the Journal Star, Lincoln’s local paper, to my read list, but it was apparent that the paper’s online division (or more likely, their tech-ignorant publishers) has not come around to embrace RSS. I wasn’t surprised, but still disappointed.

I found I wasn’t alone in my frustration.

So, I sat down to write a quick blog post complaining about their lack of feeds in the dawn of 2007 - but then I thought, “Who wants to read that?”. So I started looking for an alternative RSS-enabled news source for Lincoln, Nebraska.

Google news offers RSS feeds of searches on the site (RSS Feed for “lincoln nebraska”), but it isn’t comprehensive, and is mostly sports news anyway.

The UNL newspaper, the Daily Nebraskan, has a wide variety of feeds (DN RSS Feeds), but I really had enough of the DN in my years of attending classes at UNL. Thanks, but no thanks.

I SWEAR I had a feed of ‘most printed articles’ from the Lincoln CBS affiliate KOLN, but those ended up being mostly recipes, so I unsubscribed. Can’t find any mention of RSS there either.

I headed back to Google to see if anyone had set up a way to scrape Journal Star headlines and output them to RSS. No luck there either.

Lincoln, it seemed, wants to keep its news to itself.

As a last resort, I headed back to Google searching for all sorts of search combinations of RSS and Journal Star and I found this little gem: http://www.journalstar.com/?rss=huskerextra.

And after a bit more research, trial, and error, I found the format to their publishing system’s RSS output.

Here’s how it works. Their publishing system users a url structure like this: http://www.journalstar.com/articles/2006/12/10/news/local/doc457b548b7be44737957995.txt. The key to accessing the RSS is the directory structure between the date stamp (i.e. “2006/12/10″), and the ugly document name in the URL (i.e. “doc457b548b7be44737957995.txt”)

Just plug this directory structure in as the value for the RSS url parameter and you get: http://www.journalstar.com/?rss=news/local. Magic!

Here are some examples, with a handy link to add to Google Reader:

Even the police calls, bankruptcies and resurant inspections have RSS feeds. What fun!

Since the feeds exist, WHY wouldn’t they be promoted on the site? Clearly, they don’t realize that “RSS is the paperboy” (see question #4). An RSS subscriber is clearly more engaged and interested in the content. They are more likely to turn more pages, participate in comments, and post links on their blogs, in turn driving more traffic to the site. I don’t expect them to provide full content feeds, though some sites have had encoraging results.

So, now that this cat is out of the bag let’s hope that the Journal Star doesn’t try to stuff the cat back in. It would be simple enough to disable access by a variety of ways (.htaccess etc.). If they do disable the feeds, it will be clear that the Journal Star doesn’t understand this powerful and simple method of content delivery, doesn’t respect it’s users, and doesn’t care to learn.

We’ll see.

Trackback URL.

nic December 11, 2006 at 7:40 am

Nice, man!

Now to get crailtap an RSS feed.

Sydney December 11, 2006 at 11:20 am

you rock. thx for figuring that out. I’ve been diggin’ google reader too. Makes my news crawl a lot faster.