Scratching an Itch

Some of you may know that recently I’ve been really digging into the Getting Things Done (GTD) methodology of self organisation. Thanks to Tom Geudens for putting me on that track! I’m not going to bang on obsessively about it as it seems that can be a common tendency, suffice to say I found good geeky tools to use and so far so good it really seems to work. The one takeaway point to me is this: if you can get written down and out of your head all the thoughts and ideas of what needs to be done both long and short term then these thoughts don’t keep springing into your head at inopportune times. (It’s usually when you are concentrating on something else or being in a situation when you can’t do anything about it.)

However despite all this satisfaction there is a class of activities that I didn’t find a satisfactory solution too. These fall into the category of needing “push not pull.” A good example of this and one that this posting addresses is the case of wanting to keep abreast of Twitter tweets. I’d often arrive late into the office to be confronted by Peter asking me if I’d seen so and so had said something about NetKernel. I’d say politely “no not yet, I’ve had more important things to do that sit twitter watching.” So I looked around for some kind of notification system that would ideally be an app that would send iOS notifications when a new tweet was found that matched a saved search. I didn’t find anything but realised that actually it’s a pretty simple thing to do with NetKernel. I’m going show you how I did it…

Twitter has a public REST API which has an service for issuing searches and getting the results back as a JSON representation. For the purposes of my usage this service doesn’t need authentication and takes two parameters one is the search terms and the other a “since_id” which is some kind of a timestamp that is used only to return tweets after some datum. This is useful for only retrieving incremental changes since the last request.

What I did in NetKernel was to create a new module with a single endpoint that I would make addressable and call periodically from the CRON transport. This endpoint does the following tasks:

1) load the app configuration resource as HDS (NetKernel’s lightweight Hierarchical Data Structure replacement for XML)
2) issue a http request to twitter with the search term and since_id extracted from the config.
3) transrept the JSON response into HDS
4) extract the new since_id, update the configuration and SINK the configuration
5) iterate through new tweets extracting sender and their message and issue a request to sendmail accessor using additional parameters extracted from the configuration.

Here is the groovy code of the service endpoint:

import org.netkernel.layer0.nkf.*
import org.netkernel.layer0.representation.*
import org.netkernel.layer0.representation.impl.*
import org.netkernel.xml.util.*
import java.net.URLEncoder;
 
//Load configuration and extract parameters
config=context.source("res:/etc/system/tweetNotifyConfig.xml",IHDSNode.class);
search=config.getFirstValue("/config/search");
search=URLEncoder.encode(search, "UTF-8");
since=config.getFirstValue("/config/since");
emailTo=config.getFirstValue("/config/emailTo");
emailFrom=config.getFirstValue("/config/emailFrom");
 
//Issue search request to twitter
req=context.createRequest("active:httpGet")
req.addArgument("url", "http://search.twitter.com/search.json?q=$search&since_id=$since".toString())
req.setRepresentationClass(org.json.JSONObject.class)
rep=context.issueRequest(req)
 
// transrept to JSON to HDS
req=context.createRequest("active:JSONToXML")
req.addArgumentByValue("operand", rep)
req.setRepresentationClass(java.lang.String.class)
rep=context.issueRequest(req)
 
// Add an single XML root and parse into HDS
rep="<twitter>$rep</twitter>".toString()
feed=context.transrept(rep,IHDSNode.class);
 
//extract max_id and update since_id in config
since=feed.getFirstValue("/twitter/max_id_str");
b=new HDSBuilder();
b.importChildren(config);
b.setCursor(b.getRoot().getFirstNode("/config/since"));
b.setValue(since);
context.sink("res:/etc/system/tweetNotifyConfig.xml",b.getRoot());
 
//iterate through tweets and sendmail
for (IHDSNode result : feed.getNodes("/twitter/results"))
{ from=result.getFirstValue("from_user_name")+" (twitter)";
 text=result.getFirstValue("text");
 System.out.println(from+": "+text);
 sendEmail(emailTo,from,emailFrom,text);
}
 
context.createResponseFrom(feed).setExpiry(INKFResponse.EXPIRY_ALWAYS);

A download of the module is available here.

The final link in the chain: receiving the email. Sending to an email service that supports push means that you can be notified almost instantly. Using a real hidden gem of a product called Mail Enhancer Pro for iOS means that I can create a customised notification tone specifically for emails received from this service by matching on sender and subject fields in the email. Now if Peter blinks I may even know of a tweet before him. Job done!