With any worry about having too many buzzwords and technologies in one place cast aside, this post is the first in a series which will show you how I developed a Dropbox web application using the Dropbox restful OAuth API with a compositional approach using new and little discussed NetKernel compositional development environment nCoDE.
First a little background on compositional development and nCoDE. Compositional development is an approach that focuses on the assembly and configuration of pre-existing components to build systems. It’s an approach that has always been at the heart of NetKernel from the early days of XML processing pipelines with a rich set of accessors and DPML to bind them. With the release of NetKernel 4 our emphasis and efforts shifted ever so slightly. The resource oriented abstraction enabled lots of exciting new patterns and the freedom of representational expression (read “you can use any old immutable object for state transfer”). That was all well and good, very good in fact, but it left the compositional approach in a little bit of a dilemma. The problem is that composition works best in very constrained environments and now we had a general purpose architecture building abstraction. It’s not surprising that since then most development both internally at 1060 and with users of NetKernel has been done with procedural code, either Java or one of the scripting runtimes such as Groovy.
You’ve probably guessed where this is leading. Yep, I wasn’t happy about this state of affairs. Mainly because one on the tangible benefits of NetKernel has been complexity management. If you read this blog you know my Death Star end-game but on a day to day basis baby sitting technology isn’t my idea of fun. I’d rather it just worked. To truly achieve this you need the ability to step above the minutia of code and build with well designed and implemented blocks. We had achieved this with spacial structure using the standard module infrastructure and the space explorer but not with information processes.
DPML was ported to NetKernel 4 early on in the development cycle but didn’t get huge amounts of love. For those who don’t know it, DPML stands for Declarative Process Markup Language. Think of it as an assembly language for resource oriented processes. It’s kind of unique as a language in that it really cannot do much by itself. What it does do though is orchestrate resource requests and their responses and arguments. Even basic constructs such as conditional processing and exception handling are external to the language. This is perfect for composition.
But DPML is not the whole story. Freedom in the abstraction has lead to many different resource representations and argument passing conventions. Let me share the example of forEach with you. There are currently two forEach accessors in the code NetKernel libraries. The first one is in Layer1 module. It makes extensive use of declarative requests (an XML syntax for defining a resource request) for defining the operations of how and what their arguments are. Whilst declarative requests are very flexible, the problem is (when it comes to composition) that it makes these services need to be external which leads to a lot of fine grained services. Another forEach acessor was developed for use in nCoDE (yes we’re getting to it soon!). This one takes quite a different approach by injecting a space containing all the resources sub-requests might need. This fits more closely with lazy evaluation approach of DPML. The point of this example is not to say one approach is right and one wrong, but rather to show one of the ways in which flexibility of the abstraction has not helped the compositional approach.
Now given how quiet I’ve been of the past months you might think I’ve had my feet up in the hammock. But you have to remember that you only get a few good hammock days in the UK, the rest of the time it’s rainy and windy. So I’ve been keeping busy working towards making compositional development work. This work has included:
determining conventions and patterns that give flexibility whilst still being composable. This has led to lots of work that has been described in the weekly newsletters as suitably vague “metadata updates” but in reality has been a lot of teasing apart of assumptions to allow a compositional development environment to …just… work.
rewriting DPML to interact cleanly with the ROC abstraction as we see it now rather than how it might of been before we knew what it was! The benefits of this is better debug-ability, better caching, higher performance and no obscure corner cases.
fleshing out low-level common endpoints libraries needed to avoid needing to drop down to procedural code.
developing a workable vision and first implementation of a visual composition environment nCoDE.
So where are we at? Well we are doing well. nCoDE was released to coincide with the NKWest conference in April but has received considerable refinement since and now runs on the new DPML. All of the metadata updates have bedded in and formed the foundation that drives nCoDE. Coverage of compositional libraries is kind of patchy but that is why I’m at the stage I am of pushing nCoDE to do real things. And that leads us nicely to the topic of the mini-series of posts - Dropbox with nCoDE. TADA
Watch this space for part two and we’ll actually do something rather than just talk about it…
Ok to keep you happy hear is a teaser nCoDE process. See if you can work out what it does:
(note ironic use of groovy)
BTW if you want to start playing with nCoDE download NetKernel then go to the apposite package manager and install the nCoDE module. After it’s installed the new module wizard within the developer tools on the web based admin panel will let you create a new module and get playing straight away.