The MP3 format is a great technology for compressing audio because it can be progressively configured to reduce the complexity in an audio stream in the parts that humans are less able to perceive. It understands the psycho-acoustics of human hearing and the typical waveforms that constitute music. By working in a constrained domain it can represent the data in a more concentrated form. Similarly JPEG is a great technology for compressing photographs because it can reduce the complexity in parts of an image that humans are less able to perceive. It knows that typical photographs contain large areas of flat colour or slowly changing gradients; it knows that our eyes see less colour information than they see luminance. The algorithms behind these technologies are very smart, they understand the nature of the data and take advantage of that fact to represent the information in a more concentrated form. There are many more examples of these domain specific compressions in video encoding.
ZIP is great technology for compressing general purpose data streams. It can produce a representation of the original data which is smaller and without loss. The ZIP algorithm works by finding common patterns throughout the data and uses a language to express those patterns. Whilst this works for “typical” data it doesn’t always work. By “typical” data I mean real world data that contains patterns and is not completely random. Hold that thought…
Now lets switch gears to think about software. Imagine a language that could express typical software patterns and architectures in a more compact way - it would translate to less code. If we understood what constituted the essential essence of typical software systems with a language to express it then we could “decompress” this to create an instantiation of that system in all it’s gory detail needed by the complex hardware of today’s computers. By this I do not mean just putting some source code through a compression algorithm. That would be no good - because the compressed data could not be authored. What I mean is a language for expressing the system which enables you to achieve a lot with a little.
One example of what I mean is a technology like Ruby on Rails, a fairly small set of files can create a rich application. A small set of input data can create large software systems, in fact orders of magnitude smaller than it would be if that same application was create using a procedural programming language. It achieves this by starting with the concept of what it is trying to create, in this case a database backed web application, and just requires the minimal “configuration” to make that work. Then additional data can be added and that deviates the application away from it’s initial form in well defined and useful ways. So you can end up with a rich and powerful application with a very small amount of code.
Let me put this example into context. A long time ago I used to work on developing games on 8bit home computers. In that day hardware was very simple, particularly audio hardware. It was common to use what was called an Envelope to define sound effects and notes for music. The envelope consisted of a list of dozen or so integers expressed as bytes that defined various characteristics of the sound to be played such as how fast it’s amplitude rose from silence called attack and the pattern of how it’s pitch varied to create tremolo effects of musical arpeggios. Those dozen bytes defined all the sounds that you could create. You could actually get a surprising richness of sounds! If those bytes were put in an raw WAV file you would get much much less. But it is clear that you are never going to get the realistic sound of a piano never mind the expressiveness of the human voice from such a technology. Certain sounds, in fact most sounds, were just outside it’s capability.
Looking again at the example of Ruby on Rails, we can see that although you can create some very rich and useful applications there are many things it cannot be used for. For example it couldn’t be used for creating an Internet DNS server and it couldn’t be used for creating a 3D first person shooting game. There are lots of things it couldn’t be used for. That is not to be negative about it. It makes a very deliberate trade-off - it enables a small representation for systems which fit within it’s problem domain.
There seems to be a trend in the IT industry towards solution/frameworks like these. They appear on face value to be a good solution to creating the necessary rich and complex applications that are required whilst managing the development costs and the specialist knowledge required. What can be wrong with this? With enough frameworks we can completely cover the space and then it’s just a matter of picking the right one - right? However I believe they are not ideal. To understand why lets look at these three questions:
What to do when you hit a limit to their degrees of freedom? When you hit a constraint in a given technology then if you are lucky it’s a soft limit and you can drop out of the abstraction it offers to a lower level and solve your problem. Depending upon what you want to do that might completely sidestep any benefits derived from using it in the first place. It is essential that you have this capability though.
Can you understand enough about their scope up front to make a good choice for your future requirements? Committing to technology is hard. You know that it’s making you job easy up front or you’d already have passed it by. But what happens when you’ve got deep into you solution and you hit a limitation? What happens if you hit a performance bottleneck in production and it is an inherent characteristic of the approach? The best you can do is to do your homework before committing and hope that some subtle issue doesn’t bite hard.
What happens when different parts of your system needs different technologies to be integrated? Merging one or more technologies into a cohesive application can be hard. Different technologies come from different development teams and are based on different paradigms and approaches. The glue code to connect them is often nasty, hard to maintain and is hard to separate from your applications custom code.
So just like the ZIP compression is to MP3 let us look at software representation technology that isn’t so narrow in scope. Let us look at one which aims to be able to represent pretty much any enterprise software whilst still concentrating the representation; reducing the amount of code. Unless you thought I’d had a last minute change of mind then you know what’s coming next. Yep NetKernel. A common theme across pretty much everyone who has put a NetKernel system into production is the surprise at how little code there is. In fact one of our customers was a little embarrassed to share too much of the details of their solution to the client for worry that they’d be expecting much more for their money.
So how is this possible? That’s a good question and one that isn’t that easy to answer. We started out trying to change the economics of software in the narrow domain of XML message exchange systems back in the last century. Because of our backgrounds and the isolation from commercial pressures in the labs of HP we serendipitously discovered a new way. Over time we managed to distil this way into what we now call Resource Oriented Computing (ROC). An abstraction which when combined with a number of core technologies such as the standard module definition create basis for richly representing enterprise software very concisely, liberating the essence of software architecture from the nitty gritty details of code.