Language Oriented Programming

Recently I discovered the fascinating idea of language oriented programming and language workbenches. Centered mostly around Meta Programming System by Jetbrains and a little shady project of Intentional Software, the topic has been around for over a decade but is not mature enough yet (unfortunately it has not developed as fast as predicted by Martin Fowler). Language oriented programming is focused on bringing some major improvements to software development, the most ambitious one being narrowing the programmer involvement in development process. It seeks to enable programmers to give the domain experts required tools to let them build the software by themselves, where they would be best suited to the task. The programmers then would be able to focus on their specialty, which is the abstract world of programming itself (and finding bugs). They would create simple to use and error-avoiding domain specific languages for the experts to use, designed to the specific domain.

But beside this breakthrough idea, the language oriented programming concepts and its implementation may also revolutionize the development of existing programming languages and typical applications. It all begins with dropping code representation based on text and basing all development on direct changes to the abstract syntax tree. However, manual editing of the AST would not be very user-friendly, so it is done by a projection. In essence, the AST is projected into some editable representation (text-like most often) that interprets any changes made and updates the AST accordingly. So the AST is the single, unambiguous form being edited and the representation is derivative, not the other way around as in current state of art.

After playing with Jetbrains MPS and reading some blogs, I am very much enthusiastic about this concept:

• The projectional editor will make error checking and code completion instantaneous. It can also make typing faster after some practice if you are not a text maniac and believe it. I have to admit it takes some time to get used to and can be un-intuitive. BUT: the style wars will be gone forever (every developer may create their own looks for the language), and any developer will be able to use any style or representation they like.
• Implementation of any language is made much simpler, I would argue. Instead of writing a compiler from scratch, as it is done now, using a language workbench abstracts away all the boring work (like writing a look-ahead buffer for parsing). In an ideal world, this would become a standardized way of creating any language and all programmers would unite and cherish each other! As you can see, this is a second fact, along the lack of style wars, that Language Oriented Programming strives for world peace.
• Implementing some new features in a language becomes way simpler. No more backward compatibility issues, no more careful and hard work of hunting all the edge cases with interaction with existing features. The new features becomes just an extension to a language, integrating seamlessly with the language it wraps. Getting it involves just downloading a module from web repository after some other team member starts using the fancy feature in his code. All typical human programmers can create any syntactic sugar they like.
• The version control becomes intelligent. No more text comparing and trying to deduce logical changes from it. The version control system would be an integral part of the project being developed, seeing all changes ever made. A commit would become just a grouping of some changes. Even more importantly, changes could be automatically grouped based on logical, higher level differences, like creation of a new class, renaming a method and all references. Reordering the class members could be made transparent to the users, presenting no changes. Viewing changes of other team members would be easier and merge conflicts would be simpler to resolve.

I think these are some fundamental improvements of the way of programming making writing code much easier if one is able to trade the familiar text representation for them. While the projectional representation is not text, the MPS guys made a very good job improving it so it feels more like text. In the future, the difference might be even smaller.

Unfortunately, although via those features, creating code and collaborating would become easier, other aspects of software development remain mostly unchanged, like testing and debugging. And they are the causes making programming hard in general. This is mostly another topic, but Jonathan Edwards has some very interesting (and controversial) ideas about how to start programming profession from scratch.

Unit testing the functional way

Recently I stepped upon an interesting (although old) blog post by Christian Sunesson.
He got a point in saying that in OO programming the good design practices that make testability possible are very close to some functional programming principles. Unit testing and TDD encourages making the operations cleanly separated, asserting on value objects and refering to verification capabilities of mocks as a last resort or not at all. The more pure the method is, the more easy it is to test it. John Sonmez also refers to this topic.

But how to do unit testing with mostly pure functions and no mocks? How to design a system with small and large components and avoid dependencies between them? After all, we need to test in isolation. This brings a picture like this one below:

We unit test only the blue parts of the system and bind them together using the large green one.It does not look very scalable. However, the green part, which is the application layer, might in turn be composed of a few subsystems (and communicating with other such component groups), wrapping all hard-to-unit test logic, like concurrency. Then, only end-to-end tests could be used to verify it.

Each of the non-dependent blue leaf components could consist of only pure functional logic. But what about some leaf component that is overly complicated and should be refactored into several smaller ones? it is still a leaf from outside point of view, but now it has some dependencies inside.

In order to avoid mocks, the test isolation principle should be changed: testing the root component is not done in isolation - it uses the actual smaller parts.

But isolation paradigm is in place to quickly reason about failed tests. If some test fails, we can be sure that the faulty logic is the one it invokes in the first place, in isolation. It is not the case in testing dependent components without mocks: the fault might be inside some sub-part that is invoked in the test dealing with compound objects. It looks like it would be harder to find the bug.

But lets make an assumption, that all the sub-parts are also covered by tests. Each of the components is tested on its own (but not in isolation), being part of a test "onion" -- every higher level, more complex component test adding an extra layer on it. Then, the failed tests will form a red cross section through this onion. Finding a bug is such a test suite is just a matter of getting to the core of the cross section -- locating the most simple component test that is failing.

This kind of approach clearly asks for tooling support. If the dependency graph of components could somehow be hierarchically visualized, then locating such bugs would be simple. And many mocks could be avoided.