I really share Jimmy’s concern, especially when I think about the XML abuse I’ve been subjected to while creating languages and recipes using the Domain-Specific Language (DSL) Tools and the Guidance Automation Toolkit (GAT).
On the other hand, I recognize that it is totally unfair for me to complain; this is, after all, the price we pay for dealing with prerelease software.
In fact, the DSL Tools team already announced that will soon provide us with a modeling language able to deal with the .dsldd file and hide that ugly XML from our sensitive eyes.
I haven’t heard of announcements regarding similar improvements on the GAT side, but I choose to be optimistic and wait for new developments.

Unfortunately this problem does not affect prereleases exclusively; released software, both closed and open source, is not immune either.
Let’s face it, it is far too easy to define and process a language using XML for not taking advantage of it; in a not-that-distant future, however, I have no doubts that we will look back and laugh thinking about how we put up with manually dealing with its angle brackets notation on a daily basis for all these years.

Apart from really extreme cases such as XSLT, perhaps XML-based build languages of the likes of Ant, NAnt and MSBuild are among the most obvious and discussed examples of how a good idea can turn nasty when scenarios get a little more complex than anticipated.

Martin Fowler already suggested that today’s enterprise applications require pretty sophisticated builds which are better handled using a full-fledged programming language.

Last year, he also wrote an article about his explorations using Rake, a build language based on Ruby. In a nutshell, using an internal (text-based) domain-specific language such as Rake enables developers to easily switch to the external general-purpose language (Ruby) whenever necessary.

But how sophisticated can builds be? To put a little bit of perspective into things, let me tell you that for more than two years at InnerWorkings we have adopted a continuous integration solution based on CruiseControl. NET and NAnt; unless you’ve read Jimmy’s brilliant manuscript of his upcoming book (in which I was honored to make a small contribution as guest author) you will be surprised to hear that we manage to automatically and continuously build, obfuscate, package, test and deploy several hundred .NET solutions.

While I think that we have done more than a reasonable job at maintaining a good degree of modularity and reuse in our XML scripts, our tolerance to NAnt’s notation has been tested several times.

Resorting to a DSL written on top of a full programming language as Martin suggests would definitely help, but I have the impression that a purely text-based language may still be rather inadequate at solving one of the issues that we often face, particularly when dealing with many scripts in parallel like our scenario requires: lack of run-time visibility about the progress of each build.
Do I have a viable solution? I’m not sure, but I speculate that it won’t be long before we will see the raise of a new generation of continuous integration systems built around the upcoming Windows Workflow Foundation, with steps defined using domain-specific activities.
What do you think? Is it a daft idea?

There are several things I feel I should write about (including the amazing software architecture workshop that took place last month in Cortina d’Ampezzo, Italy) but today I will just mention the more recent Microsoft Architect Forum 2006 which was superbly organized by Bill O’Brien here in Dublin.
I really couldn’t miss the full-day event since Beat Schwegler and Ingo Rammer were set to dig deep into a topic that I simply can’t afford to ignore these days: Software Factories .

Beat and Ingo were excellent in articulating Microsoft’s vision of how to combine model-driven development, guidance, frameworks and tools together with the objective of creating one or more product lines able to systematically exploit commonalities among the members of software product families .
Bill has kindly made available all the slides in this post so I won’t go into the details of each session.

During his analysis, Beat emphasized that we should consider that, in many cases, up to 70% of the cost of a software project goes into operations rather than pure development; as a consequence, he recommended that we should start adopting model-driven development not only for code generation, but also for requirements, deployment, configuration, and, more generally, for all other activities that are involved during the full lifecycle of a project.

While in principle I don’t disagree with this thought, today I find quite unlikely that different stakeholders (business analysts, network engineers, enterprise architects, solutions architects, developers, security specialists, QA testers, etc.) would accept to use the current incarnation of tools and designers and be confined to one single hosting environment, namely Visual Studio Team System.
But hey, in fairness we are talking about a medium-to-long term goal here, so I will surely change my mind on this point whenever Microsoft (or I ) will get there.

Ingo was really impeccable throughout the day and used several examples to illustrate the capabilities of the DSL tools. In one specific instance however, I could not help but notice that the version of the domain specific language that he used did not provide a particular option he needed for his demonstration; he then resorted to manually modify the generated code to accomplish his objectives. Tut-tut Ingo, you are not supposed to do that .
I know I know, it was just a demo, and I really sound way too fussy.

It would be good however if somebody out there explained that, in the real world:

• We obviously cannot modify code once generated since the DSL models will diligently overwrite everything at the next transformation; as a consequence:
  • Put a comment header in each template to explain that “This code has been generated by a tool…do not modify…etc.”
  • Do not put the generated code under source control. That code is a dispensable artifact. Put the designer file and the templates under source control instead.
• Modifying a template is clearly a better option, generally however:
  • You need to put it under source control
  • You need to deploy it in a centralized location so that it can be shared across different applications in the same family
  • A change in the template may not be done lightly as it could easily break all other existing solutions that use the same template.
  • You need to unambiguously identify the version in use (you may put a version number in the header of even resort to change file name if changes are substantial and break existing applications that use it)
  • A change in the template that breaks existing applications may trigger the beginning of a new product line, particularly if you can’t accomplish 100% code generation of a solution and you can’t afford to retrofit all the existing solutions.
• If we are unable to build systems that achieve 100% code generation (which happens if the degree of variation in a product family is not completely known):
  • We need careful guidance (patterns) to understand how to happily write manual code beside generated code. How do we create the extension points? What do you put in the underlying frameworks and what do you keep in a template? Who should make the changes in the first place? Is it up to us to rediscover these tricks over and over? Sure we can use base classes, template methods, etc. But no, the one-size-fits-all solution of using partial classes does not work all the time (wake up people, we often deal with xml files with little or no control on the reader…has anyone heard of the app.config file for example?)
  • There is an evident abstraction leak as developers need to appreciate the generated code as they are going to write beside it and even debug it if necessary (tip: keep templates “thin” by leveraging rich frameworks instead). By the way, the idea that the “smartest” people will write DSLs and templates while the others will just use them is flawed in my opinion, but I will definitely save an explanation for another post.

I’m exhausted already and I’m aware that this list is far from complete; and I haven’t even started talking about how I see we could version a domain specific language or an entire product line using the current tools. Or maybe I should attempt to figure out how I would assemble effective teams within this new paradigm. Has anybody discussed yet how to deal with developer’s natural resistance to model-driven development? How should we address their concerns of domain-restricted job specialization and the rightful dislike for anything that contains the word factory in it? Or has everybody agreed that this is not an issue?

As often happens, the real problem is that the tools are getting there, with their capabilities and limitations, and we really need to go beyond the simple APIs to be good at using them.
But perhaps it is unfair to ask a toolmaker to tell us out how to excel at using those tools.
After all Mozart wasn’t a piano maker, right ?

However, as vendors are becoming more successful in fulfilling the promise of enabling people within organizations to model their own domain specific languages, many are realizing that simply mastering the features (and limitations) of a particular modeling tool is manifestly insufficient.

I’m certainly not trying to classify such tools as interchangeable commodities; in fact, I speculate that, at least for the next few years, the choice of one tool versus another could easily contribute to the failure of a project.
There are already several documented cases of organizations that use modeling tools to drive entire product lines with enormous success. Frankly, I just assume that the very same tools have been employed within different realities with opposite results.

One of the goals for DSL authors is to generate as much code as possible whilst hiding implementation details behind reasonably flexible models and notations. This goal presents challenges that go beyond the tool in use.

If it is that hard, who should create DSLs? Will organizations need to create a specialized development role as some are arguing?
It is realistic to assume that experienced developers/architects who fully understand the domain and have been involved in the development of enterprise frameworks are the natural candidates for authoring well thought DSLs for an individual organization.

On the other hand, there is a danger that such authors might treat DSLs exactly like class frameworks, without realizing that the forces at work are significantly different.

When building frameworks we don’t necessarily know how our components will be consumed. For example, if we create a custom control and omit to expose a particular event, the entire control might turn out to be totally unsuitable for a whole category of users. It is then reasonable to take a “safe” approach by adding extra options, in the form of events, method overloads, extra parameters, extensibility patterns, etc.

However, using this tactic for DSLs would lead us to create “flexible” implementations of a language through the introduction of (superfluous) configuration/customization options to our models, without considering that every time we add an extra element, attribute or connection to a DSL we arguably make the language more difficult to use and, ironically, less adaptable:

• It is more difficult from the user perspective because there are more elements in the language to understand and deal with.
• It is less resilient to change because those options might expose subtle aspects of design and implementation of a solution that, over time, would be harder to modify and improve.

Of course, since we want to use DSLs to solve categories of problems, we cannot totally eliminate all forms of variation, but my point is that, in order to add flexibility to a domain specific language, we should take a systematic approach and reduce options rather than blindly increase them!
Is it possible? Yes, if we narrow the scope of application of the language to a well-known domain and we favor abstractions and conventions over configuration options.

It is indeed a very remarkable proof of concept and although the implementation is still pretty rough, it represents a valuable model for discussion and learning.

Eamon is very active (particularly) within the Irish community and he is a regular speaker at the Irish .Net Developer Alliance (INDA) events where I often see him.
I briefly talked to him last night as we both were present at the excellent Roy Osherove’s talk on agile development and unit testing best practices.
Eamon told me that he only spent a couple of days developing it; this is something I consider rather impressive given the premature stage of the supporting tools currently available (I bet that his superb documentation took him more time to develop ).

The DSL designer currently allows setup and code generation for five patterns:

• Abstract Factory
• Builder
• Factory Method
• Prototype
• Singleton

Overall, I liked the organization of the code templates in the debug solution, as they are neatly located in a common folder and are referenced only through the ClipcodeEngineTransform.ReportTemplate file which acts as a template controller for all the instantiated designers.

Besides some understandable limitations of both the language and the underlying platform (i.e. the DSL Tools) that will be undoubtedly addressed in future releases, my main interest (and point of concern) lies in the level of abstraction of the model.

Using the language to generate code that realizes a pattern is pretty straightforward as the designer diligently constrains allowable classes and connections.
However, my worry is that, in order to realize a pattern, you need to select every single participant (class/object) from the toolbox and manually connect it to the permitted collaborator(s). As a consequence, you end up creating a diagram that matches quite closely the classic UML representation of each pattern. But do we really need to visualize the structure of a pattern?

As my exploration of the Domain-Specific Modeling (DSM) world progresses, I’m more and more obsessed by the idea that if we want to create the biggest ROI, it is essential to push our models towards a level of abstraction that is significantly higher than the level at which code statements and classes are written.
In other words, a one-to-one mapping between a class created in a diagram and a class generated in code does not generally save enough time and I would dare to consider it an anti-pattern.

A symptom is that the toolbox is generally filled with lots of different elements even if the main concepts (in this case the GoF patterns) are more coarse grained.

So here is a challenge that anyone interested can take. In order to realize an abstract factory, I should need to drag only one class on the designer (e.g. AbstractClassRealization) and setup its participants (AbstractFactory, ConcreteFactory, AbstractProduct, ConcreteProduct) using compartments only. Is it possible? I guess so. I will try soon anyways.

By the way, if you think about it, a GoF pattern is still at a pretty low level of abstraction. In fact, it is the very essence of an OO pattern to address recurring cross-domain concerns. So a “real” DSL may still be built on top of it and possibly provide an even higher return of investment for your organization.

After illustrating that domain specific languages are ways to improve effectiveness in communication, Jochen Seemann literally blew my mind by demonstrating what can be done today with the CTP release version of the upcoming Microsoft Domain-Specific Language (DSL) Tools.

This SDK allows you to easily build a graphical designer hosted by Visual Studio 2005 that is able to provide support for a custom domain-specific language.

What does it mean? Once the designer is built, a user can instantiate the designer, drag custom shapes from the toolbox, connect those shapes together (with validation support according to the language rules) and use the template engine to generate code/artifacts!

Building a DSL designer requires several steps, which basically involve the definition of:

• the elements of our domain in a domain model (classes and relationships)
• the notational elements (shapes and connectors) for the business entities
• the legal rules (validation/constraints) from a notational perspective, and the mapping of the notation to the elements of our domain.

Once the designer is built and instantiated we can use template-based generators to generate artifacts from our model (e.g. code, HTML, XML, WordML, etc). The templates have a notation similar to CodeSmith or classic ASP, and in the future some form of Intellisense (to query the model) is likely to be provided.

Some of these steps currently require the editing of an XML file. However Jochen said that this is really an “alpha” version of the toolkit and that better support will be provided in further iterations (indeed, the textual generator engine, code named T4, is only 6 weeks old!).
He plans to release incremental versions of the toolkit every 2-3 months, maybe even after version 1.0 will be shipped.

Currently I have only a few random observations/considerations on DSL designers that I want to dump here for my reference and eventually refine in future posts as I become more experienced in this field:

• We can build multiple designers to generate different parts in our systems, so we can (and probably should) start small.
• 2-Way synchronization (e.g. graphical model – to generated code – back to graphical model) is probably not going to be provided with version 1.0. It is a difficult problem to solve and I agree with Jochen’s pragmatism that it would be a fundamental feature only in a relatively small number of cases.
• Template/designer versioning and deployment are going to be difficult issues. We will need a defined structure and guidelines to avoid dealing with a total mess.
• A DSL designer is likely to work effectively in areas when the domain entities are very well understood. I can see how the use of a ubiquitous language would be core in this context.
• DSL designers could be used for requirements; however I have some reservation in this direction as the domain model is likely to be refined and changed way too quickly in a real world scenario. Use cases, CRC cards, and UML drafts are still likely to be my preferred option in the short term. I might be totally wrong on this, and I hope somebody can change my ideas soon (any volunteers?).
• There is a space for graphic tools and a space for code. A guy asked if Microsoft was going to provide basic notational elements that would give us very fine grained control capabilities (ifs, whiles etc). I think that writing code using shapes is insane. This is not a new idea (companies have been trying for years to propose solutions in this direction) but when you get pages and pages of icons you quickly realize that you are in the wrong space.
• Code generation driven by a designer is very cool. However writing templates is a pain and it feels like going back to messy scripting languages.
• Aspect Orientation could have a major role in this type of development. You add an attribute to a shape and (with the right template) your code magically gets to be generated. Jochen is not totally convinced by the feasibility of the AO approach as many people got burned. After talking to the guys in Avanade, however, I’ll be willing to take a serious shot in my problem space.
• It will be fundamental to maintain a clear separation between generated code and custom code. Use of partial classes that can be regenerated would be an option, separate classes and aggregation/inheritance another.
• I wish we could have several injection points in the designer and even a link to MSBuild, so that separate tasks could be executed upon code generation.

I hope this post gave you a sense of what this technology can provide. As I said at the beginning, I’ve been totally fascinated by it.
Finally let me just say that this technology gave me a fantastic idea that promises to provide an incredibly elegant solution to a huge problem in Innerworkings’ domain. Unfortunately I’m only teasing you guys because I can’t tell you what it is, as probably Fran (my CEO) would literally kill me if I did