Claudio Perrone's Monologues

In this post, Paul Gielens mentions the article ”Design and Implement a Software Factory” published with the new issue of the Architecture Journal magazine.

The article describes Microsoft’s progress in developing a proof of concept of a factory in the healthcare domain. It is definitely worth a read, particularly if you are (as I am) struggling to find a real, or at least realistic, example of a factory schema.

On a related note, you might have missed Ron Jacobs’ ArcTalk interview to Jack Greenfield and Mauro Regio on the very same subject; the interview serves as an introduction to the HL7 project and to the whole concept of a software factory.

Towards the end of the talk, Mauro gives some comments about the perceived loss of developer creativity that switching to a more “industrialized” process seem to imply. Well, I totally share his view that creativity is not gone at all, but simply repurposed to solve new and more interesting problems.

A while ago, I showed to my team an early prototype of a very simple domain-specific language that I wrote which will be part of a product line that we are building.

To be honest, there wasn’t much to it; after the quick demo, however, one of the previously skeptical developers looked at me with evident surprise and said:

“Claudio, now I finally get it! You are going to remove the boring parts of my job”.

Yes I am, buddy… yes I am .

Last week Jimmy Nilsson expressed his growing dislike for today’s overuse of XML, particularly when conceived as a surrogate programming language.

I really share Jimmy’s concern, especially when I think about the XML abuse I’ve been subjected to while creating languages and recipes using the Domain-Specific Language (DSL) Tools and the Guidance Automation Toolkit (GAT).

On the other hand, I recognize that it is totally unfair for me to complain; this is, after all, the price we pay for dealing with prerelease software.

In fact, the DSL Tools team already announced that will soon provide us with a modeling language able to deal with the .dsldd file and hide that ugly XML from our sensitive eyes.

I haven’t heard of announcements regarding similar improvements on the GAT side, but I choose to be optimistic and wait for new developments.

Unfortunately this problem does not affect prereleases exclusively; released software, both closed and open source, is not immune either.

Let’s face it, it is far too easy to define and process a language using XML for not taking advantage of it; in a not-that-distant future, however, I have no doubts that we will look back and laugh thinking about how we put up with manually dealing with its angle brackets notation on a daily basis for all these years.

Apart from really extreme cases such as XSLT, perhaps XML-based build languages of the likes of Ant, NAnt and MSBuild are among the most obvious and discussed examples of how a good idea can turn nasty when scenarios get a little more complex than anticipated.

Martin Fowler already suggested that today’s enterprise applications require pretty sophisticated builds which are better handled using a full-fledged programming language.

Last year, he also wrote an article about his explorations using Rake, a build language based on Ruby. In a nutshell, using an internal (text-based) domain-specific language such as Rake enables developers to easily switch to the external general-purpose language (Ruby) whenever necessary.

But how sophisticated can builds be? To put a little bit of perspective into things, let me tell you that for more than two years at InnerWorkings we have adopted a continuous integration solution based on CruiseControl. NET and NAnt; unless you’ve read Jimmy’s brilliant manuscript of his upcoming book (in which I was honored to make a small contribution as guest author) you will be surprised to hear that we manage to automatically and continuously build, obfuscate, package, test and deploy several hundred .NET solutions.

While I think that we have done more than a reasonable job at maintaining a good degree of modularity and reuse in our XML scripts, our tolerance to NAnt’s notation has been tested several times.

Resorting to a DSL written on top of a full programming language as Martin suggests would definitely help, but I have the impression that a purely text-based language may still be rather inadequate at solving one of the issues that we often face, particularly when dealing with many scripts in parallel like our scenario requires: lack of run-time visibility about the progress of each build.

Do I have a viable solution? I’m not sure, but I speculate that it won’t be long before we will see the raise of a new generation of continuous integration systems built around the upcoming Windows Workflow Foundation, with steps defined using domain-specific activities.

What do you think? Is it a daft idea?

Despite the fact that you probably haven’t noticed or cared, I want to apologize for my lack of posting in the last few months.

Sure, I’ve been extremely busy, but in many ways I feel like I’ve been a horrible member of our software community, unfairly learning without sharing my findings with everyone else.

So here I am again, promising that I will try to contribute more in the future with the hope to encourage others to do the same.

There are several things I feel I should write about (including the amazing software architecture workshop that took place last month in Cortina d’Ampezzo, Italy) but today I will just mention the more recent Microsoft Architect Forum 2006 which was superbly organized by Bill O’Brien here in Dublin.

I really couldn’t miss the full-day event since Beat Schwegler and Ingo Rammer were set to dig deep into a topic that I simply can’t afford to ignore these days: Software Factories .

Beat and Ingo were excellent in articulating Microsoft’s vision of how to combine model-driven development, guidance, frameworks and tools together with the objective of creating one or more product lines able to systematically exploit commonalities among the members of software product families .

Bill has kindly made available all the slides in this post so I won’t go into the details of each session.

During his analysis, Beat emphasized that we should consider that, in many cases, up to 70% of the cost of a software project goes into operations rather than pure development; as a consequence, he recommended that we should start adopting model-driven development not only for code generation, but also for requirements, deployment, configuration, and, more generally, for all other activities that are involved during the full lifecycle of a project.

While in principle I don’t disagree with this thought, today I find quite unlikely that different stakeholders (business analysts, network engineers, enterprise architects, solutions architects, developers, security specialists, QA testers, etc.) would accept to use the current incarnation of tools and designers and be confined to one single hosting environment, namely Visual Studio Team System.

But hey, in fairness we are talking about a medium-to-long term goal here, so I will surely change my mind on this point whenever Microsoft (or I ) will get there.

Ingo was really impeccable throughout the day and used several examples to illustrate the capabilities of the DSL tools. In one specific instance however, I could not help but notice that the version of the domain specific language that he used did not provide a particular option he needed for his demonstration; he then resorted to manually modify the generated code to accomplish his objectives. Tut-tut Ingo, you are not supposed to do that .

I know I know, it was just a demo, and I really sound way too fussy.

It would be good however if somebody out there explained that, in the real world:

• We obviously cannot modify code once generated since the DSL models will diligently overwrite everything at the next transformation; as a consequence:

  • Put a comment header in each template to explain that “This code has been generated by a tool…do no modify…etc.”
  • Do not put the generated code under source control. That code is a dispensable artifact. Put the designer file and the templates under source control instead.

• Modifying a template is clearly a better option, generally however:

  • You need to put it under source control
  • You need to deploy it in a centralized location so that it can be shared across different applications in the same family
  • A change in the template may not be done lightly as it could easily break all other existing solutions that use the same template.
  • You need to unambiguously identify the version in use (you may put a version number in the header of even resort to change file name if changes are substantial and break existing applications that use it)
  • A change in the template that breaks existing applications may trigger the beginning of a new product line, particularly if you can’t accomplish 100% code generation of a solution and you can’t afford to retrofit all the existing solutions.

• If we are unable to build systems that achieve 100% code generation (which happens if the degree of variation in a product family is not completely known):

  • We need careful guidance (patterns) to understand how to happily write manual code beside generated code. How do we create the extension points? What do you put in the underlying frameworks and what do you keep in a template? Who should make the changes in the first place? Is it up to us to rediscover these tricks over and over? Sure we can use base classes, template methods, etc. But no, the one-size-fits-all solution of using partial classes does not work all the time (wake up people, we often deal with xml files with little or no control on the reader…has anyone heard of the app.config file for example?)
  • There is an evident abstraction leak as developers need to appreciate the generated code as they are going to write beside it and even debug it if necessary (tip: keep templates “thin” by leveraging rich frameworks instead). By the way, the idea that the “smartest” people will write DSLs and templates while the others will just use them is flawed in my opinion, but I will definitely save an explanation for another post.

I’m exhausted already and I’m aware that this list is far from complete; and I haven’t even started talking about how I see we could version a domain specific language or an entire product line using the current tools. Or maybe I should attempt to figure out how I would assemble effective teams within this new paradigm. Has anybody discussed yet how to deal with developer’s natural resistance to model-driven development? How should we address their concerns of domain-restricted job specialization and the rightful dislike for anything that contains the word factory in it? Or has everybody agreed that this is not an issue?

As often happens, the real problem is that the tools are getting there, with their capabilities and limitations, and we really need to go beyond the simple APIs to be good at using them.

But perhaps it is unfair to ask a toolmaker to tell us out how to excel at using those tools.

After all Mozart wasn’t a piano maker, right ?