The Developer/Non-Developer Impedance Mismatch

Most software developers have probably heard of and even had experiences with the
object-relational impedance mismatch (often addressed with
ORM tools), the
object-XML impedance mismatch (often addressed with
OXM tools), and even the
developer-DBA impedance mismatch. I don’t believe that these impedance mismatches are as difficult as they are
sometimes made out to be, but for those wishing to mitigate them, we have tools such as
Java Persistence API implementations and
JDO implementations for dealing with the object-relational mismatch (and some of the developer-DBA impedance mismatch) and similarly have approaches such as
JAXB,
XMLBeans,
JiBX and
Apache Commons Digester for dealing with the object-XML mismatch (and .NET’s
LINQ deals with both
ORM and
OXM mismatches). At this point in my career, I believe the developer/non-developer impedance mismatch is perhaps the most frustrating impedance mismatch I have run into.


Although there are numerous tools and approaches for dealing with these other types of impedance mismatches, it seems we’re woefully short on similarly powerful tools, approaches, and
proven practices (my new preferred term for what I think "
best practices" was originally intended to mean) for dealing with the developer/non-developer mismatch. In this post, I look at some of the most common areas of developer/non-developer mismatch and speculate as to why they occur and what can be done to address these specific areas of developer/non-developer impedance mismatch.

Sadly, we have much less control over the developer/non-developer impedance mismatch than we do over object-relational or object-XML impedance mismatches. Although in general things like improved communication and education can help, these answers are not as tangible as the ones we’re used to for dealing with other types of software development impedance mismatches. As difficult as it is to deal with the developer/non-developer impedance mismatch, we must do so because there are numerous significant stakeholders in the software development process that are not necessarily developers (managers, clients, testers, customers, business analysts, sales people, and more). 


DRY Principle / Modularity

Almost to a fault, developers have generally adopted (at least in a theory if not always in practice) the DRY (
Don’t Repeat Yourself) principles coined in the often-referenced book
The Pragmatic Programmer: From Journeyman to Master. Although the term was coined in this 1999 book, the practice had been one that developers for decades had understood to some degree. Regardless of native spoken language or favorite programming language, developers today widely recognize virtues of some degree of DRY-ness. There may be
some debate as to what level this should be taken (I’ve seen it taken past the point of common sense), but most of us agree with the perils of repeated documentation at different levels of the software product or of repeated code (
copy-and-paste development).

Benjamin Denckla, in the post "
Faith in DRY; no hope for software," writes, "Today we produce software through a laborious, undisciplined process that combines the low quality work of many with the heroic high quality work of a few. … Not enough people believe in DRY and other good practices like it." I believe this is especially true when one considers the non-developers involved in a software development project. In my experience, the developers generally do see the value in some significant degree of DRY, but non-developer stakeholders see little value in DRY principles.

For purposes of this discussion, I’m including modularity in what I’m calling DRY practices. Most developers know there are numerous reasons to not copy-and-paste the same code into multiple places. For many decades, developers have known to place reusable code in methods, functions, modules, or other constructs that allow the same code to be used in multiple contexts and situations. It doesn’t take long for this to become second nature to the experienced developer. Sadly, many non-developers seem to not acknowledge the risks and problems associated with redundant information copied from place to place or believe these risks and problems are more theoretical than real. Although it may not be code we’re talking about when we discuss non-developers (it may be documentation, requirements, specifications, test procedures, or a host of other non-code things), the principle still applies: reproducing anything in multiple places leads to problems down the road in terms of maintenance and keeping the many versions synchronized with the latest and greatest. Developers seem to almost intrinsically "get it," but I see it less well received from many non-developers.

Readable and Maintainable Code

Many new software developers and even more non-developers do not recognize the value of code that is more readable and more maintainable. Perhaps the best experience a young developer can have is to maintain and reuse someone else’s code. Doing so helps a young developer to recognize the value of writing code as cleanly as possible. Because non-developers never really get this experience, it is not surprising that they don’t value cleanness, maintainability, and readability to the same degree as the experienced developer. Many legitimate cries for time to refactor a code base to improve its future maintainability and readability are ignored or promptly dismissed because such efforts’ value is not obvious to those making the decisions.


Overbearing Processes and Management Decisions

I have occasionally seen non-developers in management roles trying to coerce developers into very narrow and specific behaviors that they (the managers) believe is best (or worse, that they perceive as giving them the power). These folks rarely have the experience to know the full ramifications of their decisions. Good managers listen to their developers (particularly those with significant experience and in technical leadership roles) before pushing out every "good idea" they have. Experienced developers usually know what it takes to write high-quality software, but it almost takes another experienced software developer to appreciate what they argue for. Alternatively, a manager lacking software development experience can sometimes make better decisions by choosing an experienced developer that he or she trusts with technical decisions. The best non-technical managers recognize their own lack of technical knowledge and work with a trusted technical expert to make good decisions. 


Bean Counting and Pencil Pushing

Most software development is done as part of a business venture and it is often inevitable that some degree of
bean counting and
pencil-pushing will be required.. Clients or consumers directly or indirectly finance the creation of software. It can be difficult for developers to recognize and appreciate the legitimate management and metrics collection that goes on during these business-oriented phases. It can be equally difficult for managers and other non-developers to understand that some things are more subtle than the apparent "bottom line." Non-developers may over-emphasize short-term "bottom line" considerations at the expense of long-term quality and maintainability of the product while developers may overly neglect bottom line considerations and create software products that require too much time and investment to justify their likely return on investment.

The bean counter wants nothing more than to be able to count things like beans. He or she wants to use
lines of code, number of defects in various states, number of requirements, and so forth to feel like he or she has a handle on the software development progress being made. It
doesn’t really matter that these are not created equally and should not be counted as if they are equal. 


Battle for Control

It seems to be human nature and common in many relationships between humans to have battles for control. Tension can increase in the relationship between developers and non-developers as each group tries to exert control. Many non-developers, especially if they don’t understand development or coding well, resent not being able to control what is added to the baseline. Many developers resent being told what they can put into the baseline, especially when they strongly believe that the non-developer is making arbitrary calls while lacking sufficient knowledge and background to make that call.

The battle for control can become very onerous when both sides are "know-it-alls." When either side is convinced of its superiority, it can be very difficult to get either to budge. Developers often feel their experience and skillset best qualifies them for making all software decisions while clients, managers, and others often feel their position does the same for them. 


Coding: Job for a Craftsman or for a Technician?

Good software development managers recognize that software development can be a highly creative and challenging effort and requires skilled people who take pride in their work. Other not-so-good software managers consider software development to be a technician’s job. To them, the software developer is not much more than a typist who speaks a programming language. To these managers, a good enough set of requirements and high-level design should and can be implemented by the lowest paid software developers. Some software development is easier than other software development, but the simple technician work has been largely replaced by automation and code generation at this point.

Perhaps the perceptions of technician versus craftsman explain why non-developers tend to be more likely to believe that all developers are plug-and-play while experienced developers realize that there can be a wide disparity in skillsets and knowledge between any two developers. 


Appreciation of Software Development Nuances and Subtleties

It usually does not take long for a developer to realize that relatively little in software development is cut and dry. Software development often has a large amount of creativity to it and there are numerous judgment calls to be made. We sometimes call these design decisions or architecture trade-offs. Unfortunately, many who are not software developers do not understand that there are nuances and subtleties and even large amounts of creativity involved in software development. Without lack of these subtle shades, it’s not surprising that many of these people without development experience can only think in extremes (technique "A" is good and must be be used by everyone all of the time or technique "A" is always wrong and should be absolutely avoided no matter what). New developers often exhibit this trait as well, but experience usually teaches them to be more willing to judge approaches and techniques against particular contexts and reduce the amount of generalization and assumptions. 


Developers Aren’t So Different from Others, But Then They Are

The Mythical Man-Month is one of the
most often-quoted books in the areas of software development and
software development management. One of its great quotes is made early in the work (first sentence of Preface to the First Edition): "In many ways, managing a large computer programming project is like managing any other large undertaking—in more ways than most programmers believe. But in many other ways it is different—in more ways than most professional managers expect." One of the key explanations of the impedance mismatch between developers and non-developer managers seems to lie in this profound statement. Developers, as a group, probably should be more willing to buy into certain proven "traditional" management approaches, but managers need to avoid falling into the trap of thinking that the tactics outlined in the latest business management book will be sufficient for managing software developers. 


Opinions on 
What Values Most

Software developers are people too. As such, they do exhibit the same behaviors as other people. However, there are gross stereotypes of software developers that are not completely without some basis because of the high frequency of those stereotyped traits among software developers. There is great diversity in the software development community in terms of political opinions, interests, and so forth, but the idea of what is most important (quality design and code, work to be proud of, etc.) are fairly common across the industry. On the other hand, software developers (similarly to engineers in various engineering disciplines) seem to overly trivialize the need to respect the bottom line. They often cannot understand when an arguably adequate but not "best" or "perfect" solution is chosen over a better technical solution for non-technical reasons. 


Conclusion

We in the software development community tend to deal with mismatches all the time. We often spend significant energy and time "gluing" things together that weren’t necessarily designed to go together. Despite all of this technical experience we have making incongruent pieces work together, we still seem to have difficulty resolving perhaps the most difficult and most important mismatch of all: the
mismatch between software developers and people who are not software developers. Although there are some positives that come from this (such as checks-and-balances on "science fair projects"), there is significant dysfunction, angst, resentment, and demoralization caused by this impedance mismatch.


Reference:
The Developer/Non-Developer Impedance Mismatch from our
JCG partner Dustin Marx at the
Inspired by Actual Events blog.

Source : http://www.javacodegeeks.com/2012/05/developernon-developer-impedance.html

Advertisements

The Visitor Pattern Re-visited

The visitor pattern is one of the most overrated and yet underestimated patterns in object-oriented design. Overrated, because it is often chosen too quickly (
possibly by an architecture astronaut), and then bloats an otherwise very simple design, when added in the wrong way. Underestimated, because it can be very powerful, if you don’t follow the school-book example. Let’s have a look in detail.


Problem #1: The naming

Its biggest flaw (in my opinion) is its naming itself. The “visitor” pattern. When we google it, we most likely find ourselves on the
related Wikipedia article, showing funny images like this one:

Wikipedia Visitor Pattern example

Right. For the 98% of us thinking in wheels and engines and bodies in their every day software engineering work, this is immediately clear, because we know that the mechanic billing us several 1000$ for mending our car will first visit the wheels, then the engine, before eventually visiting our wallet and accepting our cash. If we’re unfortunate, he’ll also visit our wife while we’re at work, but she’ll never accept, that faithful soul.

But what about the 2% that solve other problems in their worklife? Like when we code complex data structures for E-Banking systems, stock exchange clients, intranet portals, etc. etc. Why not apply a visitor pattern to a truly hierarchical data structure? Like folders and files? (ok, not so complex after all)

OK, so we’ll “visit” folders and every folder is going to let its files “accept” a “visitor” and then we’ll let the visitor “visit” the files, too. What?? The car lets its parts accept the visitor and then let the visitor visit itself? The terms are misleading. They’re generic and good for the design pattern. But they will kill your real-life design, because no one thinks in terms of “accepting” and “visiting”, when in fact, you read/write/delete/modify your file system.

Problem #2: The polymorphism

This is the part that causes even more headache than the naming, when applied to the wrong situation. Why on earth does the visitor know everyone else? Why does the visitor need a method for every involved element in the hierarchy? Polymorphism and encapsulation claim that the implementation should be hidden behind an API. The API (of our data structure) probably implements the
composite pattern in some way, i.e. its parts inherit from a common interface. OK, of course, a wheel is not a car, neither is my wife a mechanic. But when we take the folder/file structure, aren’t they all java.util.File objects?

Understanding the problem

The actual problem is not the naming and horrible API verbosity of visiting code, but the mis-understanding of the pattern. It’s not a pattern that is best suited for visiting large and complex data structures with lots of objects of different types. It’s the pattern that is best suited for visiting simple data structures with few different types, but visiting them with hundreds of visitors. Take files and folders. That’s a simple data structure. You have two types. One can contain the other, both share some properties. Various visitors could be:

  • CalculateSizeVisitor
  • FindOldestFileVisitor
  • DeleteAllVisitor
  • FindFilesByContentVisitor
  • ScanForVirusesVisitor
  • … you name it

I still dislike the naming, but the pattern works perfectly in this paradigm.

So when is the visitor pattern “wrong”?

I’d like to give the
jOOQ QueryPart structure as an example. There are a great many of them, modelling various SQL query constructs, allowing
jOOQ to build and execute SQL queries of arbitrary complexity. Let’s name a few examples:

  • Condition
    • CombinedCondition
    • NotCondition
    • InCondition
    • BetweenCondition
  • Field
    • TableField
    • Function
    • AggregateFunction
    • BindValue
  • FieldList

There are many more. Each one of them must be able to perform two actions: render SQL and bind variables. That would make two visitors each one knowing more than… 40-50 types…? Maybe in the faraway future, jOOQ queries will be able to render JPQL or some other query type. That would make 3 visitors against 40-50 types. Clearly, here, the classic visitor pattern is a bad choice. But I still want to “visit” the QueryParts, delegating rendering and binding to lower levels of abstraction.

How to implement this, then?

It’s simple: Stick with the composite pattern! It allows you to add some API elements to your data structure, that everyone has to implement.

So by intuition, step 1 would be this

interface QueryPart {
  // Let the QueryPart return its SQL
  String getSQL();

  // Let the QueryPart bind variables to a prepared
  // statement, given the next bind index, returning
  // the last bind index
  int bind(PreparedStatement statement, int nextIndex);
}

With this API, we can easily abstract a SQL query and delegate the responsibilities to lower-level artefacts. A BetweenCondition for instance. It takes care of correctly ordering the parts of a [field] BETWEEN [lower] AND [upper] condition, rendering syntactically correct SQL, delegating parts of the tasks to its child-QueryParts:

class BetweenCondition {
  Field field;
  Field lower;
  Field upper;

  public String getSQL() {
    return field.getSQL() + ' between ' +
           lower.getSQL() + ' and ' +
           upper.getSQL();
  }

  public int bind(PreparedStatement statement, int nextIndex) {
    int result = nextIndex;

    result = field.bind(statement, result);
    result = lower.bind(statement, result);
    result = upper.bind(statement, result);

    return result;
  }
}

Whereas BindValue on the other hand, would mainly take care of variable binding

class BindValue {
  Object value;

  public String getSQL() {
    return '?';
  }

  public int bind(PreparedStatement statement, int nextIndex) {
    statement.setObject(nextIndex, value);
    return nextIndex + 1;
  }
}

Combined, we can now easily create conditions of this form: ? BETWEEN ? AND ?. When more QueryParts are implemented, we could also imagine things like MY_TABLE.MY_FIELD BETWEEN ? AND (SELECT ? FROM DUAL), when appropriate Field implementations are available. That’s what makes the composite pattern so powerful, a common API and many components encapsulating behaviour, delegating parts of the behaviour to sub-components.

Step 2 takes care of API evolution

The composite pattern that we’ve seen so far is pretty intuitive, and yet very powerful. But sooner or later, we will need more parameters, as we find out that we want to pass state from parent QueryParts to their children. For instance, we want to be able to inline some bind values for some clauses. Maybe, some SQL dialects do not allow bind values in the BETWEEN clause. How to handle that with the current API? Extend it, adding a “boolean inline” parameter? No! That’s one of the reasons why the visitor pattern was invented. To keep the API of the composite structure elements simple (they only have to implement “accept”). But in this case, much better than implementing a true visitor pattern is to replace parameters by a “context”:

interface QueryPart {
  // The QueryPart now renders its SQL to the context
  void toSQL(RenderContext context);

  // The QueryPart now binds its variables to the context
  void bind(BindContext context);
}

The above contexts would contain properties like these (setters and render methods return the context itself, to allow for method chaining):

interface RenderContext {
  // Whether we're inlining bind variables
  boolean inline();
  RenderContext inline(boolean inline);

  // Whether fields should be rendered as a field declaration
  // (as opposed to a field reference). This is used for aliased fields
  boolean declareFields();
  RenderContext declareFields(boolean declare);

  // Whether tables should be rendered as a table declaration
  // (as opposed to a table reference). This is used for aliased tables
  boolean declareTables();
  RenderContext declareTables(boolean declare);

  // Whether we should cast bind variables
  boolean cast();

  // Render methods
  RenderContext sql(String sql);
  RenderContext sql(char sql);
  RenderContext keyword(String keyword);
  RenderContext literal(String literal);

  // The context's 'visit' method
  RenderContext sql(QueryPart sql);
}

The same goes for the BindContext. As you can see, this API is quite extensible, new properties can be added, other common means of rendering SQL can be added, too. But the BetweenCondition does not have to surrender its encapsulated knowledge about how to render its SQL, and whether bind variables are allowed or not. It’ll keep that knowledge to itself:

class BetweenCondition {
  Field field;
  Field lower;
  Field upper;

  // The QueryPart now renders its SQL to the context
  public void toSQL(RenderContext context) {
    context.sql(field).keyword(' between ')
           .sql(lower).keyword(' and ')
           .sql(upper);
  }

  // The QueryPart now binds its variables to the context
  public void bind(BindContext context) {
    context.bind(field).bind(lower).bind(upper);
  }
}

Whereas BindValue on the other hand, would mainly take care of variable binding

class BindValue {
  Object value;

  public void toSQL(RenderContext context) {
    context.sql('?');
  }

  public void bind(BindContext context) {
    context.statement().setObject(context.nextIndex(), value);
  }
}


Conclusion: Name it Context-Pattern, not Visitor-Pattern

Be careful when jumping quickly to the visitor pattern. In many many cases, you’re going to bloat your design, making it utterly unreadable und difficult to debug. Here are the rules to remember, summed up:

  1. If you have many many visitors and a relatively simple data structure (few types), the visitor pattern is probably OK.
  2. If you have many many types and a relatively small set of visitors (few behaviours), the visitor pattern is overkill, stick with the composite pattern
  3. To allow for simple API evolution, design your composite objects to have methods taking a single context parameter.
  4. All of a sudden, you will find yourself with an “almost-visitor” pattern again, where context=visitor, “visit” and “accept”=”your proprietary method names

The “Context Pattern” is at the same time intuitive like the “Composite Pattern”, and powerful as the “Visitor Pattern”, combining the best of both worlds.


Reference:
The Visitor Pattern Re-visited from our
JCG partner Lukas Eder at the
JAVA, SQL, AND JOOQ blog.

Source : http://www.javacodegeeks.com/2012/05/visitor-pattern-re-visited.html