The paper Validation of an Approach for Improving Existing Measurement Frameworks by
Manoel G. Mendonça and Victor R. Basili reports their experience in validating and improving measurement frameworks used by companies.

In particular, they are mining the data gathered by the frameworks for interesting latent information using attribute focusing. Also, they are using GQM to structure the measuring activity.

The paper also contains links to several good background papers on software measurement techniques and their validation.

Jason Gorman, posted a critique on his blog about some code in the Crap4j codebase. He mentions one particularly long method, SystemCrapStats.toXml(). It was indeed quite a long method that was responsible for persisting the statistics gathered into the report.xml file. He asked the question:

Can I take it that Crap4J doesn’t test for long methods, then? Is this further evidence to support the theory that you get what you measure (or, in this case, you don’t get what you don’t measure)?

I answer, “Certainly.” Alberto expounded on our aims in his original blog posts over at Artima, but some of that bears repeating.

First, though, I would like to say that I fixed this method :-). It already delegated the project specific information and the method crap scores to their respective object’s toXml methods, but it was still doing quite a bit in this method. The new method is a composition of methods, that I think is much more comprehensible. It could certainly be even better, but we have other fish to fry and it is good enough. It looks like this:

  public String toXml() {
    MyStringBuilder s = new MyStringBuilder();
    s.start("“);
    crapProject.toXml(s);
    writeStats(s);
    writeMethods(s);
    s.end(”“);
    return s.toString();
  }

Back to our goal with Crap4j and a discussion of Jason’s question. We were looking for an unequivocal characterization of crappy code. That is, if the tool said the project is CRAP, then it had better be crappy. We believe, based on decades of experience, a lot of physical examination of code and interviewing of developers, and study and reasoning over piles of research papers that two simple measures can be pretty successful at indicating a whole class of crappy projects.

One of the properties is Cyclomatic complexity. Really complicated methods are hard to understand, thus hard to implement correctly, and probably more importantly hard to maintain and modify correctly.

The second property is the existence of automated unit tests. Tests serve to verify correctness and to catch regressions that might occur in maintenance. Without tests, the developer is left to reason through each change, and all of its implications for the rest of the code base. Tests extend the developer’s attention in a mechanical way and they show (when done reasonably well) a certain attention to correctness initially that may generally be relied on to increase confidence in the code.

Nowhere do we look at long methods. However, long methods can indeed be problematic. A straight-line method may still be hard to understand by virtue of the numerous responsibilities it has. If the definition and assignment of variables happens far from where they are used, that creates more opportunities for bugs by virtue of being more things that must be kept in the developer’s head while they work in that method. But, some long straight-line methods can arguably be pretty simple to understand. So, it is not as convincingly crappy as methods with many paths and no tests.

I would say that right after reducing the CRAPpy method count to a tolerable level, then by all means, pursue these other aspects of code quality. Look at things like long methods or the Chidamber Kemerer metrics such as cohesion and coupling. However, keep in mind that they are much more subjective due to their contextual interpretation. We don’t want CRAP to be subjective. If it says it is crappy, then everyone should agree.

That said, we expect to evolve the metric as everyone gets more experience with it. So, here is a question. Has anyone found disagreement with a particular method that Crap4j has identified as crappy? We would really appreciate knowing about it.

If you get the ‘Java started on first thread’ AWT exception running crap4j in Eclipse, here is a fix.

Inside <eclipse install dir>/Eclipse.app/Contents/MacOs/eclipse.ini, add the following two VM switches.

-Dapple.awt.usingSWT=false

-Djava.awt.headless=true

I ran into this for the first time tonight, and not sure what changed in my Eclipse 3.3.1 install. Everything still works fine in Eclipse 3.2.2. Anyway, those two switches solved the problem. Someone else mentioned removing the -XstartOnFirstThread flag, but only the combination of the two above actually worked for me.

Why could this happen? This is a perennial problem with Apple Java and Eclipse SWT. I had long been afraid it would happen since Crap4j renders the bar graph for ‘% CRAP’ using Java 2d, which loads in AWT. However, it hadn’t blown up on the Mac until today. I hope no one else will have the problem.

I just realized, via the netrenderer service, that our site looks pretty bad in IE 6 and IE 5, and that 10% of our visitors used those browsers. Yikes! I’ll be putting up a graceful degrade in the near future, but in the meantime, if you can, you might want to upgrade to IE 7. It seems like it has been out for over a year now!

Yesterday (October 29th) I gave my talk on crappy code at “The Business of Software” conference. Given the line up of other prestigious (and professional) speakers, the pressure was on but, apparently, the talk went quite well. Here’s what Bob Cramblitt had to say about it:

A Man Who Knows His Crap

Alberto Savoia of Agitar Software (www.agitar.com) knows crappy software, in every permutation. More than talking about it, his company is eradicating crappy software through better testing. Business of Software 2007 attendees were treated to a hilarious treatise on crappy software from Savoia. Those who experienced Alberto live won’t forget him anytime soon. For the rest of you, there’s the interview on this blog: http://blog.businessofsoftware.org/2007/09/alberto-savoia-.html

The talk was recorded on video-tape and I’ll try to get it posted on YouTube or on the crap4j.org website sometime soon.

After all the talks:

At the post-conference reception I had the opportunity to meet and talk CRAP with a bunch of interesting people. I met Scott Farquhar, one of the co-founders of Atlassian (a very cool and successful company doing work in software metrics) - it would be great to see the CRAP metric show up in their product.

At the end of the evening, I went to a (late) dinner with some of the other speakers and had an opportunity to meet spend some quality time talking about CRAP with Joel Spolsky and Eric Sink. After a full day of talking about software, I was actually more interested in talking to Joel about his cross-country bike trip, something I’d love to do myself one of these days. However, we ended up talking about the CRAP metric.

Joel is not a big fan of software metrics in general. He is concerned that developers might end up writing code and allocating their time to satisfy a specific metric rather than writing the best possible code and allocating the time based on more important criteria. He narrated a couple of stories about horrific metrics misuse that he witnessed first-hand and was concerned that - in the wrong hands - the CRAP metric could be used in, say, performance reviews: “You code is too crappy. You’re fired!”.

I understand that there is potential, as well as evidence, for software metrics misuse; but I don’t think that’s sufficient reason for avoiding metrics altogether. My reply to Joel was that if an organization/manager is so lazy and stupid to rely exclusively on any given code metric in evaluating programmers, then those programmers are probably better off being fired from that organization anyway. Better yet, the programmers would have great evidence to have the moronic manager fired.

While I understand that any tool, technology, or information can be abused by “evil” people and misused by stupid ones, I don’t think we should use “How could this be abused or misused?” as the primary criteria - at least not without first balancing the potential benefits.

Toward the end of the conversation, Eric Sink observed that the argument was starting to sound a lot like the perennial “Guns don’t kill people, people kill people” discussion. Great observation Eric.

On a more positive note, Joel did think that the CRAP metric could be useful when applied to existing code (i.e. post-facto) and said he might experiment with it.

I am not taking Joel’s concerns lightly; I respect his opinion and experience and I know that there are other people that feel as strongly as he does about software metrics. He has definitely given me something to think about - and blog about.

I met and talked with several other great software thinkers and practitioners besides Joel and Eric at this conference, and I will talk about them in future blogs.

In the meantime, thank you Joel and Eric for a stimulating late night debate. And Joel, the next time we meet let’s talk about biking cross-country before we start talking about software. All work and no play …

Alberto

Tomorrow I will be speaking at “The Business of Software” conference, along with some very well-known speakers like Guy Kawasaki and Joel Spolsky.The title of my talk is “Better Code: Recognizing, Avoiding, and Sanitizing Crappy Software”.I will be explaining the motivation and need for the CRAP metric and show-off Crap4j. So far most of the people who have been experimenting with Crap4J have been techies, tomorrow we’ll see how more business types respond to it. I’ll report back after the conference.Alberto

Professor Philip Johnson wrote about Crap4J in his blog. In addition having one of the best jobs I could think of (i.e. CS professor at the University of Hawaii), he runs the HackyStat project, a very well thought-out open source framework for automated collection and analysis of software engineering process and product metrics. Coincidentally, we started looking at HackyStat at the same time he started looking at Crap4J.

In his blog, Philip writes:

“Immediately after sending this link to the Hackystat Hackers, a few of us started playing with it. While the metric seems intuitively appealing (and requires one to use lots of bad puns when reporting on the results), its implementation as an Eclipse plugin is quite limiting.”

Yes, the Eclipse plug-in is quite limiting - especially if you don’t use Eclipse :-). But it looks like some HackyStat developers are going to take the metric, make it work with HackyStat, and spend some time studying it.

Dr. Johnson even hinted at a Master Thesis on the CRAP metric:

“Maybe I can get a student next semester to work on a crappy master’s thesis.”

Sounds like a great idea to me. I’d almost consider going back to school to work on it - especially if it’s in Hawaii.

Alberto

I just posted an entry on Crap4j in my blog on O’Reilly’s “Beautiful Code” website.

Here’s an excerpt from the blog entry’s introduction:

“If I had to write a tool that would attempt to decide if a particular piece of code could be considered beautiful, I would not know where to start. Detecting crappy code, however, seemed more within reach; so we threw caution to the wind and we gave it a shot. Read on and you can decide if that shot would have been better used on something else (like our foot).”

The original blog on Crap4j on Artima resulted in quite a few comments. I am curious to see if we get a similar reaction on Beautiful Code.

If interest in the C.R.A.P. metric and Crap4j continues to grow, we might write a companion book to Beautiful Code. Here’s an idea for the cover page inspired by the original:

Alberto