Jason Gorman has a thoughtful reply on his blog about using Crap4j.

He also brings up the potential for self-interest due to the fact that I work at Agitar.

“I think it’s worth highlighting the fact that Bob works at a company that sells a tool designed to help teams improve their test coverage, so I can’t help wondering if there isn’t a little bias here towards metrics that highlight lack of coverage. Which is fine, because Bob’s absolutely correct in saying that complex methods with low test assurance are pretty much universally crappy. It is a problem, and you should address it.”

I’d like to respond to that part first. Causality is always tricky. It turns out that I work at Agitar because I think these things are important. Agitar happens to be a great way to work on code quality and testing issues which are near to my heart. We invented the CRAP metric and Crap4j as ways to point out particular code quality problems. Naturally, as opposed to being cynically profit-driven, we do also offer tools that try to address these problems. Being an engineer, I don’t think I could bring myself to espouse any methodology or ideas just for my material gain. I have to believe in it. Of course, if anyone wants to offer me millions of dollars to espouse some helpful technology, I will be glad to contemplate the dilemma

Now, on to his point.

“Getting developers started with quantitative code analysis - metrics - is one of the things I do for a living, and I’ve always found the opposite. You need to try to establish balance right from the start. Focusing in on just one area of quality, no matter how valuable by itself, can lead to people ignoring other important areas.”

It is a point we discussed when trying to identify the metric. It is a valid point, and I am glad there are people like Jason working to help teams get a balanced picture of their systems. I hope eventually that the CRAP metric, or at least the tool, will do sort of a triage of code quality issues. It will present the most egregious metric failures first, and then offer more refined levels of CRAPpiness for the developer to ponder as they clear away the more immediate issues. Imagine a video game where you have to beat the level 1 boss to move to level 2.

This raises an issue that I haven’t spent time analyzing yet, but intuitively I know will present itself. There isn’t a hierarchical order between all the problems that can exist in a project, so, at some point you have to give them multiple, equally crappy, problems in a report. Perhaps this is like being able to get to a certain point in a video game where you can take multiple paths, but ultimately you have to finish the breadth of those paths to move on to the next level?

This is in tension with our other goal of giving teams a measure that is easy to act to remedy. It should be obvious what problem is being presented, and what can be done to fix it. Otherwise, I imagine that there could be paralysis, and consequently, no action taken to improve quality.

There is a severe lack of perspective around metrics, maybe even Crap4j. Sure, it has a funny name, and it was created by two really cool guys, but is it actually helping projects recognize problems and pointing the way to action that improves the success of those projects? Maybe, if we all shared our metric scores and categorized our projects in common ways we could get the necessary data to decide if metrics are useless or valuable. This would require some extra metrics like bug counts and development maintenance effort to compare metric scores against. Until that happens, at the very least, we will be able to see how we benchmark with our peers on a metric. We won’t know which competitor sucks more (or less), but we’ll know where we stand in our field!

Tags! You’re It!

Starting with the 1.1.6 release of Crap4j, we have added tagging to our benchmarking site, Crap-o-rama. This means you can compare your CRAP scores against similar projects by comparing against like-tagged projects! And it can still be done anonymously!

This feature set is just getting started and will become valuable as the number of projects grows. But it is easy and anonymous to contribute scores, so, please, share. We hope we will find out which metrics are meaningful for which categories of projects, and which metrics are categorically useless. We are starting with the CRAP metric, since that is the metric we currently produce, but we hope to add more in the future by opening it up as a web service.

What’s in a Name?

We chose tags as a mechanism for categorizing projects because it is completely free-form. The community can create the tags they find useful. It will be interesting to see how far we can take tagging, and what, if any, other classification mechanisms we will need.

I can see some obvious problems when it comes to updating a project’s CRAP score across releases. If I had a tag “v1.0”, and then changed it to “v2.0” with a new CRAP score, I will have lost the comparison against other version 1.0 projects.

Conceivably, I could write something that digs into our historical data for projects (used in presenting historical trending of CRAP scores) and find tags that were applied to a previous version of a project (like “v1.0”). That probably won’t scale well in the current implementation. Another option might be to create a new project for the next version, but share a tag on both uploads that is unique to the two of them; perhaps the project name, like “crap4j”. Then you could compare your version 1 project to your version 2 project as two separate projects. Of course, the historical trending charts wouldn’t exist across the two projects. Hmm. Have to think about this. Ideas?

Barring temporal tags, there are a lot of other useful comparisons that can be captured with tags. I’d like to discuss some of them to seed ideas for people who want to share and benchmark projects with their community. Maybe each community will come up with its own specialized set of tags that allow them to compare projects. Anyway, here are some tags that seem useful. Please provide your own and why you think they would be useful.

Examples

“java” — OK, this one is redundant right now since we are the only tool that uploads projects to Crap-o-rama and we only look at Java projects. But I know there are ports out there. They should be able to compare too. Crap4j is a language-neutral metric. So, we should apply this tag in preparation for the future. It might even show how similar apps in different languages compare (let the flame wars begin!)

“developer tool”, “financial app”, “game” — If you are writing tools for developers, it would be cool to compare against other developer tools. This is really to introduce the notion of vertical categorization tags. I happen to know that developer tools exhibit certain common types of complexity and other types of application may not exhibit the same range of CRAP scores. For example, dev tools usually do a lot of visitor patterns and switch statements, which produce particular CRAP scores. I expect other verticals may experience similar patterns.

“small team”, “5 developers”, “2 QA” — A team description might be useful to compare your CRAP score against other teams output. Again, this one is problematic because it may change over time. Leads me to think that searching historical project data, including the then-current set of tags might really be necessary. Any ideas?

“6 months time”, “1 weekends time” — Capture vaguely the amount of work involved. Again this one applies to versions so is transient.

“Waterfall methodology”, “XP method”, “Death march method” — The idea here is to find out if a methodology produces CRAPpier code than another.

“library”, “application”, “tool”, “service” — Another categorization for projects that probably have different architectural styles. For that matter, architectural styles could be labels as well. Then developers could ask, “How do I compare against other pipes-and-filters systems, or enterprise bus systems?”

We might even start to understand how different types of applications compare to each other.

These are a few ideas, I am sure that you, dear reader, have great ideas I haven’t even imagined. Please share.

Version 1.1.5 of Crap4j is available at http://www.crap4j.org/downloads/.

It has some bug fixes, and a really exciting new feature or two.

The big feature is that you can share your CRAP results anonymously (or publicly) and see how you fare against all the other projects uploaded. It even puts the global average in the report graphic to make it more useful. (You can turn this off if it’s just too painful

You can see stats on other projects on the benchmark site.

We hope that this will help us understand the CRAP metric better in the wild, and that it will provide users an opportunity to check themselves against fellow developers.

Enjoy, and please let us know how you like it.

There’s a new version of Crap4j available. The big feature is Ant support via a new Crap4j Ant task.

This won’t mean as much if you are using the Eclipse plugin, but if you want to incorporate the Crap score into your regular build, or if you want more control over the classpath setup for your project, this will be a way to achieve it.

For the Eclipse users, you can just update the existing version via the Update Manager.

For the Ant version, there is a different distribution (sometime soon I will make it one download.) You can get the zip file at http://www.crap4j.org/downloads/crap4j_ant_latest.jar.

See the full usage instructions on the Ant Usage Page.

We hope you find it useful, and please let us know how it works for you.

Mike Slinn suggests annotations to exclude methods that you don’t want contributing to the Crap score. I think some form of exclusion is definitely a good idea. I am curious what type would work best.

I can think of at least 4 ways to get various levels of exclusion:

• Add annotations that can affect the method. (May also want these at the class level or beyond)
• For class-level exclusion, use the about-to-be-released Ant task, which can do file excludes and includes.
• Have an exclude list that takes patterns like ‘MyClass#myMethod(Ljava/lang/Object;):Void’
• Use different compilation output directories for classes that do not ship. This is the way many people manage their test suites.

I am thinking that annotations are the cleanest, at least for the method level. It seems like the approach chosen depends on how many exclusions one wants to do. The annotations could be spread all over the code and troublesome to maintain, but on the other hand they are great for a small number of exclusions.

What do you think? Any altogether different way to do it? If we went with annotations, what is a good name. I find Mike’s suggestions amusing.

Greetings. We have released a new dot version of crap4j! You can get it on the downloads page.

We have fixed some bugs, and cleaned up some other things in preparation for new features. In particular, we fixed a problem with duplicate results, and some cases where it didn’t return properly from systems that use annotations. Also, the icon and menu items (popup and menu bar) are now enabled properly, and only when it is a java project that is selected. That should prevent some confusion.

We appreciate everyone’s bug reports and feedback. Thank you and we hope the new version works even better for you! (If not, please let us know or file a bug.)