Popular @
Forum Controls
Spotlight Features
The Rich Engineering Heritage Behind Dependency Injection
Andrew McVeigh takes us on a tour of the rich heritage behind dependency injection, what it represents, and tells us why its here to stay.
NetBeans 6: Matisse Updates
NetBeans 6 delivers great updates to the Matisse GUI builder. Spend a few minutes with Roman Strobl and get an expert briefing on what's new and what has changed. (sponsored)
Introduction to Groovy Part 3
In this, the third and final installation of Andres' Introduction to Groovy series, you learn about how Groovy handles variable numbers of arguments, named parameters, currying, and more about Groovy operators. Including, some new operators.
Easier Custom Components with Swing Fuse
Swing Fuse (actually just Fuse), is a framework designed to make it easier to create your own custom desktop components. In this article, Daniel Spiewak shows you how to get started and provides sample source code you can download.
Benchmark Analysis: Guice vs Spring
Willam Louth shows how he uses JXInsight Probes to investigate probable performance issues with code bases that he is not familiar with. He also highlights possible pitfalls in creating a benchmark, as well as in the analysis of results.
InfoWorld vs. Developer.com: Worst Judge of the Year Award
At 8:58 AM on Jan 17, 2008, Geertjan wrote:
Fresh Jobs for Developers Post a job opportunity
In itself, these different results are not necessarily strange, of course. Different judges have different preferences, and so on. However, let's look a bit more closely at these awards. The award to CodeGear's JBuilder ( here ) seems (from everything one can read there) to be based on an article written in March 2007. And, which other IDEs was JBuilder compared to? In other words, who else was in the running for the prestigious award? Well, read the intro: "InfoWorld last did a head-to-head comparison of these products [Java IDEs] in March 2005, and since then, the IDEs have all undergone important changes. This time, I decided to examine the winner of that review (Borland JBuilder) plus the winners of InfoWorld’s Technology of the Year awards, in the Java IDE product category, for 2006 and 2007 — IBM Rational and Sun NetBeans, respectively."
So... JBuilder was compared to IBM Rational and NetBeans IDE, on a pretty flimsy basis, and, because it came out most favorably in the comparison, it gets to be the Java IDE of the past year ?
Similar questions can be asked of the Developer.com awards. Because, in that case, JBuilder wasn't in the running at all. And what about other IDEs, such as JDeveloper by Oracle?
I realize that a lot of these awards are given after comparing products that the broader community have nominated. Hence, if no one nominated JDeveloper, for example, then it isn't considered when judging who gets the award. But, surely there's something skewed with giving an award to a product when not all its competitors (especially when there are only a handful that can be called competitors, as in the IDE space) are candidated for the award?
So, I don't know who is the worst judge, InfoWorld or Developer.com. Or maybe their decision making process is their own affair and I shouldn't be questioning it. And, of course, I am in no way questioning the excellence of all the products that have received awards. Still, despite that, something smells off in these parts of the IT world.
10 replies so far (
Post your own)
Re: InfoWorld vs. Developer.com: Worst Judge of the Year Award
I think all these "Award of the Year" things have morphed from one thing into something else over the years. When it first started, "MotorTrend's Car of the Year" was meant to tell the magazine's readers what car the editors really thought was best. But then they discovered that the magazine got "co-branded" with whatever car they chose, and they got huge amounts of free advertizing ("2002 Camry - Voted MotorTrend's Car of the Year" in every Camry ad).So now I think anyone putting out one of these awards really cares if it's accurate, they just want a bit of free advertizing for themselves. The only exception I know of is Consumer Reports which really doesn't allow anyone to say "Consumer Reports Best Buy", and really has sued those who tried. And from that vacuum has emerged another magazine "Consumer's Digest", who's business model seems to be to award the "Consumer's Digest Best Buy" award to lots of things and hope people won't notice that it's not Consumer Reports.
Also, when I read reviews like these, I always get the feeling that the author just picked up the tool and learned just enough to write about it. Take this, for example:
> JBuilder has three different sets of code auditors and analyzers: the open-source PMD, Findbugs, and Borland’s own code-inspection tool. These work well together...
If he's used these tools to any extent, he's bound to have some real opinions on them. Maybe one of them he can't live without or one he finds hard to configure or has too many false positives or whatever.
It's as if someone's telling me "all these cars have trunks to store the luggage" without ever mentioning that one trunk is huge, another is tiny, a third can be opened remotely, and a fourth can be accessed from inside the car.
Re: InfoWorld vs. Developer.com: Worst Judge of the Year Award
A lot of these awards seem to be based on or, at least, influenced by, user votes.How can a user vote impartially about products that they've never tried out? In practice, people know one, possibly two products and will automatically put them in front. There may be a fantastic product with only a hundred users in Outer Mongolia. But we will never know because only 100 people voted for it and it was discarded early in the selection process!
In fact these prizes give us an insight into how many people from a given sample use each product. So the prize should be called: Southern Florida's Most Used IDE or Java Almanack Readers' Most Used Products
Re: InfoWorld vs. Developer.com: Worst Judge of the Year Award
Some awards are done by a select group of judges and some by voting. I believe in this case, InfoWorld was judges and Developer.com was voting. Both systems have their biases and faults, but in general I think the judging awards have a bit more credibility. Voting tends to reflect which product group can 'bring out the vote'.btw, I am helping organize the Eclipse Commmunity Awards and we use judges to select the product awards but voting for individuals.
Re: InfoWorld vs. Developer.com: Worst Judge of the Year Award
I normally avoid magazines like this, because I always wonder how an author can know anything about this stuff unless he or she spends pretty much all of the time of coding.So, who actually pages through those types of magazines? Print versions happening to be lying around in the lunch room and had nothing better to do?
Re: InfoWorld vs. Developer.com: Worst Judge of the Year Award
How interesting that Ian Skerrett, who is the "Director of Marketing with the Eclipse Foundation" (according to http://ianskerrett.blogspot.com/) should consider the InfoWorld award more legitimate than the ones from Developer.com. Let's think why that might be the case -- maybe because it lost horribly to NetBeans, in the Developer.com awards? Kind of obvious that you should take issue with that, but deceptive to pretend that your reason for disagreeing is because of the judged vs. voting difference. But, isn't it even more insulting to NOT EVEN BE CONSIDERED for the InfoWorld award?Re: InfoWorld vs. Developer.com: Worst Judge of the Year Award
tbh I don't give a much about these pseudo-awards.I know which products I like to use, I know which I dislike.
I view these awards as marketing-gibberish, which must be the reason why certain products fare so good in this kind of competitions but are used by almost nobody I know.
Re: InfoWorld vs. Developer.com: Worst Judge of the Year Award
Patrick,I think that is the main problem:
We have all tried a limited number of programs. We like some and dislike others. But no one knows all the products well enough to make a judgment.
This could only be done seriously by making a weighted list of criteria and checking each program to see whether it really did it or not and, if so, how user friendly and efficient each operation is, etc. This is an enormous task and I don't think it is done for many of these awards.
Ian
Re: InfoWorld vs. Developer.com: Worst Judge of the Year Award
I still wonder, how can they leave out a tool like "Eclipse" in some of the categories.Re: InfoWorld vs. Developer.com: Worst Judge of the Year Award
Harris,Seems like you are having a bad day. I was just responding to the original article and comment #2 on why the difference in results.
Eclipse has done very well winning awards, both voting and judges, so I don't feel insulted at all. NetBeans is winning lots of awards too; so is IBM and CodeGear. Congratulations to everyone.
Re: InfoWorld vs. Developer.com: Worst Judge of the Year Award
There are a thousand Hamlets in a thousand people's eyes