Academia

Take a Stand for Double-Blind Reviewing!

5 Comments

I have long been a proponent of double-blind reviewing. People suffer from expectation bias, and double-blind reviewing is a tried and true approach to combat it. I adopted double-blind reviewing when I co-chaired VEE 2010 and just recently for WoDet 3, and have decided to take a stand to sway more program committees to implement it. Join me!

When asked to serve on a PC, agree only if double-blind reviewing is used.

This approach doesn’t always work, but the fact is that most program chairs simply had not considered it and are happy to adopt it. My advisor Kathryn McKinley‘s case for double-blind and Mike Hicks‘ fantastic FAQ on the topic make excellent ammunition. I suggested it to Todd Mowry and he implemented it for ASPLOS 2011; P. Sadayappan did the same for PPoPP 2012 (I am grateful to both for their patience!)

But there has been some backsliding; double-blind reviewing is not going to be used for POPL 2013, despite the overwhelmingly positive response of the POPL 2012 committee members.

So the next time you get asked to serve on a PC, at least bring it up. Let’s help make this a standard practice across our community.

Advertisement

5 thoughts on “Take a Stand for Double-Blind Reviewing!”

  1. One problem with double-blind (rather than single-blind) is that it prohibits people from posting their papers in their web pages and/or arXiv once the first good draft is done.

    Sometimes I feel like physics and mathematics manage to move faster than computer science, ironically enough, because they embrace these technologies fuller.

    1. Actually, this is not at all required by double-blind submission.

      Mike addresses this concern in his FAQ:

      Q: Am I allowed to post my (non-blinded) paper on my web page? Can I advertise the unblinded version of my paper on mailing lists or send it to colleagues? May I give a talk about my work while it is under review?

      A: As far as the authors’ publicity actions are concerned, a paper under double-blind review is largely the same as a paper under regular (single-blind) review. Double-blind reviewing should not hinder the usual communication of results.

      That said, we do ask that you not attempt to deliberately subvert the double-blind reviewing process by announcing the names of the authors of your paper to the potential reviewers of your paper. It is difficult to define exactly what counts as “subversion” here, but some blatant examples include: sending individual e-mail to members of the PC or ERC about your work (unless they are conflicted out anyway), or posting mail to a major mailing list (e.g. TYPES) announcing your paper. On the other hand, it is perfectly fine, for example, to visit other institutions and give talks about your work, to present your submitted work during job interviews, to present your work at professional meetings (e.g. Dagstuhl), or to post your work on your web page. PC/ERC members will not be asked to recuse themselves from reviewing your paper unless they feel you have gone out of your way to advertise your authorship information to them. If you’re not sure about what constitutes “going out of your way”, please consult directly with the Program Chair.

  2. Interesting post. I am generally in favor of double-blind reviewing (though I have flip-flopped at least a few times on the issue).

    I am troubled, however, by the recent(?) trend to refuse to serve on a PC unless the conference changes . If there are a lot of ‘s, then the community is deeply hurt by not getting the best PC and overworking more flexible members.

    If this is something you feel very deeply about, sure. But, it’s also important to realize that there can be legitimate differences of opinion over an issue and work to change those opinions in less harmful ways.

    1. I am not sure what other reasons people may have for refusing to serve; I do feel strongly about this particular topic.

      But as I point out in my post, even just bringing up double-blind is often enough to get the PC to adopt it. If enough of us do at least that, I believe we will change our communities for the better.

Comments are closed.