I am knee deep in INFOCOM reviews and like most of the TPC (Technical Program Committee), I have of course been a really good reviewer and done all of my reviews weeks in advance. Or not. On the plus side, I am on read number two after the initial skim where my trusty red or blue pen gets a work out. 6k+ characters for the last review is none too shabby though it does help when I have done work in that exact particular area. Anonymity fail perhaps with that one.
On a side note, I think I got blessed by the random paper assignment overlords this time. I have to say that I am thoroughly enjoying all of the papers in my stack. Does not mean that I am giving out all accepts but well done overall. Though I think I have a huge sample of TPC area leads. Huge kudos for whomever cut out of the rank this paper field from prior years. That was always tedious to go back to all of the reviews and rank them once you got through the pile.
Anyway, paging through the reviews, one particular thought struck me. Given that systems / networking research tends to view itself as being a bit of a "special flower" where journals are optional, does that mean that I as the reviewer for a top tier conference need to raise the bar and require additional completeness in the paper?
For example, suppose a paper is an area where the author (senior researcher) has published for 3+ years. Said author submits a paper in that same area which reads tremendously well but is fairly incremental. Should there any consideration for me as the reviewer to note that the author does not go the journal route and hence is unlikely to ever really "complete" the work if it does not end up in a thesis?
The simple answer is that no, I don't think it would be fair for me to take that into consideration as I think that would impose an unexpected burden on the author. The unforeseen burden is especially keen given that nearly all papers are driven by the students and it is definitely unfair to weigh the student with the prior behavior of the adviser (though for some reason, the converse never applies but never mind).
The more nuanced answer is that I think this is something that we as a community (networking / systems) need to deal with. I think we have long ago passed the threshold where one can get by doing "one off" papers that might have some justification for a more conference-centric view. I don't think a reviewer can do this either as that is bound to end up disastrous. We already have enough subjective evaluation, let's not add another one to the mix. It is rather something to discuss at the community level, say a TCCC or other level and I think it has to be a community ethic sort of thing.
Particularly for those who care deeply that conferences are more important than journals, I think there has to be a serious answer as to when the work gets "complete." Otherwise, it becomes yet another item to add to criticisms of why CS will have difficulty supporting the conferences-only notion for promotion and tenure. In a later posting, I'll add my thoughts on the article by my department chair (Kevin Bowyer) on conferences versus journals and how a junior faculty should view them. If you have not garnered by above, I am largely in agreement and the above is yet another reason why I think he is right for promotion / tenure.
Note: For anyone reading this if you think that I did this to your INFOCOM paper, no I did not and you are happy to drop me a note for me if you want to break the veil of anonymity. Whoot, go, go, gadget tenure :)