Journals vs. Conferences

I am knee deep in INFOCOM reviews and like most of the TPC (Technical Program Committee), I have of course been a really good reviewer and done all of my reviews weeks in advance.  Or not.  On the plus side, I am on read number two after the initial skim where my trusty red or blue pen gets a work out.  6k+ characters for the last review is none too shabby though it does help when I have done work in that exact particular area.  Anonymity fail perhaps with that one. 

On a side note, I think I got blessed by the random paper assignment overlords this time.  I have to say that I am thoroughly enjoying all of the papers in my stack.  Does not mean that I am giving out all accepts but well done overall.  Though I think I have a huge sample of TPC area leads.  Huge kudos for whomever cut out of the rank this paper field from prior years.  That was always tedious to go back to all of the reviews and rank them once you got through the pile.

Anyway, paging through the reviews, one particular thought struck me.  Given that systems / networking research tends to view itself as being a bit of a "special flower" where journals are optional, does that mean that I as the reviewer for a top tier conference need to raise the bar and require additional completeness in the paper?

For example, suppose a paper is an area where the author (senior researcher) has published for 3+ years.  Said author submits a paper in that same area which reads tremendously well but is fairly incremental.  Should there any consideration for me as the reviewer to note that the author does not go the journal route and hence is unlikely to ever really "complete" the work if it does not end up in a thesis?

The simple answer is that no, I don't think it would be fair for me to take that into consideration as I think that would impose an unexpected burden on the author.  The unforeseen burden is especially keen given that nearly all papers are driven by the students and it is definitely unfair to weigh the student with the prior behavior of the adviser (though for some reason, the converse never applies but never mind). 

The more nuanced answer is that I think this is something that we as a community (networking / systems) need to deal with.  I think we have long ago passed the threshold where one can get by doing "one off" papers that might have some justification for a more conference-centric view.  I don't think a reviewer can do this either as that is bound to end up disastrous.  We already have enough subjective evaluation, let's not add another one to the mix.  It is rather something to discuss at the community level, say a TCCC or other level and I think it has to be a community ethic sort of thing.  

Particularly for those who care deeply that conferences are more important than journals, I think there has to be a serious answer as to when the work gets "complete."  Otherwise, it becomes yet another item to add to criticisms of why CS will have difficulty supporting the conferences-only notion for promotion and tenure.  In a later posting, I'll add my thoughts on the article by my department chair (Kevin Bowyer) on conferences versus journals and how a junior faculty should view them.  If you have not garnered by above, I am largely in agreement and the above is yet another reason why I think he is right for promotion / tenure.  
 
Note: For anyone reading this if you think that I did this to your INFOCOM paper, no I did not and you are happy to drop me a note for me if you want to break the veil of anonymity.  Whoot, go, go, gadget tenure :)   

Visit - Sprint at the Wireless Institute

One of the nice perks of being at Notre Dame is the wicked, cool set of speakers that we get to have the privilege of listening to.  For this past week, we were the host of Bob Azzi, Sprint's senior vice president of network operations.  Unfortunately due to obligations related my Associate Chair hat, I had to duck out early from the question and answer session and missed the second half of the session.

A few interesting bits from the talk and Q&A that stood out for me:

• A certain other cell network when they had employed data caps expecting to see revenue generation.  In reality, users heavily capped their behavior when they hit the limit and it ended up not generating any substantial new revenue.  It definitely raises some interesting questions related to our WiFi offloading study as a comment was raised whether or not users will modify their behaviors as they get closer to the cap.  With our Cell Phone Study getting our data services from Sprint, this was not an issue due to the unlimited data plans.  It certainly would have some interesting effects for non-Sprint customers if we could monitor several in our study.  From a larger perspective, I have to think data like that would give execs at Verizon pause as the whole shared data plan sort of thing could get ugly in a heartbeat. 
• DPI (Deep Packet Inspection) and video translation for the win.  18-20% reduction by whacking certain low hanging video streams from 1080p when the type of mobile device is known (non-tethered too I assume) to be more appropriate for the screen size.  I'll have to do a bit more digging to see what devices are involved but very smart plays to re-encode said video to alleviate the last mile. 
• As academics in networking, we sometimes can forget that the real network is messy and that any sort of pico / metro cell magic is going to be awful due to the sheer scope / complexity of rolling it out for real.  Techniques like SON (Self-Optimizing Networks) are going to be essential for realizing HetNets.  Right of way / property are powerful things indeed (i.e. how do power / cable / get permission to install a pico or metro cell).
• Similarly, cell networks are not nearly as monolithic as one would imagine, often involving many agreements across a variety of networks to be able to provide full nation-wide coverage.  It definitely adds a unique complexity wrinkle to deployment that the community needs to be more mindful of.
• Finally, like all things technology, technology is only one part of it and policy, particularly public policy is a very, very important part.  We really need more engineers who can correctly inform technology decisions.

Hopefully we will be able to post up a video on the Wirelss Institute webpage in the next week or so with a recording of the core part of Azzi's talk.  

Android Layout XML - Oh my

Despite the fact that my schedule by in large is fairly awful, I have managed to set aside at least some time each week to do a bit of research / development.  I'll avoid posting which time that is lest someone try to schedule a meeting during that block but the project this week has been to look at finally doing a bit of Android development, particularly on my recently replaced iMac and the Nexus 7.  

The process of getting everything set up was not bad, a few minor glitches here and there, but nothing too severe.  Nice to see that is the case.

Beyond getting the basic Hello World example working, my first project is to try to get a Bluetooth Serial Port Profile (SPP) listener going for the purposes of instrumenting my eBike.  The eBike or really electric motor scooter (it does 60 mph, whoot) was my sanity prize for doing our departmental accreditation efforts.  That and the family vacation to Hawaii last year.  Even then, I am not sure it was enough but lest I dwell too much on accreditation and become grumpy, I'll get back to the task at hand.

Basically, the prototype variant that I had does not have the new fancy display that the current bikes ship with but it does have a serial TTL output.  I found a fairly neat Serial TTL to Bluetooth converter at MDFly and got it rigged up late last year.  At the time, I had been using a simple Serial TTL / USB cable and a bit of C# code on my notebook.  Unfortunately, the bike reared its prototype / test pilot nature this year and has largely been out of action, not allowing me significant riding time until this past week.  Current Motor has been wonderful about troubleshooting and hopefully the wonder that was troubleshooting the bad BMS (Battery Management System) led to better troubleshooting for other riders.

Now that my bike is back, I had been occasionally using a simple Bluetooth SPP application from the Play Store.  However, that is not a whole lot of fun as it does not give me GPS or a display for my bike.  I could also just settle for the simple SparkFun adapter to log things to a microSD but again, where is the fun of not having a full display :)  

The task then this week was to actually build a real Android app together with a dash of Bluetooth.  Not too bad on the core of the Bluetooth (Google's documentation is pretty sweet) but holy lord of obfuscation, the layout code is abysmal.  Granted, I am a bit biased from working with WPF and the C# side of things but wow, it is pretty awful.  I thought WPF had some awful kludges (properties and the magic XAML compilation behind the scenes) but Android layout definitely takes the cake.  I am hoping the awful obfuscation is just a byproduct of the fact that I am using sample code but I am not terribly optimistic.  

If anyone would like to educate me why I am off base and espouse the merits of Android's layout XML over WPF / XAML, feel free to do so.    

SIGCOMM CellNet

As always, a bit late on the blog post but better late than never with regards to discussing various papers / highlights from SIGCOMM as well as the workshop that I attended, CellNet.

• Positively wonderful opening note / tutorial.  The organizers (Li and Morley) asked Zoltán Turányi from Ericsson to give a nice over of emerging problems in cellular networks.  It was a nice twist that I think served as a good foundation for the rest of the day.  I am not sure this is viable at a general conference (versus say a key note) but it worked well for the workshop.
• I was a bit less sanguine about the Open Radio work by Sachin Katta from Stanford.  They had a wicked cool paper at the core SIGCOMM conference but it felt a bit like cognitive radio redux.  That being said, pulling off full speed 802.11n with TI + USRP goodness is pretty awesome and I have to give them serious props for that.  We will certainly have to do a bit of looking into the TI C66x DSP that their group had been using.

With that, we had a small break for coffee though we were by far running late.  It was then that we realized that HotSDN had 180 attendees (wow!).  The good news was that we got back on time in fairly short order for the workshop.

• Neat paper on Buffer Bloat presented by Sue Moon of KAIST who was subbing for Rhee from NC State.  Bummer was that Sprint was by far one of the worst violators of Buffer Bloat.  Evidently they have an upcoming IMC paper examining tweaks to TCP to try to improve things. Amusing anecdote of noting that AQM (Active Queue Management) is doomed out of the gate which I think is pretty much the community consensus.
• Two papers by AT&T on the core of their network primarily from a modeling perspective.  Both were invited and some neat graphs on RNC / tower performance with regards to TCP performance.

Nice standard conference lunch though I must say, the Finlandia lunches all week were excellent.  Got a bit crowded with the whole conference there on Tuesday through Thursday.  After lunch, it was time for our talk on WiFi offloading which I can now freely discuss on the blog now that it has finally dropped.  We ended up trading with the Stanford folks so KK's adviser could catch his talk.

• Needed more coffee for my talk and I realized I cut perhaps one too many slides in the interest of time.  Post lunch + jet lag means a far less than impressive talk.  Evidently Shu (my student) taped it but I'm not sure I will be adding it to YouTube any time soon.
• KK Yap had a nice bit about using multiple adapters to get better performance.  They used an underlying Linux virtual switch on Android to allow one to fuse together multiple adapters and derive better performance.  He did a nice job presenting his talk.
• Neat work on multi-path TCP definitely getting into the firm weeds, sort of a different perspective taking the standards route for multi-path TCP rather than the bit more of ad hoc approach of the Stanford talk.  
• The session wrapped up with a talk by Suman Banerjee about their wireless bus work.  Very cool and some interesting enterprise level problems when you have numerous buses / data plans to manage.  Definitely problems that most academics are not pondering but are likely to be quite important at a macro level.  Very impressive with how far the bus work has come from several years back when he had presented it visiting us at ND.  Well done guys.

Two final talks wrapped things up with one on modeling (not necessarily my cup of tea) and one on assisted GPS, namely how bad the assisted GPS has been implemented on many devices.  Lots of low hanging fruit for vendors to fix.  The modeling issue brought up an interesting point that with 3GPP and the potential to signal back to the base station, how much should we signal back?

The workshop wrapped up with a panel discussing various issues / open items for cellular networks.  The poor panelists barely got in their slides before we started discussing each particular person's results.  The net result was that Li did not get a huge amount of time to go over his thoughts on a cellular SDN which was a bit of a bummer but the back and forth on the panel was excellent.

All in all, it was a nice workshop.  Quite well attended (though not bursting like HotSDN).  Well done Li and Morley!

Reviews / TPCs

I'll sneak in a semi-related post with regards to TPCs (Technical Program Committee) that relates a bit to McKeown's comment about making the SIGCOMM tent bigger, not as big as SIGGRAPH but about four times as big attendee-wise as it is now.

Having recently gotten back a few sub-par sets of reviews, I am feeling a storm of grouchiness with regards to the overall review process and the on-going conference versus journal debate.  Note that this is in no way related to SIGCOMM as it has been a few years since I have submitted a paper there.  We are much better at self-filtering and as of late, most of our work has not been terribly conducive to SIGCOMM either.  Though my top negative review of all time from SIGCOMM still holds a place near and dear in my heart :)

On to my thoughts / recommendations for TPCs / reviewers:

• I had posted on this a few year's back but I think it is still the case.  Conference papers are for interesting / thought-provoking work, not necessarily perfect nor complete work.  We had a workshop review two years ago for a paper where the reviewers were asking for essentially a journal quality evaluation rather than thinking about it properly through the lens of the workshop / conference.  For that particular workshop review, I joked with my students that if we had the answers at that time when submitting to the workshop (*cough* workshop at INFOCOM 2011 *cough*), we sure as heck would not have been submitting to some podunk workshop.  
• Having chaired a few workshops, it makes me shake my head when one gets back any peer-reviewed paper that has less than three reviews.  That is just a failure of the TPC co-chairs at quality control and at worst case, one can always backfill reviews as the co-chair.  That is kind of your job.  Some might object and point out that the ultimate job is to ensure good papers but I would argue conferences are there to encourage feedback and discussion.  Bad reviews outside of the top tier conferences (you sort of have no choice on those conferences) makes me unlikely to submit in the future nor encourage others to do so as well.
• More recently, there have been a few reviews coming back that contain only reviewer comments but no actual ratings, requiring one to click the link to actually see ratings.  Really?  Are we not CS profs and not otherwise capable of automating such things?    
• A weird review recently came in where a journal reviewer for our E2E QoS work considered the number of citations with regards to evaluating the work.  Seriously?  I must have missed the memo where the solidity of the science is dependent on how many cites your conference work gets. Then again, the reviewer clearly had a bias against all things QoS mumbling about all that QoS work that thinks about bandwidth as a scalar quantity.  On the plus side, the reviewer's comments were so wrong that the review might inspire me to finally write the dynamics paper I had in mind to prove them wrong as I firmly believe the Roberts work from the mid-2000's is still as valid as ever  when looking at large scale aggregation characteristics of flows (i.e. VBR really does become CBR in the aggregate).  The big trick is how to get the data not being the employee of a large ISP.    

I'll wrap up with a note regarding a nice paper written by our chair about advising junior faculty on the role of conferences versus journals.  I won't steal his thunder but the basic gist is that the notion that systems researchers are special flowers that can only do conferences is not really supported by the data. From the conclusion of the article:

"Look for research questions that other people will care about and that you can see a novel way to approach. Do the best research that you are able to do. Publish your work in the best conferences that you can get your work into. Also publish in the best journals that you can get your work into. Don’t think either-or, think both-and.  ...  Don’t buy into the “only-conferences-matter, conferences-are-better-than-journals” viewpoint. ... the empirical evidence is that this view is wrong." 

Food definitely for thought.

SIGCOMM - Day 1 (Tuesday)

SIGCOMM comments are still being queued up and my CellNet impressions are perhaps a bit delayed.  I should have something later on today now that I brought my notebook to the conference rather than my iPad.

A few comments / thoughts from Day 1 of SIGCOMM 2012:

• Wonderful keynote by Nick McKeown with regards to his experiences over his career.  He encouraged people to work on problems that industry did not care about (yet) and preferably those problems that make industry angry.  Certainly he is accomplishing that with SDN (Software Defined Networking) and Cisco.  The other interesting aspect is that he puts everything he does in the public domain to make collaborations easier, i.e. no IP (Intellectual Property) at risk means no issues in fighting over IP.  Plus, the bonus argument of taxpayer funds means it should be fully available / free.  
• Interesting was the tail end of Nick's talk discussing the future of SIGCOMM and pushing the notion that SIGCOMM is still pretty insular and needs more people.  He pitched that SIGCOMM should double the number of papers and aim for 2,000 attendees.  Not entirely bad I would say.
• Extremely uncomfortable moment during the first paper session.  One of the questioners (in typical SIGCOMM fashion) pointed out bluntly that the paper being presented should not have been accepted as someone already offers the exact service noted for $49 / month.  You could have heard a pin drop after the commenter made his remark.  He was most certainly right though but it was probably a bit much to request that everyone keep track of all various startups / efforts in the space.  Kudos to the presenter for staying quite composed during the questioning.  
• That being said, the relative insularity of SIGCOMM citations seems still as prevalent as ever.  That paper in question failed to cite anything out of the typical SIGCOMM sphere of conferences (NSDI, CoNext, etc.).  It did seem though that this year's SIGCOMM was definitely broader in terms of the authors.  Quite a few of the usual suspects but not overwhelmingly so.  Stanford's wireless group was heavily represented and they have some pretty cool work (see Day 2 comments coming up). 
• SIGCOMM definitely does student outreach right.  Very well done with N2Women (Day 2) and the student banquet.  Though with two hundred students in attendance, things probably got a bit diluted due to the popularity and location of being in Europe.  Bummer though that none of the industry sponsors came to the student banquet.  

To wrap up, holy number of analyzed streams by Conviva (Stoica / Zhang and company).  I am thinking that is a space we should just stay away from as it is hard to compete with 2 billion flows that Conviva analyzes.  Simply amazing but man, either I am old or something as their titles on the slides are nearly impossible to read with green text on a white background.

SIGCOMM 2012

I am currently out in Helsinki this week for SIGCOMM 2012, specifically to present our paper on WiFi offloading at the CellNet 2012 workshop.  It has been just over ten years since I was last in Helsinki for ICC 2001.  I am really looking forward to the CellNet workshop as our paper is probably one of the better ones that we have done in a while in terms of the utility of the results.  Perhaps not our finest writing but the results are quite profound for carriers when debating how much gain they will see for WiFi offloading.

As such, I am going to try do my best to put up impressions of several of the best and perhaps not so great papers / posters / demos from the four days of workshops and posters that I will be attending.  One odd paper on Wednesday on MIMO by Katabi out of MIT.  SIGCOMM seems a very odd place for such a paper though and I will be curious to see how much of a systems-ish focus there is in the paper versus an EE / PHY layer focus.