[Pan-users] Astraweb Is Sending Bad Headers! -was: Articles Pan Can't Read

[Mark S Bilk]

When a newsreader requests article headers from a news server
with an OVER (formerly XOVER) command, the server sends the 
headers of each article combined into a single line, with the 
contents of the various headers separated by tabs, in this 
order (quoting from Section 8.3.2 of RFC 3977
http://tools.ietf.org/html/rfc3977#section-8.3):

      "0" or article number
      Subject header content
      From header content
      Date header content
      Message-ID header content
      References header content
      :bytes metadata item
      :lines metadata item

   For all fields, the value is processed by first removing all 
   CRLF pairs (that is, undoing any folding and removing the 
   terminating CRLF) and then replacing each TAB with a single 
   space.

Since tabs separate the various header content fields of the
line the server sends to the newsreader, obviously those fields
must not contain any tabs.  Otherwise the newsreader won't be
able to count the tabs in order to distinguish and separate
the fields, and thus reconstruct the headers.  That's why
RFC 3977 says that tabs in any of the header contents must be
replaced by spaces before those contents are combined and sent
to the newsreader.

Unfortunately Astraweb doesn't replace those tabs with spaces.

Tabs occur most frequently in the References header, when the
article is two or more followup levels below the original post.
The Pan newsreader is unable to retrieve, or sometimes even see,
articles from Astraweb whose References header is folded using 
tabs.  Most newsreaders don't post that way, but some do.

That's why Pan couldn't download from Astraweb three of the 
five test articles that I posted.  And why, from Astraweb, 
Pan can't see all the articles in deeply nested followup 
trees, and sometimes sees a tree broken into smaller ones.

This was all determined by capturing the stream of alt.test
headers, from Astraweb and from Blocknews, using the wireshark 
network packet analyzer.  Blocknews formats the header records 
properly, removing tabs from the field values before combining 
them.

I'll send another trouble-ticket to Astraweb, but after their 
previous performance I don't know if they'll care.  It shouldn't
be hard to insert code into Pan to interpret the incoming header
stream defensively and determine when the References field ends,
which is what KNode and other newsreaders must be doing, since 
they work OK with Astraweb.