Re: [Pan-users] Pan/0.139 - threading issue

[Dave]

On Tuesday 20 August 2013 08:14:51 Duncan wrote:
> Dave posted on Tue, 20 Aug 2013 00:21:15 +0100 as excerpted:
> > Is anyone seeing newsgroup theading issue with Pan 0.139?
> > It's natively compiled on FreeBSD 9.1
> 
> I use pan built from live-git, here (as can be seen from my headers, since
> I'm posting with pan via the gmane list2news service), so I'm a but past
> 0.139 release, but I don't believe there have been any threading changes
> in quite some time... several versions, tho not being a C++ coder (tho I
> can do sysadmin's level coding, that is, the occasional patch) I go more
> by the git commit comments than the code.
> 
> I suppose it's possible that it's a library issue, however.  What version
> of gmime?  (FWIW, 2.6.16, here on gentoo/~amd64.)

I appear to have two version of gmime installed

gmime-24-2.4.33  
gmime-26-2.6.16 

I just did a pkg_delete -f gmime-24-2.4.33, which did not report any 
dependancies as it normally would if something required that version.  I'll 
test again now that's gone.

 
> > On the group in question:
> > newsgroup:  free.virginmedia.test
> > subject:   Testing custom User Agent header
> > date:  19th Aug, 11pm local (GMT/UTC+1)
> > 
> > There are replies I've made to myself but they don't thread properly,
> > showing up as replies to the original post rather than the one it's
> > actually in reply  to.  Others, using other newsreaders including OE
> > and Agent have commented on the threading problems.
> > 
> > I've tested replies with full automatic message quoting, highlighting
> > text then reply to only quote specific test, switched off/on body pane
> > word wrap, and wondered if the splitting of headers in replies might be
> > a cause, eg References: header splits in the middle of references
> > instead of at the end, ie at a comma.  Likewise,the User Agent: header
> > is huge and gets split across line.
> 
> It'd have to be the references header.  None of the rest of the things you
> mention should have anything at all to do with threading.  It's worth
> noting that (AFAIK) pan threads exclusively via message-id as found in
> the references header, while certain other news and/or mail clients,
> including OE at one point (altho I'm not sure it still applies, I
> switched to freedomware instead of crossing the eXPrivacy line, tho back
> in the day I ran IE/OE betas so I know more than average about historical
> versions), fall back to subject line threading as well -- they'll thread
> identical (but for Re: etc) subject lines together even in the absence of
> references headers.
> 
> The RFCs (Internet Request For Comments documents, which ultimately
> become STDs, Standards, but by that point everybody is used to referring
> to them by the RFC number, so RFCs is how they're normally referenced)
> are the definitive standard here, specifying the contents and format of
> the references header as up-thread message-IDs, as well as the proper
> method for "header folding."
> 
> > On the other hand, looking at the "Testing custom User Agent header"
> > thread using knode,  everything looks just fine with all messages
> > threaded and indented correctly.
> 
> ... Which means the information must be there in the references header
> for knode to use.  It's apparently a bit less strict in its header
> parsing, however, or unfolding folded headers differently (maybe without
> an added space/comma at the split?), tho my memory of the internet
> messaging RFCs is fuzzy enough I couldn't tell you which would be
> "correct" without looking it up.
> 
> > If you can't access that newsgroup but have some ideas/help/advice, I
> > can post headers here if required.
> 
> That would be helpful.  Probably just the references headers, perhaps
> with one header on either side just to be sure we keep the context, if it
> happens to be important.  Be sure to retain verbatim header folding, as
> that's almost certainly the issue.
> 
> (I very likely technically have access to the group as I have an
> unexpiring block account, but I don't regularly use it (so the unexpiring
> bit is good! =:^), and if the information I need is all there I tend to
> reply immediately, where as if I have to look, I'll often skip it (not
> often) or mark it unread again, to reply to later (the usual case)...
> which can be MONTHS (!!) later, at which point I may well decide it's not
> likely to be relevant any longer and ultimately skip it then.  Posting
> the headers is thus likely to be in practice the fastest way to a proper
> reply, at least from me.)
> 
> Heinrich (who would likely be creating the fix if needed) or others may
> well reply in the mean time.
> 
> 
> Meanwhile, based on the evidence so far (your mention of "folded"
> references header, header folding being the term used in the RFCs, and
> knode apparently getting it right) I strongly suspect that it's down to
> that, tho whether it's on the posting end, folding the references header
> incorrectly according to the RFCs, or the receiving end, unfolding them
> incorrectly, I can't say.
> 
> What I CAN say, however, is that the RFCs have a very well known general
> policy of being strict in observance with what you send, but rather more
> tolerant when parsing what you receive.  So it's quite likely that the
> information is all there (as can be seen from knode getting it right),
> but either pan isn't being sufficiently RFC-strict in how it folds the
> references header when it sends the message, and knode is simply being
> tolerant in reconstruction as it should be (while other clients including
> pan aren't quite that tolerant), or pan is using an RFC allowed but
> relatively uncommon and thus not often tested mid-ID split, and due to
> the relative rarity of the case, few clients are properly coping with an
> RFC-folding-compliant references header.
> 
> I may just go lookup the RFC and see what it says as that's an
> interesting question regardless, but I'll send this off first, in case I
> don't get to that.

Thanks Duncan, I knew you'd be first and comprehensive :-)

Xnews, Outlook MesNews and Thunderbird never wrap the References: header, 
Turnpike does, but never splits mid-reference.

Here's a new thread showing three replies, all of which thread in Pan as if 
they are replies to the original post but, as shown by the quoted text 
indentations are a sequence of replies to each preceding post.


Path: not-for-mail
From: DaveG <address@hidden>
Subject: thread test1
Newsgroups: free.virginmedia.test
User-Agent: Pan/0.139 (Sexual Chocolate; GIT bf56508
 git://git.gnome.org/pan2)
Message-ID: <address@hidden>
MIME-Version: 1.0
Lines: 8
NNTP-Posting-Host: 92.238.71.115
X-Complaints-To: http://netreport.virginmedia.com
X-Trace: 1377002508 92.238.71.115 (Tue, 20 Aug 2013 12:41:48 UTC)
NNTP-Posting-Date: Tue, 20 Aug 2013 12:41:48 UTC
Organization: virginmedia.com
Date: Tue, 20 Aug 2013 12:41:48 GMT
X-Received-Bytes: 905
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

test 1

######################

Path: not-for-mail
From: DaveG <address@hidden>
Subject: Re: thread test1
Newsgroups: free.virginmedia.test
References: <address@hidden>
User-Agent: Pan/0.139 (Sexual Chocolate; GIT bf56508
 git://git.gnome.org/pan2)
Message-ID: <address@hidden>
MIME-Version: 1.0
Lines: 12
NNTP-Posting-Host: 92.238.71.115
X-Complaints-To: http://netreport.virginmedia.com
X-Trace: 1377002540 92.238.71.115 (Tue, 20 Aug 2013 12:42:20 UTC)
NNTP-Posting-Date: Tue, 20 Aug 2013 12:42:20 UTC
Organization: virginmedia.com
Date: Tue, 20 Aug 2013 12:42:20 GMT
X-Received-Bytes: 934
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

On Tue, 20 Aug 2013 12:41:48 +0000, DaveG wrote:

> test 1

reply 1

############################

Path: not-for-mail
From: DaveG <address@hidden>
Subject: Re: thread test1
Newsgroups: free.virginmedia.test
References: <address@hidden> <pan.2013.08.20.12.42.
        address@hidden>
User-Agent: Pan/0.139 (Sexual Chocolate; GIT bf56508
 git://git.gnome.org/pan2)
Message-ID: <address@hidden>
MIME-Version: 1.0
Lines: 16
NNTP-Posting-Host: 92.238.71.115
X-Complaints-To: http://netreport.virginmedia.com
X-Trace: 1377002564 92.238.71.115 (Tue, 20 Aug 2013 12:42:44 UTC)
NNTP-Posting-Date: Tue, 20 Aug 2013 12:42:44 UTC
Organization: virginmedia.com
Date: Tue, 20 Aug 2013 12:42:44 GMT
X-Received-Bytes: 1052
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

On Tue, 20 Aug 2013 12:42:20 +0000, DaveG wrote:

> On Tue, 20 Aug 2013 12:41:48 +0000, DaveG wrote:
> 
>> test 1
> 
> reply 1

reply 2

####################################

Path: not-for-mail
From: DaveG <address@hidden>
Subject: Re: thread test1
Newsgroups: free.virginmedia.test
References: <address@hidden> <pan.2013.08.20.12.42.
        address@hidden>
User-Agent: Pan/0.139 (Sexual Chocolate; GIT bf56508
 git://git.gnome.org/pan2)
Message-ID: <address@hidden>
MIME-Version: 1.0
Lines: 20
NNTP-Posting-Host: 92.238.71.115
X-Complaints-To: http://netreport.virginmedia.com
X-Trace: 1377002590 92.238.71.115 (Tue, 20 Aug 2013 12:43:10 UTC)
NNTP-Posting-Date: Tue, 20 Aug 2013 12:43:10 UTC
Organization: virginmedia.com
Date: Tue, 20 Aug 2013 12:43:10 GMT
X-Received-Bytes: 1128
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

On Tue, 20 Aug 2013 12:42:44 +0000, DaveG wrote:

> On Tue, 20 Aug 2013 12:42:20 +0000, DaveG wrote:
> 
>> On Tue, 20 Aug 2013 12:41:48 +0000, DaveG wrote:
>> 
>>> test 1
>> 
>> reply 1
> 
> reply 2

reply 3