Re: [Pan-users] Incorrect threading - 0.139 GIT bf56508

[Duncan]

Jim Henderson posted on Tue, 10 Dec 2013 20:37:16 +0000 as excerpted:

> On Tue, 10 Dec 2013 20:26:34 +0000, Jim Henderson wrote:
> 
>> On Tue, 10 Dec 2013 20:07:04 +0000, Jim Henderson wrote:

>>> I just upgraded my systems to openSUSE 13.1 (from 12.2), and the
>>> version of pan jumped from 0.136 to 0.139.

[Later clarified.  Same version of pan, 0.139.]

>>> With this update, it seems that responses are being attached to the
>>> parent of the post I'm responding to (or the root of a thread if there
>>> is no parent) rather than to the post that I'm actually replying to.

You've apparently not been paying close enough attention to this list/
group over the last months, and have been hit with a known problem that 
had quite a thread while the issue was explored, first after someone else 
hit it on one of the BSDs, then after I updated and ended up hitting it 
too.  But it's *NOT* pan, it's the gmime library pan uses that's at 
fault... which explains why the same version of pan is behaving 
differently. =:^(

>>> Oddly, though, the references header seems to include the correct
>>> message IDs, but I wonder if they're in reverse order or something.

It's incorrect references header "folding", splitting the header over 
multiple lines as allowed, but splitting in the wrong places.

>> I've built the latest git code, and it still seems to be a problem
>> here.

As I said, it's not pan, so rebuilding pan won't fix the issue.  It's 
gmime, but the issue *HAS* been fixed upstream, in gmime 2.6.19.  FWIW 
affected gmime versions are 2.6.16-2.6.18.

> Also, just to clarify, this isn't a thread display issue - I've had
> reports from other NNTP users using other readers (Xananews, Knode, etc)
> that my posts are threaded incorrectly on the server.

Yes, as I said, incorrect references header folding.  The first affected 
post will include all the correct message-ids, but will break/fold the 
lines at the wrong place according to the RFCs, so it won't thread 
correctly, and further replies will actually be missing some message-ids 
as neither pan nor most other clients can make sense of the screwed up 
folding after that and will generally drop further IDs when they start to 
make no sense.

Getting technical:

A message-id is of a very similar form to an email address (split with 
spaces here to avoid gmane's address encryption routines), but with a 
unique bit, often including the current time and/or a random bit so as to 
create a GUID (globally unique ID):

< globally.unique.semi-random-bit.ad7249c37 @ domain.sample.com >

Now, normal header-folding rules would allow folding at various 
characters, including dots and either side of the @, as well as 
immediately inside the <>, and that's what newly broken code in IIRC 
gmime.2.6.16 (IIRC 2.6.15 was the last unaffected version on the low end, 
as I said, 2.6.19 fixed it, so 2.6.16-2.6.18 are affected) did with 
references as well.

But reference headers folding is somewhat more strict.  I read the 
relevant RFCs and figured all this out in ordered to trace and file the 
bug once I hit it, but I'm going from memory here, but with that 
disclaimer, IIRC references header folding MUST avoid folding inside each 
"side", and SHOULD avoid folding immediately either side of the @ and 
immediately inside the <>, tho that's a SHOULD not a MUST.  (SHOULD and 
MUST are used here in the RFC normal sense, an implementation breaking a 
MUST is non-compliant, while one breaking a SHOULD remains technically 
RFC compliant but is breaking guidelines that some more obscure 
implementations follow and will thus break those implementations.  An 
implementation MUST follow MUSTs or it WILL break things for nearly 
everyone, but the recommendation is to receive/accept incoming SHOULD 
violations, while never sending/outputting them.  Never-the-less, sending 
a SHOULD violation does remain technically compliant at the MUST level.)

The broken gmime versions fold at both the dots (breaking a MUST) and at 
the @s (breaking a SHOULD).

So update gmime to 2.6.19+, or drop back to 2.6.15-, and the problem 
should disappear.  (The original reporter here, on one of the BSDs, fixed 
it for him by dropping back to the 2.4 series, which I believe is in deep 
enough maintenance mode it never got the bad code in the first place.)

Of course you'll need to check gmime reverse dependencies (including but 
perhaps not limited to pan) on your system and may need to rebuild them 
as well, altho hopefully you won't, as long as you stick with the gmime 
2.6.* series.

I'd recommend filing an OpenSuSE bug too, as their gmime maintainer 
obviously didn't note this bug and either bump their gmime version 
accordingly or backport the patch.  You may want to include some/all of 
the following links in that bug as well, particularly the first one, the 
upstream gmime bug I filed that lead to the fix in 2.6.19.


Links (vertically spaced to avoid inappropriate wrapping in replies):


Upstream gmime bug that I filed (note that I used "split" in the bug 
summary, while the proper RFC-technical term as used above is "fold"):

gmime 2.6.16-2.6.18 split references headers in the wrong place, 2.6.15 
works fine

https://bugzilla.gnome.org/show_bug.cgi?id=709031


Current RFC on the topic (linked in the above bug as well):

Internet Message Format

http://tools.ietf.org/html/rfc5322


The original pan-user-list thread on gmane, broken threading and all.
=:^\  Unfortunately the references headers aren't available via the web 
interface, but you can use the information there to look up the thread in 
your mail client or on gmane's news interface, if desired.  (Again, this 
is linked from the upstream gnome/gmime bug as well.)

Pan/0.139 - threading issue

http://comments.gmane.org/gmane.comp.gnome.apps.pan.user/14350


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman