bug#38011: 27.0.50; [PATCH] WIP on allowing Gnus backends to return header data directly

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> On 01/03/21 08:45 AM, Lars Ingebrigtsen wrote:
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Sure thing. It's in girzel/gnus-headers now. I made a few more sneaky
>>> last minute changes, so yes... testing is in order.
>>
>> I'm now running this branch, and good news: No breakages so far.  :-)
>
> Great!
>
>>> It gets complicated because the cache and the agent need to mix their
>>> saved headers into whatever newly-fetched headers we get from the
>>> server. So instead of having them call `gnus-retrieve-headers' and
>>> mixing their cached text into the nntp-server-buffer, they now call
>>> `gnus-fetch-headers' on the server, which actually returns real headers.
>>
>> Yes, that was the main bit I was unsure of.  The braiding stuff tries to
>> be efficient and avoid parsing things twice (or more) -- you usually get
>> a bunch of headers from the NNTP server, and then you have even more
>> headers in the agent/cache, and it stitches them all together as text,
>> and parses the resulting mess.  (If I remember correctly; it's been at
>> least a decade since I looked at that code.)
>>
>> Are headers parsed more than once now and then merged?
>
> I certainly hope they're not parsed more than once, but it would
> definitely be good to have more eyes on that part of the code. For
> instance, `gnus-cache-retrieve-headers' finds cached articles in the
> group, then uses (gnus-sorted-difference articles cached) to find
> uncached articles. The uncached articles are sent to
> `gnus-fetch-headers', and the cached articles are parsed from the cache
> file using:
>
> (gnus-get-newsgroup-headers-xover
>  (gnus-sorted-difference
>  cached uncached-articles))
>
> Then the two sets of headers are merged and sorted. The agent does
> something similar. I don't think the parsing should do any duplicate
> work, though it's possible that adding the newly-received headers back
> into the cache file is not optimal: it appends the new headers at the end
> of the buffer, then re-sorts the whole buffer. 
>
> Crud, I just realized that the agent will re-sort its cache file using
> `gnus-agent-check-overview-buffer', but gnus-cache doesn't do anything
> similar. So the write process will have to insert the new headers into
> the buffer in sorted order.

Oh hang on, we're not supposed to be adding any new headers to the
gnus-cache cache files here at all. So that chunk of code can just be
removed. Just when you thought you had a handle on what was happening...