[Pan-users] Re: Weird binary splitting

[Duncan]

Victor Ducedre <address@hidden> posted
address@hidden, excerpted below, on  Wed, 22 Nov 2006
21:29:25 -0500:

> I'm coming across an occasional oddity in 0.119 when saving large binary
> files to disk:  the file gets split into two parts.  Instead of getting,
> e.g.
> 
>    filename.r01         15,000,000
> 
> I end up with 
> 
>    filename.r01         14,750,400
>    filename_copy_2.r01     249,600
> 
> If I delete the saved files and resave it again from the cached copy, it
> will consistently split it like the previous save.  If I redownload it
> (instead of the cached copy) it may resplit again, or it may save just
> fine.  Aaargh!
> 
> If I cat the two parts together (since it appears that bytewise it's all
> there), par2repair reports missing data blocks.

This would appear to be due to a problem in your connection.  yenc
attachments include per-individual-segment crc32 checksums.  I've not seen
pan do this and didn't know it did, but the logical explanation here would
be that most of the segments come in fine, but one fails the checksum
validation.  pan is apparently then splitting it at the checksum error.

Since the error is in the individual segments as downloaded to cache,
simply re-saving from cache results in the same error at the same spot. 
Killing the cached copy and redownloading may result in a clear copy
coming in, or not.  Concatenating the parts using cat of course results in
a unified file with the problem still embedded, thus the complaint about
missing blocks from par2.  At least it should be recoverable using par2,
as long as enough recovery blocks are available.

As I've explained before but you may have missed as the below hints that
you are fairly new to new-pan and thus possibly to this list/group, I
prefer to cache everything first (into a multi-gig cache, size must be
increased by editing preferences.xml directly as there is no GUI method
for setting it), then do my saving and sorting locally, after everything is
in cache. People have previously complained about some saves never
completing -- they get almost done and just stick there.  I attributed
this to the same crc32 checksum invalid issue, but hadn't seen your
particular variant -- split attachments.  Or at least I hadn't personally
seen their issue since I cache first, and if I've seen yours I've just
been routinely working around it using par2 and hadn't noticed the issue
as you mention.

> On an unrelated topic, has the rules/filters idea gone?  I miss the raw
> power to (locally) delete large swaths of the Usenet population. :)

Rules have disappeared for the moment, yes.  They will be replaced, but
not until after 1.0, which is imminent.  Scoring does much of what filters
did, but there's presently no way to do anything with it except alter
what's displayed -- not displaying ignored articles by default, and
color-coding the score column for the others.

>From the resulting discussions here, Charles seems to think that the way
rules were implemented wasn't intuitive enough, and many used the feature
little if at all as a result.  I'd tend to agree -- it wasn't that
intuitive -- tho it was certainly a nice feature for us power users.

Anyway, the discussed replacement will take a bit different form, and
won't be called rules or be as difficult (or as flexible) to configure,
but should pretty much allow folks to do what they were doing with rules
anyway.  The replacement as discussed will be in the form of several
checkboxes in preferences, automating certain tasks.  With the appropriate
checkboxes checked, pan will then automatically:

* delete ignored articles.

* mark-as-read negative-scored articles (that don't reach ignored).

* download watched articles.

* There has been some discussion about a fourth option as well, extending
the auto-download to anything positively scored, not just watched articles.

As a replacement for rules has been much requested and this is (mostly)
it, and Charles has already indicated his thinking on the matter, this is
fairly likely to appear relatively early in the 1.1 development cycle,
after the release of 1.0 for which we are in feature-freeze at this time.

The above implementation should be simpler, yet combined with the already
existing new adjustable expiration (the other reason I used filters/rules
in old-pan, semi-automating expiration of old posts on gmane's list2news
archive service, which as an archive doesn't expire posts), should allow
folks to do pretty much all of what rules did previously, tho it'll
naturally require a bit of adjustment in ordered to do so.

Meanwhile, it's not automated, but using scores you can already do the
categorization (that part's automated), then by sorting by score
(unthreading if necessary to get full score sort), you can group all your
ignored/low-scored/whatever together and delete them, and all your high
scored and download them, without having to manually go thru everything
else.  That's what I've been doing.  It's better than nothing, but
I'm definitely looking forward to having it all automated again.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman