Re: [Pan-users] In the new file-upload logic, we n eed to change how Pan apparently slices the binary data …

[SciFi]

Hi,

On Mon, 04 Jul 2011 06:24:47 +0200, Heinrich Mueller wrote:

> On 07/03/11 19:22, SciFi wrote:
>>
>> The actual binary data needs to be split _before_ it is encoded
>> (before uuencoded, before yEncoded, before MIME'd, etc.)
>> or in the view of the end-user/downloader, _after_ decoding.
> Well, normally the uploader does that by rar-ing and splitting the files
> he wants to upload into several equally-sized chunks.
> pan could do that, too, but as it doesn't handle the par2 creation
> I think that would be counterproductive.

Ah, you are mistaking
the "part" splits (done by RAR and other apps)
versus
the "article-size" splits (which one cannot control outside the NNTP binary 
posting app).

It's like what a three-dimensional cake looks like:
a round cylinder-full of spongey baked batter
divided by (horizontal) layers ("parts"),
and further divided by "personal-sized slices" within each layer.
This is how big files are being posted to Usenet
designed to fit every slice under an article-size limit.

To be absolutely clear:
I am talking about the "article-size" splits
when even the "part" split is too big to fit
inside a single article/message.
People usually select rather big "parts" on the order of 50MB each (+/-),
which together will form, say, a 700MB AVI file or similar.
The 50MB "part" *must* be split further, in this regard,
by the posting app itself (so Reference lines can be used to
join-up the thread of articles if needed),
or else the single 50MB message will
likely be quickly lost in the NNTP server system
(with a possible "too big to store" error message back to the poster).
This kind of split happens the bulk of the times during uploading.
A lot of people do not realize this.
There is nothing that takes into consideration
the "article-size" splits
until the file-posting app is actually dealing with them,
directly, _during_ posting (in real time).
This is usually hidden inside the NNTP posting app itself,
and changing these settings is too technical for most people.

Furthermore,
This "article-size" split needs to match the PAR2 size (or a multiple)
chosen by the poster ahead of time (whether or not he knows it)
when he generates those recovery files.
Setting this inside e.g. Pan should be done again by the same poster.
With the way you are designing Pan to control this aspect,
by the "counting encoded lines" logic,
the poster CANNOT accurately control this value.
This mechanism needs to be changed, please.
We need to be able to match BY BYTE COUNT the article size at this level.
So we want the article-size split to be done on the current "part" file
BEFORE it is encoded, see.

This aspect, matching article-size to the PAR2 blocksize (or a multiple),
is an unwritten rule that ought to be included in a revamped GNKSA.
(It's even worked-into the various parms for yEnc itself already.)
Too many apps (and people accepting their settings) are violating it.
And it will ruin Pan's new status if your mechanism isn't changed.

Yes the vast uploads do work, but most don't work _properly_
taking several cruddy shortcuts in their formations.
I for one intend to do _proper_ uploads, following these unwritten rules,
and Pan needs to follow the rules if I am to
use Pan for uploading the big files.

btw adjusting the relevant settings
is easily done in the parms for the yencee project,
which is why it's my favorite app for posting binary files.

It's also a fine art
where the PAR2 & article blocksize
comes out "even-steven"
with the size of each "part" file.
When realizing this,
PAR2 is suppose to be designed to re-create each "message" (article)
that effectively forms the part/file,
if reconstruction of a corrupt download is needed, see.

And to re-iterate:
we cannot control this article-size properly
with the current "counting encoded lines"
being designed into Pan.

I guess I'll finish this writeup for the general reader.

Take a look at a binary posting in the world-wide Usenet.
There are a number of articles
that will formulate the "part" (actual file)
being posted.
The actual number of articles
is seen as the '(number)' (in parentheses)
at the very-tail-end
of Pan's subject lines
in its Header Pane view
for such postings.
(Other newsreaders usually show it the same way.)

(btw the Subject lines are *not* to contain the actual '(number)' string
at the tail-end when posted,
these '(number)' strings are calculated by the end-user's app.
Else you'd get a doubled-up '(number)' at the end of Subject lines. <g>)

Each article must be joined by the
Reference header line
pointing to the previously posted article (Message-ID)
all chained-together
until the "part" (file) is finished (at EOF).
Else the articles won't follow each-other in the manner
needed to re-create the original file.

(Surely you know what I mean by all this.
I'm spelling it out for the general Usenet patron
who might read this for better understanding what's going on.)

> Cheers.

I would strongly urge you to look at yencee and other posting apps
that do things _correctly_ (deep-down in the 'engine' as it were)
to base your coding for Pan, please.

I would like to help, but I don't know C++ as well as plain-C,
yet I could try by showing the shell-scripts I use with yencee etc.
that compute the "even-steven" values as I mentioned above.

And I would like to not need to type so much to explain myself.  ;D