Re: [Pan-users] (solved) Re: Errors downloading binary news

[Duncan]

Volker Wysk posted on Sun, 29 Jul 2018 20:18:57 +0200 as excerpted:

> I've figured out, what's going on.

=:^)

> I had been trying to use the newsserver of the internet provider
> IN-Berlin (in-berlin.de). I've contacted their support, and he kind of
> hinted that this is a server issue. They are deleting all but the last
> part of multipart messages, on purpose (so it seems). They don't want
> users to download binary news. I can kind of understand this, but just
> doing so, without any warning, is very bad.

Now that we know the problem, I can expand on the brief treatment of this 
possibility in my earlier reply...

It's not that they're /deleting/ all but the last part, it's that they 
have a size filter on their incoming feed from whoever they're peering 
with, probably because they get charged by the GiB, and because they are 
an ISP that sells connections to the net, not specifically an NSP that 
specifically sells news, so news isn't their priority (the actual net 
connectivity is, and should be), they don't really care that the size 
filter breaks most binary news.  That may actually be a legal benefit as 
well, but it's almost certainly more accident than intent.

Because text posts tend to be far smaller than binaries, the filter lets 
text posts thru, as it will very small binaries.  But when binary files 
are large enough to split into parts for posting, unless the size of the 
file is an exact multiple of the split-size (set by the poster or their 
client), all the parts but the last will be whatever the split-size is 
set at, with the last part being whatever odd bit smaller than the split-
size remains.

Think of a binary file of say 13.3 MiB, with a split size of say 1 MiB.  
To make the math easy we'll pretend yenc is 100% efficient and the 5% 
encoding overhead doesn't exist.  That'll be a 14-part post, with 13 
parts of the 1 MiB split-size each, and a last part of 0.3 MiB.

Now say the ISP's size filter is set to half a MiB.  All but the last 
part will not be downloaded to the ISP's server, but the last part will 
be, because at 0.3 MiB it fits under the 0.5 MiB filter limit.


Of course the obvious workaround for a poster concerned about people on 
such poor quality news providers is to cut their split size down 
similarly, to say a tenth of a MiB instead of the 1 MiB split-size we 
used in the example above.  And a few posters actually do that.

But of course that means nearly ten times the number of parts as well, so 
instead of 14 parts, we have /almost/ 140, actually 133 if the file is 
/exactly/ 13.3 MiB or just a bit smaller.  And the more parts there are 
the more chances for one of them to get lost, plus there's actually more 
overhead as well, because a higher percentage of each part is headers and 
similar overhead, so there's a definite negative to it.

And most posters consider it a losing battle to try to make allowances 
for poor quality news providers beyond a certain point anyway, whether it 
be outsmarting size filters or providing ridiculous amounts of redundancy 
in the form of par files.  Instead, users who care enough are assumed to 
eventually switch to a good quality news provider, where they won't have 
these sorts of problems.

And the fact is, even the bad dedicated news providers tend to be better 
than all but the extreme best bundled news provider, because unlike the 
bundled news providers that are actually selling something else, say 
connections to the net, news is what the dedicated news providers are 
/selling/, and if they don't do at least a half-decent job at it, they 
pretty quickly lose their customers and go out of business.

So like it or not, the fact is that if you care about news, particularly 
binary news, you pretty much *must* pay for a dedicated news provider.  
But there are ways to make it more economical.  See below.

> I've tried it with another news provider (Disputo), and it appears to
> works fine, with the same version of pan.

=:^)

FWIW, there are two primary sales models for dedicated news providers, 
the price-per-period model, usually monthly tho some offer discounts for 
pay-in-advance out to annual or so, and the price-per-GiB, aka the "block-
at-a-time" model, generally with the price-per-gig going down 
substantially as the size of the block goes up.  The key is that with 
proper block plans the credits don't expire, so if you don't download 
much, your block can last you months or years.

The per-period model is the most common, but for the many not downloading 
at TiB/month scale, it's also the most expensive, often by far.  The key 
is to figure out how much you're likely to actually be downloading, and 
see which is most economical from there.

There are two block-mode news sellers I know of, astraweb.com, which I 
have and which has both blocks and monthly subscriptions available, and 
blocknews.net.

Astraweb's biggest block (and thus lowest per-gig price) available is 
1000 GB (base-10), $50, so $.05/GB (USD from the US, German prices may 
differ).  The credits don't expire so if you don't use it that much.  
Headers are discounted to 20% so 10 GB of headers will be charged 2 GB of 
usage.

Blocknews seems to be higher priced, at least at the top end, 1024 GB for 
$75 or 3072 for $215.  They only do blocks but refer to usenetnow.net for 
monthly accounts.

So usenetnow has three months for $30.49, say $10/mo (tho it's $12/mo if 
done monthly).  And astraweb has $20/3mo (so ~$7/mo) for 10 Mbit/sec 
speed, $39/3mo (or $15/mo) uncapped speed.  I don't know what the going 
monthly rate is but we'll take those as representative.

Which makes astraweb's $50 1000 GB block ~7.5 months of astraweb's 
$20/3mo service, a bit under 4 months of their $$39/3mo uncapped service, 
or ~5 months of usenetnow's uncapped service.

Taking the longest of those, 7.5 months, the breakover point would be 
~133 GB/mo.  If you're downloading more than that, the monthly may make 
sense.  If you're downloading less, or if you sometimes go quite some 
time with little to no activity and would still be paying, then the block 
account is very likely a better deal for you.  Of course at the other end 
you have (under) 4 months, which would be a 250 GB breakover.  You can do 
the math for your own figures.

Here, since I don't do much main usenet, binary or text, at all (I do a 
reasonable amount of text, but it's all on mailing lists via gmane's 
list2news service, as I prefer my mailing lists in news format), when I 
spent that $50 for 1000 GB from astraweb over two years ago (with over 
9/10ths of it still left today), I figured at my current usage that may 
well last my lifetime, and even if it didn't, I figured it'd last me at 
least a year and thus be a much better buy than monthly, and the very 
fact that it was used so fast would mean that I got back into binaries 
heavily, probably /because/ I had the block account to play with, which 
would still make it a good buy.

OTOH, I know one guy who downloads multiple TV series, for friends and 
extended family as well as his own household, who often clears a TB/mo on 
news.  Obviously a block account's not a good buy at all for him!

Now if you're just now sorting out the problem above, that means you're 
not yet, at least, downloading news at TB scale, and likely not at the 
133 GB/mo scale either, or if you are, it may not last.

Which means a block account that you can use when you want, and forget 
about it for awhile without it expiring so you can come back to it months 
later, without paying for months not used in the mean time, might 
actually make a lot of sense for now, even if you ultimately get to 
downloading enough to consistently hit the breakover point and make a 
monthly account make sense.

Something to consider anyway, and for me, it made it economical to have a 
dedicated provider. =:^)

Meanwhile, if anyone knows of any other block account NSPs, please post 
'em!  Even if they're more expensive, having more than two alternatives 
for block providers is something I'd consider a good thing!  And since 
block accounts don't expire, I'd consider getting another account 
somewhere else too, just to give myself some flexibility.

(Note that setting a server's connection count to 0 in pan disables it 
without losing the config, or setting priority to backup means it isn't 
used unless all primary servers don't have a particular message.  Further 
note that while the GUI only has primary/backup, those translate to 
numbers in the config file, and by hand-editing, you can add as many 
levels of backup as desired.  So it's possible to set a configured server 
to only use if necessary (backup), or to disabled but still configured (0 
connections), if desired.)

Meanwhile (2), FWIW, while most dedicated news providers allow say 20-50 
connections, if you need more than say the four connections that pan's 
GUI allows to max your allowed bandwidth, something's probably wrong 
somewhere.  Number of connections shouldn't be an issue, unless like one 
guy I know, you have your main downloader machine with its connections, 
your just browsing machine with a few more connections, your wife has her 
own machine with her own groups for a few more, etc.  And then it's not 
to maximize the bandwidth, it's simply to allow multiple machines in your 
household to connect without having to worry too much about exceeding 
your allowed number of connections.

(And BTW, if you /do/ want to up pan's number of connections, simply edit 
servers.xml by hand.  Pan's GUI allows no more than 4 connections per 
server as that's what GNKSA requires, and GNKSA compliance is something 
pan's users and devs tend to be proud of.  But GNKSA does /not/ say that 
pan can't honor a hand-edited config, just that pan can't allow a user to 
configure more than that.  So that's how we handle it, if a user wants 
more than four connections per server they can hand configure it and pan 
will honor that.)

Hope it's useful! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman