Re: wget -p shall honor -H (isn´t used unless -r is given)

[Elmar Stellnberger]

Hi Tim

  No, the images are simple, plain <img src=xx images that get 
displayed as part of the web page without any Javascript, otherwise the 
grep -o| sed would not have worked. The point is that the images do not 
get downloaded by a usual invocation, because their domain name is 
different from www.esquire.de, that is static.esquire.de instead of 
www.esquire.de. Note that for wget, -D remains without any effect as 
long as you do not specify -H. The -r should not be needed for a wget -p 
as these are two different semantics: download a single web page and 
recursively download a sub-directory, one or more whole domains.

  The problem about:
wget -p -r -l 1 -N -H -D static.esquire.de 
https://www.esquire.de/life/reisen/schoenste-wasserfaelle-welt-natur
... is that it will only download the directly referenced .-html page 
from www.esquire.de, no supporting .css or .js files from www.-. Note 
also that when -H isn´t given, it won´t even download a single file from 
static.esquire.de. That is a -D different.domain without an additional 
-H results in -p and -r being disregarded.

  Also:
wget -p -r -l 1 -N -H -D esquire.de --exclude-domains www.esquire.de 
https://www.esquire.de/life/reisen/schoenste-wasserfaelle-welt-natur
does nothing different with wget 1.21.3 (In the former email I had used 
a newer version of wget).

  That is apparently you first need a wget -p for www.esquire.de and 
then download the images at static.esquire.de with -r in a second run. 
Not ideal.
  Forget the posted screen output from grep -o/sed download of the 
static.-images in my last email; wget must not behave like that 
(binary/cpu exposes wrong behaviour). It is not a bug (see also: 
https://www.elstel.org/uni/, SAT-solver master thesis, Epilogue, 
starting on from point 6 as well as countless other examples).

  Having a try with wget2 will be an interesting thing, though I´d 
personally consider wget -p to be a standard feature and I´d like to see 
it work there also when you have more than one domain on a specific page 
and that currently needs to be given with -H -D xy.tld. "-D xy.tld" is 
needed if you don´t want to download Javascripts from Google, although 
this may as well be interesting whenever you intend to view the page 
entirely offline, afterwards. Look at my last posting "wget 
--page-requisites/-p should download as for a web browser" which is 
pretty much about similar stuff:
https://lists.gnu.org/archive/html/bug-wget/2023-10/msg00008.html

  To me wget[2] -r -D without -H would be an interesting thing for to 
tell that additional domains shall be considered but without recursively 
fetching content from there (useful for -p as well as -r, as long as you 
don´t download recursively from two or more domains by one invocation). 
That is the first example in this email made sense also when -H is not 
given.
  Basically if you implement a feature like this you could make it work 
for multi-domain recursive downloads as well, that is you could have 
separate options for both behaviours like -H xy.tld,wz.tld -D 
add-non-recusrive.tld. As by that example the parameter of -H would need 
to start without a minus and you needed wget -H -- https://..., as I 
usually do that, here to avoid searching for ://.).

Regards,
Elmar


Am 08.10.23 um 19:48 schrieb Tim Rühsen:
> Hey Elmar,
>
> did you try the following?
>
> wget2 -p -r -l 1 -N -D static.esquire.de 
> https://www.esquire.de/life/reisen/schoenste-wasserfaelle-welt-natur
>
> It downloads 94 files, 44 are .jpg files in static.esquire.de/.
>
> TBH, I am not 100% sure what you are trying to do, so excuse me if I am 
> off the track.
> The -p option is for downloading the files you need for displaying a 
> page (e.g. inlined images). If the images are just links, they are not 
> downloaded by -p. In this case, -r -l 1 help. If images that are 
> displayed in the browser are downloaded/displayed by javascript, 
> wget/wget2 won't help you.
>
> Regards, Tim
>
> On 9/13/23 00:14, Elmar Stellnberger wrote:
> > Hi to all!
> >
> >    Today I wanted to download the following web page for means of
> > archiving:
> > https://www.esquire.de/life/reisen/schoenste-wasserfaelle-welt-natur
> >
> >    The following command line did not do what I want:
> > wget -p -N -H -D esquire.de --tries=10 
> > https://www.esquire.de/life/reisen/schoenste-wasserfaelle-welt-natur
> >
> >    The following seemed to do:
> > wget -p -r -N -H -D esquire.de --exclude-domains www.esquire.de 
> > --tries=10 
> > https://www.esquire.de/life/reisen/schoenste-wasserfaelle-welt-natur
> > : files downloaded:
> > now/static.esquire.de/1200x630/smart/images/2023-08/gettyimages-1391653079.jpg
> > now/www.esquire.de/life/reisen/schoenste-wasserfaelle-welt-natur
> > : dld.log:
> > ...
> > BEENDET --2023-09-12 23:18:01--
> > Verstrichene Zeit: 1,2s
> > Geholt: 2 Dateien, 246K in 0,07s (3,62 MB/s)
> > i.e. diz "two files fetched, no error"
> >
> >    Without -r & --exclude-domains it did download 52 files (most of them
> > .js), all from www.esquire.de and none from static.esquire.de. Finally I
> > succeeded to download the images desired by me by á: (here starting from
> > the second file as I did a manual download of the first)
> > grep -o "https://static.esquire.de/[^ ]*\.jpg" 
> > schoenste-wasserfaelle-welt-natur.html | sed -n '2,500/./p' | while 
> > read line; do wget -p "$line"; done
> >
> >    Might (theoretically) be a bug of wget 1.21.4 (1.mga9, i.e. Mageia 9
> > i686) that it did not download more than two files at the second attempt,
> > though that may also be supposed to be a public-avail-silicon fallacy by
> > whomever wants it to assume.
> >
> >    BTW: 'wpdld' is my scriptlet to archive the web pages I read. 
> > Regarding
> > the pages it works for (using wget) I prefer this over a Firefox
> > save-page, as it keeps the web page more or less in pristine state to be
> > mirrored like at the Wayback machine, if necessary. Not to save on disk
> > what I read is something I have experienced that it can be nasty, caus´
> > not every article in news is kept online forever, or be it that it is
> > just deleted from the indexes of search engines (and on-page searches).
> > I would also have 'wpv' for viewing, but alas that isn´t multidomain or
> > non-relative link ready - Hi, what about a make-relative feature of
> > already downloaded web pages on disk for wget2? (would be my desire as I
> > prefer to download non-relative and doing that on disk allows a 'dircmp'
> > (another self-written program to compare (and sync) directories; using it
> > more or less since 2008).)
> >
> > Regards,
> > Elmar Stellnberger