Re: Problems downloading files on Windows with wget2 2.0.0

[Tim Rühsen]

Just tried
  https://eternallybored.org/imgs/large/255k.png
with a MinGW64 build on Debian Linux (bookworm).

At least that downloads without issues.

+digest +https +ssl/gnutls +ipv6 +iri +large-file -nls -ntlm -opie +psl 
-hsts +iconv +idn2 -zlib -lzma -brotlidec -zstd -bzip2 -lzip -http2 -gpgme

But from what I see in the gnulib sources, the code for the poll() 
function there differentiates between a real Windows build and a MinGW 
build.

In order to reproduce the issue I possibly need more details about how 
you build wget2 (platform and exact command lines).
Please be aware that I do not own a Windows license and thus can not 
test/build on a Windows OS.

Regards, Tim

On 04.10.21 17:26, Jernej Simončič wrote:
> On Sunday, October 3, 2021, 11:21:58, Tim Rühsen wrote:
>
> > while I try to reproduce here (have to build all the deps first), could you 
> > possibly do some more tests to narrow down the issue !?
>
> > a) test with --max-threads=1 --no-http2
> > b) test with --max-threads=1 using a plain text (http://) URL
>
> Not much difference with either.
>
> D:\Stuff\wget\2.0.0\64>wget2 -d --max-threads=1 http://192.168.56.1/bigtest
> 03.135106.352 name=max-threads value=1 invert=0
> 03.135106.353 name=max-threads value=1 invert=0
> 03.135106.354 Local URI encoding = 'CP1250'
> 03.135106.355 Input URI encoding = 'CP1250'
> 03.135106.356 Fetched HSTS data from 
> 'W:\Users\ender\AppData\Local/wget/.wget-hsts'
> 03.135106.358 Fetched HPKP data from 
> 'W:\Users\ender\AppData\Local/wget/.wget-hpkp'
> 03.135106.359 Fetched OCSP hosts from 
> 'W:\Users\ender\AppData\Local/wget/.wget-ocsp_hosts'
> 03.135106.361 Fetched OCSP fingerprints from 
> 'W:\Users\ender\AppData\Local/wget/.wget-ocsp'
> 03.135106.363 set_exit_status(0)
> 03.135106.364 *url =
> 03.135106.364 *3 http://192.168.56.1/bigtest
> 03.135106.365 transcoded 7 bytes from 'utf-8' to 'CP1250'
> 03.135106.365 local filename = 'bigtest'
> 03.135106.366 host_add_job: job fname bigtest
> 03.135106.367 host_add_job: 0xba8f0 http://192.168.56.1/bigtest
> 03.135106.367 host_add_job: qsize 1 host-qsize=1
> 03.135106.368 queue_size: qsize=1
> 03.135106.369 queue_size: qsize=1
> 03.135106.369 [0] action=1 pending=0 host=0x0
> 03.135106.370 dequeue job http://192.168.56.1/bigtest
> 03.135106.372 resolving 192.168.56.1:80...
> 03.135106.373 has 192.168.56.1:80
> 03.135106.374 trying 192.168.56.1:80...
> 03.135106.375 established connection 192.168.56.1
> [0] Downloading 'http://192.168.56.1/bigtest' ...
> 03.135106.378 cookie_create_request_header for host=192.168.56.1 path=bigtest
> 03.135106.379 # sent 232 bytes:
> GET /bigtest HTTP/1.1
> Host: 192.168.56.1
> Accept-Encoding: gzip, deflate, bzip2, xz, lzma, br, zstd, lzip
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> User-Agent: wget2/2.0.0
> Connection: keep-alive
>
> 03.135106.388 [0] action=2 pending=1 host=0xba8a0
> 03.135106.389 ### req 0x32d9090 pending requests = 1
> 03.135106.390 # got header 129 bytes:
> HTTP/1.1 200 OK
> Connection: keep-alive
> Content-Type: application/octet-stream
> Content-Length: 2935293952
> Server: Indy/9.00.10
>
> Saving 'bigtest'
> 03.135106.397 method 2
> Just got 71303168 of 2935293952 bytes
> 03.135106.454 set_exit_status(7)
> HTTP response 200 OK [http://192.168.56.1/bigtest]
> Unexpected body length 71303168.
> 03.135106.457 Removing bigtest
> 03.135106.462 [0] action=1 pending=0 host=0xba8a0
> 03.135106.464 closing connection
> 03.135106.465 [0] action=1 pending=0 host=0x0
> 03.135106.466 main: wake up
> 03.135107.462 [0] action=1 pending=0 host=0x0
> 03.135107.463 dequeue job http://192.168.56.1/bigtest
> 03.135107.465 Found dns cache entry 192.168.56.1:80
> 03.135107.466 trying 192.168.56.1:80...
> 03.135107.467 established connection 192.168.56.1
> [0] Downloading 'http://192.168.56.1/bigtest' ...
> 03.135107.470 cookie_create_request_header for host=192.168.56.1 path=bigtest
> 03.135107.471 # sent 232 bytes:
> GET /bigtest HTTP/1.1
> Host: 192.168.56.1
> Accept-Encoding: gzip, deflate, bzip2, xz, lzma, br, zstd, lzip
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> User-Agent: wget2/2.0.0
> Connection: keep-alive
>
> 03.135107.479 [0] action=2 pending=1 host=0xba8a0
> 03.135107.481 ### req 0xbadc0 pending requests = 1
> 03.135107.483 # got header 129 bytes:
> HTTP/1.1 200 OK
> Connection: keep-alive
> Content-Type: application/octet-stream
> Content-Length: 2935293952
> Server: Indy/9.00.10
>
> Saving 'bigtest'
> 03.135107.488 method 2
> Just got 80871424 of 2935293952 bytes
> HTTP response 200 OK [http://192.168.56.1/bigtest]
> Unexpected body length 80871424.
> 03.135107.598 Removing bigtest
> 03.135107.604 [0] action=1 pending=0 host=0xba8a0
> 03.135107.605 closing connection
> 03.135107.606 [0] action=1 pending=0 host=0x0
> 03.135107.607 main: wake up
> 03.135109.604 [0] action=1 pending=0 host=0x0
> 03.135109.605 dequeue job http://192.168.56.1/bigtest
> 03.135109.606 Found dns cache entry 192.168.56.1:80
> 03.135109.608 trying 192.168.56.1:80...
> 03.135109.609 established connection 192.168.56.1
> [0] Downloading 'http://192.168.56.1/bigtest' ...
> 03.135109.612 cookie_create_request_header for host=192.168.56.1 path=bigtest
> 03.135109.614 # sent 232 bytes:
> GET /bigtest HTTP/1.1
> Host: 192.168.56.1
> Accept-Encoding: gzip, deflate, bzip2, xz, lzma, br, zstd, lzip
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> User-Agent: wget2/2.0.0
> Connection: keep-alive
>
> 03.135109.621 [0] action=2 pending=1 host=0xba8a0
> 03.135109.624 ### req 0xbad00 pending requests = 1
> 03.135109.625 # got header 129 bytes:
> HTTP/1.1 200 OK
> Connection: keep-alive
> Content-Type: application/octet-stream
> Content-Length: 2935293952
> Server: Indy/9.00.10
>
> Saving 'bigtest'
> 03.135109.630 method 2
> Just got 64421888 of 2935293952 bytes
> HTTP response 200 OK [http://192.168.56.1/bigtest]
> Unexpected body length 64421888.
> 03.135109.685 Removing bigtest
> 03.135109.689 [0] action=1 pending=0 host=0xba8a0
> 03.135109.691 closing connection
> 03.135109.692 [0] action=1 pending=0 host=0x0
> 03.135109.693 main: wake up
> 03.135112.701 [0] action=1 pending=0 host=0x0
> 03.135112.702 dequeue job http://192.168.56.1/bigtest
> 03.135112.703 Found dns cache entry 192.168.56.1:80
> 03.135112.705 trying 192.168.56.1:80...
> 03.135112.706 established connection 192.168.56.1
> [0] Downloading 'http://192.168.56.1/bigtest' ...
> 03.135112.708 cookie_create_request_header for host=192.168.56.1 path=bigtest
> 03.135112.710 # sent 232 bytes:
> GET /bigtest HTTP/1.1
> Host: 192.168.56.1
> Accept-Encoding: gzip, deflate, bzip2, xz, lzma, br, zstd, lzip
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> User-Agent: wget2/2.0.0
> Connection: keep-alive
>
> 03.135112.718 [0] action=2 pending=1 host=0xba8a0
> 03.135112.719 ### req 0xbad00 pending requests = 1
> 03.135112.721 # got header 129 bytes:
> HTTP/1.1 200 OK
> Connection: keep-alive
> Content-Type: application/octet-stream
> Content-Length: 2935293952
> Server: Indy/9.00.10
>
> Saving 'bigtest'
> 03.135112.726 method 2
> Just got 83558400 of 2935293952 bytes
> HTTP response 200 OK [http://192.168.56.1/bigtest]
> Unexpected body length 83558400.
> 03.135112.794 Removing bigtest
> 03.135112.800 [0] action=1 pending=0 host=0xba8a0
> 03.135112.801 closing connection
> 03.135112.802 [0] action=1 pending=0 host=0x0
> 03.135112.803 main: wake up
> ^C
>
>
> D:\Stuff\wget\2.0.0\64>wget2 -d --max-threads=1 --no-http2 
> https://eternallybored.org/misc/bigtest
> 03.142042.284 name=max-threads value=1 invert=0
> 03.142042.286 name=http2 value=https://eternallybored.org/misc/bigtest invert=1
> 03.142042.288 name=max-threads value=1 invert=0
> 03.142042.289 name=http2 value=https://eternallybored.org/misc/bigtest invert=1
> 03.142042.291 Local URI encoding = 'CP1250'
> 03.142042.292 Input URI encoding = 'CP1250'
> 03.142042.295 Fetched HSTS data from 
> 'W:\Users\ender\AppData\Local/wget/.wget-hsts'
> 03.142042.297 Fetched HPKP data from 
> 'W:\Users\ender\AppData\Local/wget/.wget-hpkp'
> 03.142042.299 Fetched OCSP hosts from 
> 'W:\Users\ender\AppData\Local/wget/.wget-ocsp_hosts'
> 03.142042.302 Fetched OCSP fingerprints from 
> 'W:\Users\ender\AppData\Local/wget/.wget-ocsp'
> 03.142042.305 set_exit_status(0)
> 03.142042.306 *url =
> 03.142042.306 *3 https://eternallybored.org/misc/bigtest
> 03.142042.308 transcoded 5 bytes from 'utf-8' to 'CP1250'
> 03.142042.309 local filename = 'bigtest'
> 03.142042.310 host_add_job: job fname bigtest
> 03.142042.312 host_add_job: 0x2a940 https://eternallybored.org/misc/bigtest
> 03.142042.313 host_add_job: qsize 1 host-qsize=1
> 03.142042.315 queue_size: qsize=1
> 03.142042.316 queue_size: qsize=1
> 03.142042.317 [0] action=1 pending=0 host=0x0
> 03.142042.318 dequeue job https://eternallybored.org/misc/bigtest
> 03.142042.319 resolving eternallybored.org:443...
> 03.142042.324 has 84.255.206.8:443
> 03.142042.325 trying 84.255.206.8:443...
> Enabling Windows certificate store.
> 03.142042.333 OpenSSL initialized
> 03.142042.335 Sending 'status_request' extension in handshake
> 03.142042.336 No cached TLS session available. Will run a full handshake.
> 03.142042.343 No HPKP pinning found for host 'eternallybored.org'
> 03.142042.345 OCSP URI not given and not found in certificate. Skipping OCSP 
> check for cert 0.
> 03.142042.348 OCSP URI not given and not found in certificate. Skipping OCSP 
> check for cert 1.
> 03.142042.351 No HPKP pinning found for host 'eternallybored.org'
> 03.142042.352 No HPKP pinning found for host 'eternallybored.org'
> 03.142042.354 *** OCSP response status:
> 03.142042.355 successful
> 03.142042.357 No HPKP pinning found for host 'eternallybored.org'
> 03.142042.359 No HPKP pinning found for host 'eternallybored.org'
> 03.142042.360 *** OCSP cert status:
> 03.142042.361 good
> 03.142042.362 Got a stapled OCSP response. Length: 503. Status: OK
> 03.142042.364 Handshake completed (full handshake - not resumed)
> 03.142042.366 TLS session discarded
> 03.142042.367 established connection eternallybored.org
> [0] Downloading 'https://eternallybored.org/misc/bigtest' ...
> 03.142042.370 cookie_create_request_header for host=eternallybored.org 
> path=misc/bigtest
> 03.142042.372 # sent 241 bytes:
> GET /misc/bigtest HTTP/1.1
> Host: eternallybored.org
> Accept-Encoding: gzip, deflate, bzip2, xz, lzma, br, zstd, lzip
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> User-Agent: wget2/2.0.0
> Connection: keep-alive
>
> 03.142042.380 [0] action=2 pending=1 host=0x2a8f0
> 03.142042.381 ### req 0x32c1bc0 pending requests = 1
> 03.142042.382 # got header 381 bytes:
> HTTP/1.1 200 OK
> Date: Sun, 03 Oct 2021 12:20:42 GMT
> Server: Apache
> Strict-Transport-Security: max-age=31536000; includeSubDomains
> Upgrade: h2
> Connection: Upgrade, Keep-Alive
> Last-Modified: Mon, 04 Jul 2016 21:36:33 GMT
> ETag: "21adce70-536d62156d300"
> Accept-Ranges: bytes
> Content-Length: 565038704
> Keep-Alive: timeout=15, max=100
> Content-Type: application/x-7z-compressed
>
> Saving 'bigtest'
> 03.142042.396 method 2
> Failed to read 102400 bytes (24)
> Failed to read -1 bytes (24)
> Just got 16335807 of 565038704 bytes
> 03.142042.520 set_exit_status(7)
> HTTP response 200 OK [https://eternallybored.org/misc/bigtest]
> Unexpected body length 16335807.
> 03.142042.524 Removing bigtest
> 03.142042.526 [0] action=1 pending=0 host=0x2a8f0
> 03.142042.528 closing connection
> 03.142042.529 [0] action=1 pending=0 host=0x0
> 03.142042.530 main: wake up
> 03.142043.527 [0] action=1 pending=0 host=0x0
> 03.142043.528 dequeue job https://eternallybored.org/misc/bigtest
> 03.142043.529 Found dns cache entry eternallybored.org:443
> 03.142043.531 trying 84.255.206.8:443...
> 03.142043.532 Sending 'status_request' extension in handshake
> 03.142043.533 No cached TLS session available. Will run a full handshake.
> 03.142043.540 No HPKP pinning found for host 'eternallybored.org'
> 03.142043.542 OCSP URI not given and not found in certificate. Skipping OCSP 
> check for cert 0.
> 03.142043.544 OCSP URI not given and not found in certificate. Skipping OCSP 
> check for cert 1.
> 03.142043.547 No HPKP pinning found for host 'eternallybored.org'
> 03.142043.548 No HPKP pinning found for host 'eternallybored.org'
> 03.142043.550 *** OCSP response status:
> 03.142043.551 successful
> 03.142043.552 No HPKP pinning found for host 'eternallybored.org'
> 03.142043.554 No HPKP pinning found for host 'eternallybored.org'
> 03.142043.556 *** OCSP cert status:
> 03.142043.557 good
> 03.142043.557 Got a stapled OCSP response. Length: 503. Status: OK
> 03.142043.559 Handshake completed (full handshake - not resumed)
> 03.142043.560 TLS session discarded
> 03.142043.561 established connection eternallybored.org
> [0] Downloading 'https://eternallybored.org/misc/bigtest' ...
> 03.142043.564 cookie_create_request_header for host=eternallybored.org 
> path=misc/bigtest
> 03.142043.566 # sent 241 bytes:
> GET /misc/bigtest HTTP/1.1
> Host: eternallybored.org
> Accept-Encoding: gzip, deflate, bzip2, xz, lzma, br, zstd, lzip
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> User-Agent: wget2/2.0.0
> Connection: keep-alive
>
> 03.142043.575 [0] action=2 pending=1 host=0x2a8f0
> 03.142043.576 ### req 0x32a3350 pending requests = 1
> Failed to read 102400 bytes (24)
> 03.142043.578 host_increase_failure: eternallybored.org failures=1
> 03.142043.580 [0] action=3 pending=1 host=0x2a8f0
> 03.142043.581 closing connection
> 03.142043.583 released job https://eternallybored.org/misc/bigtest
> 03.142043.585 [0] action=1 pending=0 host=0x0
> 03.142043.586 host eternallybored.org is paused 992ms
> 03.142043.587 main: wake up
> 03.142044.588 [0] action=1 pending=0 host=0x0
> 03.142044.589 dequeue job https://eternallybored.org/misc/bigtest
> 03.142044.590 Found dns cache entry eternallybored.org:443
> 03.142044.592 trying 84.255.206.8:443...
> 03.142044.594 Sending 'status_request' extension in handshake
> 03.142044.595 No cached TLS session available. Will run a full handshake.
> 03.142044.604 No HPKP pinning found for host 'eternallybored.org'
> 03.142044.605 OCSP URI not given and not found in certificate. Skipping OCSP 
> check for cert 0.
> 03.142044.608 OCSP URI not given and not found in certificate. Skipping OCSP 
> check for cert 1.
> 03.142044.610 No HPKP pinning found for host 'eternallybored.org'
> 03.142044.612 No HPKP pinning found for host 'eternallybored.org'
> 03.142044.614 *** OCSP response status:
> 03.142044.615 successful
> 03.142044.617 No HPKP pinning found for host 'eternallybored.org'
> 03.142044.618 No HPKP pinning found for host 'eternallybored.org'
> 03.142044.620 *** OCSP cert status:
> 03.142044.621 good
> 03.142044.622 Got a stapled OCSP response. Length: 503. Status: OK
> 03.142044.623 Handshake completed (full handshake - not resumed)
> 03.142044.625 TLS session discarded
> 03.142044.627 established connection eternallybored.org
> [0] Downloading 'https://eternallybored.org/misc/bigtest' ...
> 03.142044.629 cookie_create_request_header for host=eternallybored.org 
> path=misc/bigtest
> 03.142044.631 host_increase_failure: eternallybored.org failures=2
> 03.142044.633 [0] action=3 pending=1 host=0x2a8f0
> 03.142044.635 closing connection
> 03.142044.636 released job https://eternallybored.org/misc/bigtest
> 03.142044.637 [0] action=1 pending=0 host=0x0
> 03.142044.638 host eternallybored.org is paused 1993ms
> 03.142044.640 main: wake up
> 03.142046.633 [0] action=1 pending=0 host=0x0
> 03.142046.635 dequeue job https://eternallybored.org/misc/bigtest
> 03.142046.636 Found dns cache entry eternallybored.org:443
> 03.142046.638 trying 84.255.206.8:443...
> 03.142046.639 Sending 'status_request' extension in handshake
> 03.142046.640 No cached TLS session available. Will run a full handshake.
> 03.142046.648 No HPKP pinning found for host 'eternallybored.org'
> 03.142046.649 OCSP URI not given and not found in certificate. Skipping OCSP 
> check for cert 0.
> 03.142046.652 OCSP URI not given and not found in certificate. Skipping OCSP 
> check for cert 1.
> 03.142046.655 No HPKP pinning found for host 'eternallybored.org'
> 03.142046.656 No HPKP pinning found for host 'eternallybored.org'
> 03.142046.658 *** OCSP response status:
> 03.142046.659 successful
> 03.142046.660 No HPKP pinning found for host 'eternallybored.org'
> 03.142046.663 No HPKP pinning found for host 'eternallybored.org'
> 03.142046.664 *** OCSP cert status:
> 03.142046.665 good
> 03.142046.666 Got a stapled OCSP response. Length: 503. Status: OK
> 03.142046.667 Handshake completed (full handshake - not resumed)
> 03.142046.669 TLS session discarded
> 03.142046.670 established connection eternallybored.org
> [0] Downloading 'https://eternallybored.org/misc/bigtest' ...
> 03.142046.673 cookie_create_request_header for host=eternallybored.org 
> path=misc/bigtest
> 03.142046.675 host_increase_failure: eternallybored.org failures=3
> 03.142046.676 [0] action=3 pending=1 host=0x2a8f0
> 03.142046.678 closing connection
> 03.142046.679 released job https://eternallybored.org/misc/bigtest
> 03.142046.681 [0] action=1 pending=0 host=0x0
> 03.142046.682 host eternallybored.org is paused 2993ms
> 03.142046.683 main: wake up
> ^C

OpenPGP_signature
Description: OpenPGP digital signature