wget2 | Can't read duplicate URLs from STDIN (#587)

[Sean Palmer (@sep332)]

Sean Palmer created an issue: https://gitlab.com/gnuwget/wget2/-/issues/587



I'm using wget to test a web server, specifically reused connections, 
keep-alive, timeouts and so on. My test case is to request the same URL many 
times. Here is the command line I'm having trouble with (note this produces a 
LOT of output on STDERR):

```$ yes "http://127.0.0.1:4001/ping"; | ./wget2 --no-http2 --debug -O - -i - ```

This only makes one request and produces many copies of this:

```
20.155151.926 *url =
20.155151.926 *3 http://127.0.0.1:4001/ping
20.155151.926 [0] action=1 pending=0 host=0x0
20.155151.926 not requesting 'http://127.0.0.1:4001/ping'. (Already Seen)
20.155151.926 queue_size: qsize=0
```
And qsize is always 0. There is no other output until I ctrl-c and it prints 
the stats line:
```
Downloaded: 1 files, 5  bytes, 0 redirects, 0 errors
```
For contrast, if I use the input file ```/dev/stdin```, it works fine. Output 
includes:
```
20.155659.970 *url =
20.155659.970 *3 localhost:4001/ping
20.155659.970 not requesting 'http://localhost:4001/ping'. (Already Seen)
20.155659.970 host_add_job: job fname (null)
20.155659.970 host_add_job: 0x561891cbb440 http://localhost:4001/ping
20.155659.970 host_add_job: qsize 6 host-qsize=6
```
With qsize incrementing as expected, interspersed with messages about actually 
connecting and getting data.

I found a couple of changes that fix this, but I don't have enough experience 
with the project to know if either of these would cause other problems. One is 
to add ```URL_FLG_NO_BLACKLISTING``` to the queue_url_from_local() call in 
wget.c line 1310. The other is to remove the whole ```if (isatty(STDIN_FILENO)) 
{...}``` block and always do what's in the else block.

-- 
Reply to this email directly or view it on GitLab: 
https://gitlab.com/gnuwget/wget2/-/issues/587
You're receiving this email because of your account on gitlab.com.