[Bug-wget] [bug #55229] memory exhausted even though stating: Omitting download

[Fenevad Kan]

URL:
  <https://savannah.gnu.org/bugs/?55229>

                 Summary: memory exhausted even though stating: Omitting
download
                 Project: GNU Wget
            Submitted by: fenevadkan
            Submitted on: Sat 15 Dec 2018 09:28:03 AM UTC
                Category: Memory Leak
                Severity: 3 - Normal
                Priority: 5 - Normal
                  Status: None
                 Privacy: Public
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
         Discussion Lock: Any
                 Release: 1.18
        Operating System: GNU/Linux
         Reproducibility: Every Time
           Fixed Release: None
         Planned Release: None
              Regression: None
           Work Required: None
          Patch Included: None

    _______________________________________________________

Details:


Not a classic memory leak, but in practical result it eats up memory because
of wrong behaviour. It can be reproduced with the same memory and command
usage.
 
Host memory: free -tm:
              total        used        free      shared  buff/cache  
available
Mem:           3955         655        2135          10        1164       
3068
Swap:           713         405         308
Total:         4669        1060        2444

#lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.6 (stretch)
Release:        9.6
Codename:       stretch



Real Use Case: trying to mirror/sync Centos ISO images with this command:

wget -nd -m --continue
http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/ -P
/var/www/ISO/  -A "*.iso"


That should download these files:
CentOS-7-x86_64-DVD-1810.iso         
CentOS-7-x86_64-Everything-1810.iso  
CentOS-7-x86_64-LiveGNOME-1810.iso   
CentOS-7-x86_64-LiveKDE-1810.iso     
CentOS-7-x86_64-Minimal-1810.iso     
CentOS-7-x86_64-NetInstall-1810.iso  

In the end it downloaded everything except the last one with Netinstall iso.
>From that point no matter how often I try to repeat the command to continue
downloading it fails because of not enough memory even at checking phase when
there is nothing to be downloaded just yet. As it walks through the older
files and stating omitting the seems to stuck at the
CentOS-7-x86_64-DVD-1810.iso file and I watch free memory in another console
and see how it goes to 0 then the wget:memory exhausted message follows.

Here is the exact output with the additional debug (-d) option:
...attached as a file


wget --version:
GNU Wget 1.18 built on linux-gnu.

-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls 
+ntlm +opie +psl +ssl/gnutls 

Wgetrc: 
    /etc/wgetrc (system)
Locale: 
    /usr/share/locale 
Compile: 
    gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc" 
    -DLOCALEDIR="/usr/share/locale" -I. -I../../src -I../lib 
    -I../../lib -Wdate-time -D_FORTIFY_SOURCE=2 -I/usr/include 
    -I/usr/include/p11-kit-1 -DHAVE_LIBGNUTLS -DNDEBUG -g -O2 
    -fdebug-prefix-map=/build/wget-N5RMXj/wget-1.18=. 
    -fstack-protector-strong -Wformat -Werror=format-security 
    -DNO_SSLv2 -D_FILE_OFFSET_BITS=64 -g -Wall 
Link: 
    gcc -I/usr/include/p11-kit-1 -DHAVE_LIBGNUTLS -DNDEBUG -g -O2 
    -fdebug-prefix-map=/build/wget-N5RMXj/wget-1.18=. 
    -fstack-protector-strong -Wformat -Werror=format-security 
    -DNO_SSLv2 -D_FILE_OFFSET_BITS=64 -g -Wall -Wl,-z,relro -Wl,-z,now 
    -L/usr/lib -lpcre -luuid -lnettle -lgnutls -lz -lpsl -lidn 
    ftp-opie.o gnutls.o http-ntlm.o ../lib/libgnu.a 







    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Sat 15 Dec 2018 09:28:03 AM UTC  Name: wget-memory-exhausted.txt  Size:
18KiB   By: fenevadkan

<http://savannah.gnu.org/bugs/download.php?file_id=45670>

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?55229>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/