Re: [bug-gawk] Gawk feature request - file list

[Nelson H. F. Beebe]

Daniele Zanoli <address@hidden> wrote on Wed, 1 Feb 2017
13:01:20 +0100 about a problem in processing large numbers of files in
gawk running on Microsoft Windows, which allocates only 32KB for
storage of a command line.

One solution that he proposed is a new command-line option in gawk to
supply a list of files in a file that gawk would read and treat as if
they had been explicitly supplied on the command line.  Others showed
that you can do this already in gawk by assignment to the ARGV[]
array.

However, the problem of limited command-line length also hits Unix
systems where the limits are larger, but still finite, and is not
specific to gawk, but affects any program that needs to process large
numbers of files.

The solution in the Unix world is the pair of utilities, find and
xargs.  The first is used to select files by metadata attributes or by
filename patterns, and produce a list of those names on stdout. The
xargs tool then reads that list, and runs a command supplied on xargs'
own command line, supplying just enough files to avoid hitting the
nasty ARG_MAX limit.

Thus, in a Cygwin bash terminal on Microsoft Windows, I would run
something like this:

        find /cygdrive/c/data -name '*.dat' | xargs gawk -f myprog.awk

Windows 10 itself does not have xargs, but Cygwin is free, and once
installed, provides about 95% of the power of Unix on top of the GUI
rich, but tool poor, Windows operating system.  Visit
http://cygwin.com/ for download and installation details.  I maintain
a Web page about it at

        http://www.math.utah.edu/~beebe/gnu-on-windows.html

-------------------------------------------------------------------------------
- Nelson H. F. Beebe                    Tel: +1 801 581 5254                  -
- University of Utah                    FAX: +1 801 581 4148                  -
- Department of Mathematics, 110 LCB    Internet e-mail: address@hidden  -
- 155 S 1400 E RM 233                       address@hidden  address@hidden -
- Salt Lake City, UT 84112-0090, USA    URL: http://www.math.utah.edu/~beebe/ -
-------------------------------------------------------------------------------