Re: delete large list of files

[Raleigh Rinehart]

On 4/4/2011 3:39 PM, Eli Zaretskii wrote:
> > Date: Mon, 04 Apr 2011 14:57:48 -0500
> > From: Raleigh Rinehart<address@hidden>
> >
> > Indeed, this is the very issue I am trying to work around.  The limit
> > seems to be much higher that 256, but It doesn't seem to match the
> > microsoft docs of 32K (for CreateProcess) either.
> I can reliably get 32K long command lines when I invoke commands from
> Make on Windows.
>
> > I'm guessing it is around the limits of cmd.exe's 8192 chars.
> That's if you go via the shell.  But GNU Make does not go through the
> shell if it can invoke the command (rm.exe in this case) directly.
>
> Perhaps you should go back to trying a single "rm files..." command,
> if this is the problem for which you went to this chunky solution.
>
> > Unfortunately, using xargs won't work either.  To do that we would need
> > to write the list of things, files in this instance, to a text file,
> > which of course brings back around to the first problem.  There is no
> > way that I have found to get the list of things, out of make in one
> > chunk (on win32).  Hence the rather esoteric function to break the list
> > up into smaller pieces.
> How long is the original full list of files?

This particular list is 46430 characters (according to wc -c).  So in 
this case it would seem I cannot directly invoke a single rm.   Stuck 
with a chunky solution, or trying to figure a way to break up the clean 
amongst the modules.

The original issue manifested itself several years ago while were still 
on 3.80.   That is when I originally went to a chunky solution.  I don't 
recall the lengths of the lists back then but I'm fairly sure they were 
greater than 8k and less than 32k in length.  Perhaps there was a bug in 
that version?

The original solution was a stop-gap that looked like this:
define remove-files
    $(if $(DEBUG_MAKE), $(call log.echo,removing files in chunks))
    $(RM) $2 $(wordlist 1,75,$1)
    $(if $(wordlist 76,150,$1), $(RM) $2 $(wordlist 76,150,$1))
    $(if $(wordlist 151,225,$1), $(RM) $2 $(wordlist 151,225,$1))
    $(if $(wordlist 226,250,$1), $(RM) $2 $(wordlist 226,250,$1))
    $(if $(wordlist 251,275,$1), $(RM) $2 $(wordlist 251,275,$1))
    $(if $(wordlist 276,$(words $1),$1),$(RM) $2 $(wordlist 276,$(words 
$1),$1))
endef

It worked but wasn't very elegant.  Also, the problem has become more 
pronounced as we have modified our build system in such a way that the 
path's are longer, therefore we would need to modify that function if 
the list grew beyond what we allowed for.  So that's what lead me to try 
something a bit different and more "intelligent".