RE: [EXTERNAL] Re: New command consideration - curtail

[Wolaver, David]

>On 12/11/18 06:03, Wolaver, David wrote:
>> On 08/11/18 09:02, Wolaver, David wrote:
>>> Hi all,
>>>
>>> I have developed a command line program that I would like to have 
>>> considered >>> for inclusion in a future revision of coreutils.  The name 
>>> of the program is "curtail".  It reads stdin and appends to an output file 
>>> while ensuring that the file will never exceed a specified maximum size.  
>>> The main intent is to place a bound on the output of a program such that it 
>>> cannot create a runaway log file.  The output file is truncated (in 
>>> multiple of block size) at the beginning such that the output file always 
>>> contains the latest output of the program.
>>>
>>> user_program | curtail -s 4M output.txt
>> 
>> It's a perfect name for such a command anyway :)
>> 
>> Does this depend on FALLOC_FL_COLLAPSE_RANGE support.
>> Would this command fail on file systems not supporting this?
>> Being block based seems a little problematic, especially considering 
>> multi-byte characters, but maybe that's OK.
>> 
>> cheers,
>> Pádraig
>> 
>> Hi Pádraig,
>> 
>> Thanks for spending some time to think about curtail.  I purposely omitted 
>> implementation details from the first post because they are irrelevant 
>> unless the following can be established...
>> 
>> 1. There exists problem(s) that can be addressed by such a program (I 
>> have only presented one scenario) 2. Consensus that existing 
>> coreutils/scripts cannot produce the same result - at least not in an 
>> efficient manner 3. A solution to the problem belongs in coreutils
>> 
>> To answer your question.. yes FALLOC_FL_COLLAPSE_RANGE is the mechanism by 
>> which files are truncated in the Linux implementation.  Any OS/filesystem 
>> can be supported if it provides a mechanism to query block size and remove 
>> blocks from the beginning of a file.
>
>Implementation details do consolidate things in my mind at least.
>They're important as coreutils tend to be general in that they'll fallback to 
>more generic solutions where system support for certain features is lacking.
>
>As for the need for a separate command, playing devil's advocate, given that 
>this is not just a view of the data (like tail(1)), and controls the eviction 
>of older data, perhaps this functionality is less generally useful, and more 
>suited as an operation mode of logrotate etc.
>
>cheers,
>Pádraig

It would be great to have this generally supported, but that will requires more 
OS/filesystem support of primitives that make it possible.  Why hold back a 
useful feature from the users that could benefit from it today?

As for logrotate, this is typically run as a cron job instead of being used in 
a pipeline.  With that being said, I'm not adverse to including it in other 
programs - it already creates a lib specifically for this purpose.  I really 
just want to make the feature available for others to benefit from.

-Dave