RE: some concern about the fix of " tail: consistently output all data for truncated files"

[Lian, George (Nokia - CN/Hangzhou)]

Hi,
>> Add one more suggestion, if we have not a perfect solution to consider all 
>> the case of truncate, could we add an option to tail, such like tail 
>> -no-truncate
>> If tail run with this option, than application not consider any truncate 
>> case.
>> 
>> For example, I suppose syslog output file will not have any truncate case in 
>> our environment, then the tail could use the option to avoid the 
>> mis-truncated case?

>Note for case 2) above, we only update fspec->size _after_ the read,
>so I'm not sure how practical the race with reading a _smaller_ st_size after 
>that is?
>I.E. the heuristic is fairly good I think,
>so an option may be overkill.
>We'd have to see a demonstratable issue to consider such an option.

We have an issue now for tail a syslog file which stored in a network-based 
file system. A automated cased need tail the syslog about one hour to get the 
syslog of that period,
in that period of one hour , happen 6 times of  un-expected file truncated 
issue, so the output of tail has 6 times full syslog file, so the output file 
is so huge and eat all of the disks.
The network-based file system maybe not so easy to change to meet the current 
implement of "tail" application.
So I need helps from yours :)

And which your mean for demonstratable?  The issue we encounter could be easy 
to reproduce, maybe the file-system is not so strict like ext4 file system, 
but I still suggest "tail" application could do some change to adapt this kinds 
network-based file system?


Thanks & Best Regards,
George


-----Original Message-----
From: Pádraig Brady [mailto:address@hidden] 
Sent: Monday, November 07, 2016 11:06 PM
To: Lian, George (Nokia - CN/Hangzhou) <address@hidden>; address@hidden
Cc: Zhang, Bingxuan (Nokia - CN/Hangzhou) <address@hidden>; Li, Deqian (Nokia - 
CN/Hangzhou) <address@hidden>; Zizka, Jan (Nokia - CZ/Prague) <address@hidden>; 
Bao, Xiaohui (Nokia - CN/Hangzhou) <address@hidden>
Subject: Re: some concern about the fix of " tail: consistently output all data 
for truncated files"

On 07/11/16 07:18, Lian, George (Nokia - CN/Hangzhou) wrote:
> From: Lian, George (Nokia - CN/Hangzhou)
> Sent: Monday, November 07, 2016 1:17 PM
> To: 'address@hidden' <address@hidden>
> Cc: Zizka, Jan (Nokia - CZ/Prague) <address@hidden>; Zhang, Bingxuan (Nokia - 
> CN/Hangzhou) <address@hidden>; Li, Deqian (Nokia - CN/Hangzhou) 
> <address@hidden>; Bao, Xiaohui (Nokia - CN/Hangzhou) <address@hidden>
> Subject: some concern about the fix of " tail: consistently output all data 
> for truncated files"
> 
> 
> Hi,
> 
> There are a fix about truncated for tail application of coreutils,
> 
> The change messages is "tail: consistently output all data for truncated 
> files",
> 
> Now I have some concern for this case
> 
> 1)      I suppose Truncate maybe not always truncate to ZERO, for example, 
> sometimes a file maybe only has been truncate the bottom 100 lines of 1M 
> lines , in this case , SEEK from head will be confused by the users.
> 2)      Sometimes truncated maybe not really happen even "stats.st_size < 
> fspec->size" is meet,  for example, a network based filesystems, for it's 
> current implementation, it will active the read data-cache first and then 
> update the meta-data cache, it is not an atomic operation, I suppose it is 
> reasonable that the "tail application" will read the data-cache with new 
> updated data, but get stat size with old value, in this case, the tail assume 
> it is truncated and dump the data from the begin, it also make more confuse 
> to users.
> 3)      As the comments in your changes, the stat size is bigger thand the 
> length has read, it also maybe happ truncate.
> 4)      Why can't get size of fstat first, and read not more than the size of 
> fstat during the tail_forever loop?
> 
> Could you please share your comments for the above concerns?
> Hi,
> 
> Add one more suggestion, if we have not a perfect solution to consider all 
> the case of truncate, could we add an option to tail, such like tail 
> -no-truncate
> If tail run with this option, than application not consider any truncate case.
> 
> For example, I suppose syslog output file will not have any truncate case in 
> our environment, then the tail could use the option to avoid the 
> mis-truncated case?

Note for case 2) above, we only update fspec->size _after_ the read,
so I'm not sure how practical the race with reading a _smaller_ st_size after 
that is?
I.E. the heuristic is fairly good I think,
so an option may be overkill.
We'd have to see a demonstratable issue to consider such an option.

thanks,
Pádraig