Re: [Bug-tar] Small bug in tar-1.29

[Jack Howarth]

On Tue, Sep 06, 2016 at 09:01:17PM -0400, Jack Howarth wrote:
> On Tue, Sep 06, 2016 at 08:24:21PM -0400, Jack Howarth wrote:
> > On Tue, Sep 06, 2016 at 06:48:23PM -0400, Jack Howarth wrote:
> > > On Tue, Sep 06, 2016 at 04:12:12PM -0400, Jack Howarth wrote:
> > > > On Tue, Sep 06, 2016 at 08:12:05PM +0100, Ralph Corderoy wrote:
> > > > > Hi Jack,
> > > > > 
> > > > > > > gnutar: Child died with signal 11
> > > > > ...
> > > > > > stop reason = signal SIGPIPE
> > > > > 
> > > > > SIGPIPE is 13.  We want 11, SIGSEGV.  Perhaps a "c" or two for 
> > > > > continue
> > > > > will have the debugger carry on until the SEGV bites.
> > > > 
> > > > $ ulimit -c unlimited
> > > > $ gnutar -zcvf test_data.tar.gz test_data 
> > > > test_data/
> > > > test_data/.noe.tbl.swp
> > > > test_data/dihedral.tbl
> > > > test_data/file.nam
> > > > test_data/io.mc
> > > > test_data/noe.tbl
> > > > gnutar: test_data.tar.gz: Cannot write: Broken pipe
> > > > gnutar: Child died with signal 11
> > > > gnutar: Error is not recoverable: exiting now
> > > > $ lldb -c /cores/core.21601 `which gnutar`
> > > > (lldb) target create "/opt/local/bin/gnutar" --core "/cores/core.21601"
> > > > warning: (x86_64) /cores/core.21601 load command 84 LC_SEGMENT_64 has a 
> > > > fileoff + filesize (0x296fe000) that extends beyond the end of the file 
> > > > (0x296fd000), the segment will be truncated to match
> > > > Core file '/cores/core.21601' (x86_64) was loaded.
> > > > (lldb) bt
> > > > * thread #1: tid = 0x0000, 0x00007fff94ebb6d7 
> > > > libdispatch.dylib`_dispatch_queue_push_queue + 463, stop reason = 
> > > > signal SIGSTOP
> > > >   * frame #0: 0x00007fff94ebb6d7 
> > > > libdispatch.dylib`_dispatch_queue_push_queue + 463
> > > >     frame #1: 0x00007fff7478c4d0 
> > > > libdispatch.dylib`_OS_dispatch_mach_vtable + 48
> > > >     frame #2: 0x00007fff94eb9b06 
> > > > libdispatch.dylib`_dispatch_queue_wakeup_with_qos_slow + 126
> > > >     frame #3: 0x00007fff94ec013f 
> > > > libdispatch.dylib`_dispatch_mach_msg_send + 1952
> > > >     frame #4: 0x00007fff94ebf8dc libdispatch.dylib`dispatch_mach_send + 
> > > > 262
> > > >     frame #5: 0x00007fff89448fc9 
> > > > libxpc.dylib`xpc_connection_send_message_with_reply + 131
> > > >     frame #6: 0x00007fff8e31ebdf 
> > > > CoreFoundation`__66-[CFPrefsSearchListSource 
> > > > generationCountFromListOfSources:count:]_block_invoke_2 + 143
> > > >     frame #7: 0x00007fff8e31ea0d 
> > > > CoreFoundation`_CFPrefsWithDaemonConnection + 381
> > > >     frame #8: 0x00007fff8e31db96 
> > > > CoreFoundation`__66-[CFPrefsSearchListSource 
> > > > generationCountFromListOfSources:count:]_block_invoke + 150
> > > >     frame #9: 0x00007fff8e31d933 
> > > > CoreFoundation`-[CFPrefsSearchListSource 
> > > > generationCountFromListOfSources:count:] + 179
> > > >     frame #10: 0x00007fff8e31d214 
> > > > CoreFoundation`-[CFPrefsSearchListSource alreadylocked_copyDictionary] 
> > > > + 324
> > > >     frame #11: 0x00007fff8e31ce5c 
> > > > CoreFoundation`-[CFPrefsSearchListSource 
> > > > alreadylocked_copyValueForKey:] + 60
> > > >     frame #12: 0x00007fff8e31cdec 
> > > > CoreFoundation`___CFPreferencesCopyAppValueWithContainer_block_invoke + 
> > > > 60
> > > >     frame #13: 0x00007fff8e314b10 
> > > > CoreFoundation`+[CFPrefsSearchListSource 
> > > > withSearchListForIdentifier:container:perform:] + 608
> > > >     frame #14: 0x00007fff8e314867 
> > > > CoreFoundation`_CFPreferencesCopyAppValueWithContainer + 183
> > > >     frame #15: 0x00000001021a6538 
> > > > libintl.8.dylib`_nl_language_preferences_default + 70
> > > >     frame #16: 0x00000001021a462f libintl.8.dylib`libintl_dcigettext + 
> > > > 667
> > > >     frame #17: 0x00000001021491f2 gnutar`sys_child_open_for_compress + 
> > > > 125
> > > >     frame #18: 0x000000010213011f gnutar`open_archive + 603
> > > >     frame #19: 0x00000001021355ab gnutar`create_archive + 67
> > > >     frame #20: 0x000000010214b494 gnutar`main + 3510
> > > >     frame #21: 0x00007fff895605ad libdyld.dylib`start + 1
> > > >     frame #22: 0x00007fff895605ad libdyld.dylib`start + 1
> > > > (lldb)
> > > > 
> > > 
> > > Using the same approach that I took in 
> > > http://savannah.gnu.org/bugs/?46261 to determine that the threading in 
> > > the CoreFoundation
> > > framework linked through libintl was causing breakage on darwin15 in make 
> > > 4.0/4,1...
> > > 
> > > rebuilding gettext with the additional configure options...
> > > 
> > > gt_cv_func_CFPreferencesCopyAppValue=no \ 
> > > gt_cv_func_CFLocaleCopyCurrent=no \ 
> > > gt_cv_func_CFPreferencesCopyAppValue=no \ 
> > > gt_cv_func_CFLocaleCopyCurrent=no 
> > > 
> > > to change the linkage from....
> > > 
> > > $ otool -L /opt/local/lib/libintl.8.dylib
> > > /opt/local/lib/libintl.8.dylib:
> > >   /opt/local/lib/libintl.8.dylib (compatibility version 10.0.0, current 
> > > version 10.5.0)
> > >   /opt/local/lib/libiconv.2.dylib (compatibility version 8.0.0, current 
> > > version 8.1.0)
> > >   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current 
> > > version 1226.10.1)
> > >   
> > > /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
> > >  (compatibility version 150.0.0, current version 1258.1.0)
> > > 
> > > $ otool -L /opt/local/lib/libintl.8.dylib
> > > /opt/local/lib/libintl.8.dylib:
> > >   /opt/local/lib/libintl.8.dylib (compatibility version 10.0.0, current 
> > > version 10.5.0)
> > >   /opt/local/lib/libiconv.2.dylib (compatibility version 8.0.0, current 
> > > version 8.1.0)
> > >   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current 
> > > version 1226.10.1)
> > > 
> > > does indeed suppress the bug in tar 1.29 when LANG is unset. In the case 
> > > of make, this was fixed by
> > > the addition of pselect() calls to protect the threads before forking...
> > > 
> > > http://git.savannah.gnu.org/cgit/make.git/commit/?id=85c788572d054bc2c41b84007875edbd37ad3ed5
> > > 
> > > [SV 46261] Use pselect() for jobserver where supported.
> > > * Makefile.am, configure.ac: Check for pselect() and sys/select.h.
> > > * main.c (main): Block SIGCHLD if we have pselect() support.
> > > * posixos.c (jobserver_acquire): If we support pselect() then use
> > > it to query the jobserver pipe, while also listening for SIGCHLD.
> > > Also pselect() supports a timeout so avoid alarm() calls.
> > > 
> > > Perhaps something similar needs to be done for the xpipe() call in tar 
> > > 1.29?
> > 
> > I think that 
> > http://git.savannah.gnu.org/cgit/tar.git/commit/?id=589ba77faf60e4632771d80e2b25d9e3778d4a3f
> > 
> > Catch compressor execution errors.
> > * src/misc.c (write_fatal_details): Move to buffer.c
> > * src/buffer.c (write_fatal_details): Close the archive and wait for
> > the compressor program to terminate in order to catch eventual execution
> > errors.
> > * src/system.c (sys_child_open_for_compress): Ignore SIGPIPE so tar will
> > not silently terminate when unable to write to the compressor.
> > * tests/comperr.at: New file.
> > * tests/Makefile.am: Add comperr.at
> > * tests/testsuite.at: Include comperr.at
> > 
> > seems a likely candidate for code change causing this threading conflict 
> > with CoreFoundation.
> > In particular, the change...
> > 
> > diff --git a/src/system.c b/src/system.c
> > index 9414233..dc0b2b0 100644
> > --- a/src/system.c
> > +++ b/src/system.c
> > @@ -330,6 +330,7 @@ sys_child_open_for_compress (void)
> >    pid_t grandchild_pid;
> >    pid_t child_pid;
> >  
> > +  signal (SIGPIPE, SIG_IGN);
> >    xpipe (parent_pipe);
> >    child_pid = xfork ();
> >  
> > Perhaps pselect() could be used to fix this on darwin?
> 
> Confirmed with tar from git using...
> 
> git checkout d02c81df15e3ac3764c654f1e3bf24b5f45b3ed7
> ./configure --without-xattrs  CPPFLAGS="-I/sw/include -I/usr/lib" 
> LDFLAGS="-L/sw/lib -L/usr/lib"
> make
> 
> that the failure with tar 1.29git when LANG is unset doesn't exist at....
> 
> commit d02c81df15e3ac3764c654f1e3bf24b5f45b3ed7
> Author: Sergey Poznyakoff <address@hidden>
> Date:   Sun Nov 29 20:51:08 2015 +0200
> 
>     Fix extraction from concatenated incremental archives with renamed 
> directories.
>     
>     Complements 15c02c2b.
>     
>     * src/extract.c (delayed_set_stat): Change type of file_name.
>     (delay_set_stat): Allocate file_name member.
>     (free_delayed_set_stat): Free file_name.
>     (fixup_delayed_set_stat): New function.
>     (rename_directory): Call fixup_delayed_set_stat on success.
>     
>     * tests/incr11.at: New testcase.
>     * tests/incr10.at: Improve description.
>     * tests/Makefile.am: Add incr11.at
>     * tests/testsuite.at: Add incr11.at
> 
> but using...
> 
> git checkout 589ba77faf60e4632771d80e2b25d9e3778d4a3f
> ./configure --without-xattrs  CPPFLAGS="-I/sw/include -I/usr/lib" 
> LDFLAGS="-L/sw/lib -L/usr/lib"
> make 
> 
> produces the failures now seen in tar 1.29 when LANG is unset so the 
> offending commits are...
> 
> commit 589ba77faf60e4632771d80e2b25d9e3778d4a3f
> Author: Sergey Poznyakoff <address@hidden>
> Date:   Sat Dec 5 16:54:26 2015 +0200
> 
>     Catch compressor execution errors.
>     
>     * src/misc.c (write_fatal_details): Move to buffer.c
>     * src/buffer.c (write_fatal_details): Close the archive and wait for
>     the compressor program to terminate in order to catch eventual execution
>     errors.
>     * src/system.c (sys_child_open_for_compress): Ignore SIGPIPE so tar will
>     not silently terminate when unable to write to the compressor.
>     * tests/comperr.at: New file.
>     * tests/Makefile.am: Add comperr.at
>     * tests/testsuite.at: Include comperr.at
> 
> commit 6ea9e62bb398b2f81e52f671749e585367286820
> Author: Sergey Poznyakoff <address@hidden>
> Date:   Sat Dec 5 08:48:03 2015 +0200
> 
>     Upgrade paxutils
> 
> which produces issues with CoreFoundation's threading out of the libintl 
> linkage on tar.

To further clarify...

git checkout 6ea9e62bb398b2f81e52f671749e585367286820
./configure --without-xattrs  CPPFLAGS="-I/sw/include -I/usr/lib" 
LDFLAGS="-L/sw/lib -L/usr/lib"
make 

produces a tar which doesn't suffer from the bug so commit 
589ba77faf60e4632771d80e2b25d9e3778d4a3f
is the sole cause of this bug.
> 
> > 
> > > > 
> > > > > 
> > > > > -- 
> > > > > Cheers, Ralph.
> > > > > https://plus.google.com/+RalphCorderoy