Re: [Bug-tar] Small bug in tar-1.29

[Jack Howarth]

On Tue, Sep 06, 2016 at 09:30:02PM -0400, Jack Howarth wrote:
> On Tue, Sep 06, 2016 at 09:01:17PM -0400, Jack Howarth wrote:
> > On Tue, Sep 06, 2016 at 08:24:21PM -0400, Jack Howarth wrote:
> > > On Tue, Sep 06, 2016 at 06:48:23PM -0400, Jack Howarth wrote:
> > > > On Tue, Sep 06, 2016 at 04:12:12PM -0400, Jack Howarth wrote:
> > > > > On Tue, Sep 06, 2016 at 08:12:05PM +0100, Ralph Corderoy wrote:
> > > > > > Hi Jack,
> > > > > > 
> > > > > > > > gnutar: Child died with signal 11
> > > > > > ...
> > > > > > > stop reason = signal SIGPIPE
> > > > > > 
> > > > > > SIGPIPE is 13.  We want 11, SIGSEGV.  Perhaps a "c" or two for 
> > > > > > continue
> > > > > > will have the debugger carry on until the SEGV bites.
> > > > > 
> > > > > $ ulimit -c unlimited
> > > > > $ gnutar -zcvf test_data.tar.gz test_data 
> > > > > test_data/
> > > > > test_data/.noe.tbl.swp
> > > > > test_data/dihedral.tbl
> > > > > test_data/file.nam
> > > > > test_data/io.mc
> > > > > test_data/noe.tbl
> > > > > gnutar: test_data.tar.gz: Cannot write: Broken pipe
> > > > > gnutar: Child died with signal 11
> > > > > gnutar: Error is not recoverable: exiting now
> > > > > $ lldb -c /cores/core.21601 `which gnutar`
> > > > > (lldb) target create "/opt/local/bin/gnutar" --core 
> > > > > "/cores/core.21601"
> > > > > warning: (x86_64) /cores/core.21601 load command 84 LC_SEGMENT_64 has 
> > > > > a fileoff + filesize (0x296fe000) that extends beyond the end of the 
> > > > > file (0x296fd000), the segment will be truncated to match
> > > > > Core file '/cores/core.21601' (x86_64) was loaded.
> > > > > (lldb) bt
> > > > > * thread #1: tid = 0x0000, 0x00007fff94ebb6d7 
> > > > > libdispatch.dylib`_dispatch_queue_push_queue + 463, stop reason = 
> > > > > signal SIGSTOP
> > > > >   * frame #0: 0x00007fff94ebb6d7 
> > > > > libdispatch.dylib`_dispatch_queue_push_queue + 463
> > > > >     frame #1: 0x00007fff7478c4d0 
> > > > > libdispatch.dylib`_OS_dispatch_mach_vtable + 48
> > > > >     frame #2: 0x00007fff94eb9b06 
> > > > > libdispatch.dylib`_dispatch_queue_wakeup_with_qos_slow + 126
> > > > >     frame #3: 0x00007fff94ec013f 
> > > > > libdispatch.dylib`_dispatch_mach_msg_send + 1952
> > > > >     frame #4: 0x00007fff94ebf8dc libdispatch.dylib`dispatch_mach_send 
> > > > > + 262
> > > > >     frame #5: 0x00007fff89448fc9 
> > > > > libxpc.dylib`xpc_connection_send_message_with_reply + 131
> > > > >     frame #6: 0x00007fff8e31ebdf 
> > > > > CoreFoundation`__66-[CFPrefsSearchListSource 
> > > > > generationCountFromListOfSources:count:]_block_invoke_2 + 143
> > > > >     frame #7: 0x00007fff8e31ea0d 
> > > > > CoreFoundation`_CFPrefsWithDaemonConnection + 381
> > > > >     frame #8: 0x00007fff8e31db96 
> > > > > CoreFoundation`__66-[CFPrefsSearchListSource 
> > > > > generationCountFromListOfSources:count:]_block_invoke + 150
> > > > >     frame #9: 0x00007fff8e31d933 
> > > > > CoreFoundation`-[CFPrefsSearchListSource 
> > > > > generationCountFromListOfSources:count:] + 179
> > > > >     frame #10: 0x00007fff8e31d214 
> > > > > CoreFoundation`-[CFPrefsSearchListSource 
> > > > > alreadylocked_copyDictionary] + 324
> > > > >     frame #11: 0x00007fff8e31ce5c 
> > > > > CoreFoundation`-[CFPrefsSearchListSource 
> > > > > alreadylocked_copyValueForKey:] + 60
> > > > >     frame #12: 0x00007fff8e31cdec 
> > > > > CoreFoundation`___CFPreferencesCopyAppValueWithContainer_block_invoke 
> > > > > + 60
> > > > >     frame #13: 0x00007fff8e314b10 
> > > > > CoreFoundation`+[CFPrefsSearchListSource 
> > > > > withSearchListForIdentifier:container:perform:] + 608
> > > > >     frame #14: 0x00007fff8e314867 
> > > > > CoreFoundation`_CFPreferencesCopyAppValueWithContainer + 183
> > > > >     frame #15: 0x00000001021a6538 
> > > > > libintl.8.dylib`_nl_language_preferences_default + 70
> > > > >     frame #16: 0x00000001021a462f libintl.8.dylib`libintl_dcigettext 
> > > > > + 667
> > > > >     frame #17: 0x00000001021491f2 gnutar`sys_child_open_for_compress 
> > > > > + 125
> > > > >     frame #18: 0x000000010213011f gnutar`open_archive + 603
> > > > >     frame #19: 0x00000001021355ab gnutar`create_archive + 67
> > > > >     frame #20: 0x000000010214b494 gnutar`main + 3510
> > > > >     frame #21: 0x00007fff895605ad libdyld.dylib`start + 1
> > > > >     frame #22: 0x00007fff895605ad libdyld.dylib`start + 1
> > > > > (lldb)
> > > > > 
> > > > 
> > > > Using the same approach that I took in 
> > > > http://savannah.gnu.org/bugs/?46261 to determine that the threading in 
> > > > the CoreFoundation
> > > > framework linked through libintl was causing breakage on darwin15 in 
> > > > make 4.0/4,1...
> > > > 
> > > > rebuilding gettext with the additional configure options...
> > > > 
> > > > gt_cv_func_CFPreferencesCopyAppValue=no \ 
> > > > gt_cv_func_CFLocaleCopyCurrent=no \ 
> > > > gt_cv_func_CFPreferencesCopyAppValue=no \ 
> > > > gt_cv_func_CFLocaleCopyCurrent=no 
> > > > 
> > > > to change the linkage from....
> > > > 
> > > > $ otool -L /opt/local/lib/libintl.8.dylib
> > > > /opt/local/lib/libintl.8.dylib:
> > > >         /opt/local/lib/libintl.8.dylib (compatibility version 10.0.0, 
> > > > current version 10.5.0)
> > > >         /opt/local/lib/libiconv.2.dylib (compatibility version 8.0.0, 
> > > > current version 8.1.0)
> > > >         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, 
> > > > current version 1226.10.1)
> > > >         
> > > > /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
> > > >  (compatibility version 150.0.0, current version 1258.1.0)
> > > > 
> > > > $ otool -L /opt/local/lib/libintl.8.dylib
> > > > /opt/local/lib/libintl.8.dylib:
> > > >         /opt/local/lib/libintl.8.dylib (compatibility version 10.0.0, 
> > > > current version 10.5.0)
> > > >         /opt/local/lib/libiconv.2.dylib (compatibility version 8.0.0, 
> > > > current version 8.1.0)
> > > >         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, 
> > > > current version 1226.10.1)
> > > > 
> > > > does indeed suppress the bug in tar 1.29 when LANG is unset. In the 
> > > > case of make, this was fixed by
> > > > the addition of pselect() calls to protect the threads before forking...
> > > > 
> > > > http://git.savannah.gnu.org/cgit/make.git/commit/?id=85c788572d054bc2c41b84007875edbd37ad3ed5
> > > > 
> > > > [SV 46261] Use pselect() for jobserver where supported.
> > > > * Makefile.am, configure.ac: Check for pselect() and sys/select.h.
> > > > * main.c (main): Block SIGCHLD if we have pselect() support.
> > > > * posixos.c (jobserver_acquire): If we support pselect() then use
> > > > it to query the jobserver pipe, while also listening for SIGCHLD.
> > > > Also pselect() supports a timeout so avoid alarm() calls.
> > > > 
> > > > Perhaps something similar needs to be done for the xpipe() call in tar 
> > > > 1.29?
> > > 
> > > I think that 
> > > http://git.savannah.gnu.org/cgit/tar.git/commit/?id=589ba77faf60e4632771d80e2b25d9e3778d4a3f
> > > 
> > > Catch compressor execution errors.
> > > * src/misc.c (write_fatal_details): Move to buffer.c
> > > * src/buffer.c (write_fatal_details): Close the archive and wait for
> > > the compressor program to terminate in order to catch eventual execution
> > > errors.
> > > * src/system.c (sys_child_open_for_compress): Ignore SIGPIPE so tar will
> > > not silently terminate when unable to write to the compressor.
> > > * tests/comperr.at: New file.
> > > * tests/Makefile.am: Add comperr.at
> > > * tests/testsuite.at: Include comperr.at
> > > 
> > > seems a likely candidate for code change causing this threading conflict 
> > > with CoreFoundation.
> > > In particular, the change...
> > > 
> > > diff --git a/src/system.c b/src/system.c
> > > index 9414233..dc0b2b0 100644
> > > --- a/src/system.c
> > > +++ b/src/system.c
> > > @@ -330,6 +330,7 @@ sys_child_open_for_compress (void)
> > >    pid_t grandchild_pid;
> > >    pid_t child_pid;
> > >  
> > > +  signal (SIGPIPE, SIG_IGN);
> > >    xpipe (parent_pipe);
> > >    child_pid = xfork ();
> > >  
> > > Perhaps pselect() could be used to fix this on darwin?
> > 
> > Confirmed with tar from git using...
> > 
> > git checkout d02c81df15e3ac3764c654f1e3bf24b5f45b3ed7
> > ./configure --without-xattrs  CPPFLAGS="-I/sw/include -I/usr/lib" 
> > LDFLAGS="-L/sw/lib -L/usr/lib"
> > make
> > 
> > that the failure with tar 1.29git when LANG is unset doesn't exist at....
> > 
> > commit d02c81df15e3ac3764c654f1e3bf24b5f45b3ed7
> > Author: Sergey Poznyakoff <address@hidden>
> > Date:   Sun Nov 29 20:51:08 2015 +0200
> > 
> >     Fix extraction from concatenated incremental archives with renamed 
> > directories.
> >     
> >     Complements 15c02c2b.
> >     
> >     * src/extract.c (delayed_set_stat): Change type of file_name.
> >     (delay_set_stat): Allocate file_name member.
> >     (free_delayed_set_stat): Free file_name.
> >     (fixup_delayed_set_stat): New function.
> >     (rename_directory): Call fixup_delayed_set_stat on success.
> >     
> >     * tests/incr11.at: New testcase.
> >     * tests/incr10.at: Improve description.
> >     * tests/Makefile.am: Add incr11.at
> >     * tests/testsuite.at: Add incr11.at
> > 
> > but using...
> > 
> > git checkout 589ba77faf60e4632771d80e2b25d9e3778d4a3f
> > ./configure --without-xattrs  CPPFLAGS="-I/sw/include -I/usr/lib" 
> > LDFLAGS="-L/sw/lib -L/usr/lib"
> > make 
> > 
> > produces the failures now seen in tar 1.29 when LANG is unset so the 
> > offending commits are...
> > 
> > commit 589ba77faf60e4632771d80e2b25d9e3778d4a3f
> > Author: Sergey Poznyakoff <address@hidden>
> > Date:   Sat Dec 5 16:54:26 2015 +0200
> > 
> >     Catch compressor execution errors.
> >     
> >     * src/misc.c (write_fatal_details): Move to buffer.c
> >     * src/buffer.c (write_fatal_details): Close the archive and wait for
> >     the compressor program to terminate in order to catch eventual execution
> >     errors.
> >     * src/system.c (sys_child_open_for_compress): Ignore SIGPIPE so tar will
> >     not silently terminate when unable to write to the compressor.
> >     * tests/comperr.at: New file.
> >     * tests/Makefile.am: Add comperr.at
> >     * tests/testsuite.at: Include comperr.at
> > 
> > commit 6ea9e62bb398b2f81e52f671749e585367286820
> > Author: Sergey Poznyakoff <address@hidden>
> > Date:   Sat Dec 5 08:48:03 2015 +0200
> > 
> >     Upgrade paxutils
> > 
> > which produces issues with CoreFoundation's threading out of the libintl 
> > linkage on tar.
> 
> To further clarify...
> 
> git checkout 6ea9e62bb398b2f81e52f671749e585367286820
> ./configure --without-xattrs  CPPFLAGS="-I/sw/include -I/usr/lib" 
> LDFLAGS="-L/sw/lib -L/usr/lib"
> make 
> 
> produces a tar which doesn't suffer from the bug so commit 
> 589ba77faf60e4632771d80e2b25d9e3778d4a3f
> is the sole cause of this bug.

Also confirmed that reverting 
http://git.savannah.gnu.org/cgit/tar.git/patch/?id=589ba77faf60e4632771d80e2b25d9e3778d4a3f
from the stock tar 1.29 release sources produces a tar executable, when built 
against libintl linked to CoreFoundation,
that no longer fails on 'tar -zcvf' when LANG is unset.

> > 
> > > 
> > > > > 
> > > > > > 
> > > > > > -- 
> > > > > > Cheers, Ralph.
> > > > > > https://plus.google.com/+RalphCorderoy