bug#61386: [PATCH] cp,mv,install: Disable sparse copy on macOS

[George Valkov]

> On 2023-02-24, at 12:23 AM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> 
> On 2/20/23 13:14, Pádraig Brady wrote:
>> Since https://github.com/coreutils/gnulib/commit/4db8db34
>> gnulib has been unconditionally replacing lseek() on macos.
>> Now with SEEK_DATA undefined that replaced lseek()
>> will run the code intended for BeOS.
>> So either the lseek.m4 or lseek.c needs adjusting accordingly.
> 
> Good catch, thanks. I updated the Gnulib patch accordingly; see attached.

Nice: I just downloaded a fresh copy of Savannah, and it already includes the 
attached patch, so no action needed on my side.

The copy created by cp is good. I noticed that you have added a —debug switch, 
so I gave it a test:
1. If the target exists I get this output. I would assume zeroes means that all 
data is read from the source, but pages containing only zeroes are skipped, 
while pages containing data are written to the target, which is fine. I would 
guess 4 KB page granularity or some form of detection takes place.

./coreutils-2023-02-24/src/cp --debug cc1-mmap cc1-ori
'cc1-mmap' -> 'cc1-ori'
copy offload: avoided, reflink: unsupported, sparse detection: zeros

2. If the target does not exist, the file is cloned. You might want to report 
something about that in the debug output. Otherwise the clone is good.

./coreutils-2023-02-24/src/cp --debug cc1-mmap cc1-ori
'cc1-mmap' -> 'cc1-ori’


My first attempt to run the tests stopped here, so I had to interrupt it 10 
minutes later with Control+C.
make check-TESTS
make[1]: Entering directory '/Volumes/coreutils/coreutils-2023-02-24'
PASS: tests/misc/help-version.sh
PASS: tests/misc/help-version-getopt.sh

Then I ran it again, and I can see the tests run very slowly now. This time it 
hanged after:

PASS: tests/rm/isatty.sh
^Cmake[1]: *** Deleting file 'tests/rm/empty-inacc.log'
make[1]: *** [Makefile:21399: tests/rm/empty-inacc.log] Error 130
make: *** [Makefile:21381: check-TESTS] Interrupt: 2

make check-TESTS

make[1]: Entering directory '/Volumes/coreutils/coreutils-2023-02-24'
PASS: tests/misc/help-version.sh
PASS: tests/misc/help-version-getopt.sh
^Cmake[1]: *** Deleting file 'tests/tail-2/inotify-race.log'
^C^C^C^C

Now it hangs here and I cannot interrupt it. I don’t see any CPU usage related 
to the tests. My laptop is idle. Attempting to close the Terminal window gave 
me this warning:

Do you want to terminate running processes in this tab?
Closing this tab will terminate the running processes: gdb, bash, make (2), sh 
(4).

That’s a good culprit. Two days ago brew updated gdb to version 13.1 and there 
was a message asking me to sign it with some entitlements to make it more 
functional. So, I signed gdb. At first though make check-TESTS does not play 
nice when gdb is signed, so I removed the signature, but that did not help. 
Next I restored version 12.1_2 and ran make check-TESTS again, which still 
hangs at the same points. So gdb signature and version have nothing to do with 
this new issue.

Next I ran the tests on top of my previous unpatched build directory coreutils 
cf80f988eeb97cc3f8c65ae58e735d36f865277b, gnulib 
32c16c45d7378b014d9aac6130104c4d02a9acdb and it works fine, so I would assume 
something related to the tests has been broken recently in coreutils or gnulib. 
I restored gdb 13.1 signed and the tests completed again.

Back to latest unmodified coreutils 5c8c2a5161c0b6f84212778f694c526105f10dab, 
gnulib 7352d9fec59398c76c3bb8a2ef86ba58818f0342, the tests hang.

make check-TESTS
make[1]: Entering directory '/Volumes/coreutils/coreutils-2023-02-24'
PASS: tests/misc/help-version.sh
PASS: tests/misc/help-version-getopt.sh
^Cmake[1]: *** Deleting file 'tests/tail-2/inotify-race.log'
^C^C^C^C^C^C

killall gdb
ps -A |grep gdb
29584 ??         0:00.09 gdb -nx --batch-silent --eval-command=break 1475 
--eval-command=run --pid=29583 -f file --eval-command=quit tail
23269 ttys010    0:00.09 gdb -nx --batch-silent --eval-command=break 1475 
--eval-command=run --pid=23268 -f file --eval-command=quit tail

killall -9 gdb
ps -A |grep gdb

Killing gdb allowed the tests to continue, I had to do it twice:
https://httpstorm.com/share/.openwrt/test/2023-02-06_coreutils-9.1/test-06-5c8c2a5161c0b6f84212778f694c526105f10dab-ori.txt


> On 2/20/23 02:21, George Valkov wrote:
> > What is the correct way to apply your patch?
> 
> Sounds like patching is a bit of a hassle, so to make things easier I 
> installed the attached patch into Gnulib, and propagated this into Coreutils.

I’m starting to think I made a mistake by trying to apply your patch inside 
coreutils, since it already had links to the affected files. I guess I should 
have applied the patch inside coreutils/gnulib?

So my question was: what command do you use to apply your patch and in which 
directory do you run it?


> You should be able to get the latest version of Coreutils from 
> savannah.gnu.org, and then run './bootstrap; ./configure; make' if you have 
> suitable development tools like Autoconf. You can run ./bootstrap on a 
> GNU/Linux platform and then copy the directory to macOS and run './configure; 
> make' on macOS. Please give it a try.

All the development tools should already be installed on the Mac using brew. I 
can confirm autoconf is installed:

brew list | grep autoconf
autoconf

I always start with these commands:

git clone https://github.com/coreutils/coreutils.git
cd coreutils
git submodule foreach git pull origin master
./bootstrap
./configure
make -j 16
git checkout -b 2023-02-24

Next I apply the changes you want, make clean ; make -j 16
Finally I proceed with testing.


> > If you know what conditions are required to reproduce the issue on FreeBSD,
> 
> I don't. I'm relying on FreeBSD bug report 256205. I cited that in the 
> attached patch.

I see they reproduce the bug on both arm64 and x64. Based on the conversation 
their issue is very similar: they use a linker to craft a file, that linker is 
likely to use mmap. But their issue is fixed with fsync. And they also talk 
about nulls. macOS is indeed based on BSD.

Also if I read correctly, they do not experience the issue when working with 
the root file-system on FreeBSD, its the same on macOS: I need to mount an APFS 
sparse disk image, they use nullfs.

Comments say they seem to have fixed it partiality in 
42881526d401e7a9c09241e392b7ffa18cfe11d6, and then completely later. I am too 
busy to play with FreeBSD at the moment, as it takes more advanced preparation.


> On 2023-02-24, at 2:51 AM, Pádraig Brady <P@draigBrady.com> wrote:
> 
>> On 2/20/23 02:21, George Valkov wrote:
>>  > What is the correct way to apply your patch?
>> Sounds like patching is a bit of a hassle, so to make things easier I
>> installed the attached patch into Gnulib, and propagated this into
>> Coreutils.
> 
> The latest coreutils that should include all fixes for this issue is at
> https://www.pixelbeat.org/cu/coreutils-9.1.160-5c8c2.tar.gz
> That should be buildable with the standard ./configure && make combo

Pádraig, I compiled from your archive, cp produces a good copy, however make 
check-TESTS hangs the same way as with the clone I compiled on the latest 
Savannah master.


Good luck!


Georgi Valkov
httpstorm.com
nano RTOS