Re: dd: copy blocks in reverse order

[Pádraig Brady]

On 22/12/15 03:03, Ivan Pozdeev wrote:
>> On 06/12/15 18:32, Ivan Pozdeev wrote:
>>>> On 06/12/15 15:20, Ivan Pozdeev wrote:
>>>>> Hello Coreutils,
>>>>>
>>>>> The subj is a requirement if I'm copying between overlapping regions, and 
>>>>> the
>>>>> destination is further in the media than the source.
>>>>>
>>>>> The specific task I'm having is to move a partition a number of sectors 
>>>>> forward.
>>>>>
>>>>> To be completely clear, the algorithm is: copy a block, then seek to the
>>>>> previous block (i.e. 2*bs bytes back).
>>>>>
>>>>> I don't currently see a use case for setting this for input/output 
>>>>> independently.
>>>>>
>>>>> An idea for the option name is `d'.
>>>>>
>>>>> There is a workaround for my particular case - `dd if=<block_device> 
>>>>> bs=<2*shift_bytes> | dd
>>>>> of=<block_device> seek=<shift_blocks>' but it requires a few times 
>>>>> 2*shift_bytes of memory.
>>>
>>>> I see dd_rescue (and ddrescue) have "operate in reverse" options.
>>>> The reason stated being:
>>>
>>>> "If you have one spot of bad sectors within the partition,
>>>> it might be a good idea, to approach this spot from both sides."
>>>
>>> I already looked into dd_rescue. It can only copy a full device,
>>> its purpose being data rescue rather than tinkering.
>>>
>>>> For your use case you could compute the size and offset of the overlap,
>>>> and use 2 dd invocations to copy the overlap area, then the rest.
>>>
>>> Not applicable. The overlap is >1/2, so its destination is also within the
>>> overlapped area.
>>>
>>>> Or perhaps more simply use a loop to iterate in reverse
>>>> using an appropriately large blocksize to minimize dd invocations.
>>>
>>> Looked into this, too. 78000000+ invocations in my case.
>>> I don't even want to think of how long this would take.
>>>
>>>> So while this is useful it is a bit of an edge case
>>>> and there are alternatives using other tools or multiple dd invocations.
>>>
>>> dd's sole job is to "take blocks from here, put them into there".
>>> Unless there's another tool whose sole job is to shuffle the input,
>>> the order of taking/putting seems to be an intrinsic trait of that job.
>>> Though I agree that it's the minority of cases where this trait matters.
>>>
>>> There's yet another argument:
>>> As you outlined, currently one has to resort to different methods for 
>>> different
>>> data sizes/overlap percentages, which are far from being obvious (e.g. I 
>>> came
>>> with my stated workaround only after having composed the letter). Basically,
>>> what I'm suggesting is the "one obvious way" ((c)Python Zen) - a method 
>>> whose
>>> chief advantage is being immediately apparent (as well as being highly
>>> applicable and readable in code).
>>>
>>>> I'd be 60:40 against adding it.
> 
>> Good arguments thanks.  Worth considering.
> 
>> Another option for this case where the read and write offsets are close,
>> would be to read into a circular buffer, and write while maintaining
>> a minimum number of blocks (the overlap) in the buffer.
> 
>> We previously talked about adding such a util to coreutils:
>> http://lists.gnu.org/archive/html/bug-coreutils/2005-11/msg00190.html
> 
>> A simple untested patch to debian's buffer(1) attached might achieve this
>> in a more optimum fashion by supporting read and write sizes > overlap size.
>> That would be used something like:
> 
>> dd bs=... if=... | buffer -s 1m -P 50 | dd seek=... of=...
> 
>> cheers,
>> Pádraig.
> 
> My aforementioned workaround didn't work. The 2nd `dd' still overwrites
> some sectors before the 1st one reads them.
> 
> The line with `buffer' just above (the System Rescue CD that I'm using has
> `mbuffer' instead) didn't work, either: -P delays writes initially but doesn't
> protect from buffer underruns.
> 
> You didn't specify which specific source your patch is against so I couldn't 
> test it.
> Yet, due to the previous paragraph, I suspect it won't help, either.

mkdir buffer
cd buffer
wget http://http.debian.net/debian/pool/main/b/buffer/buffer_1.19.orig.tar.gz
wget 
http://http.debian.net/debian/pool/main/b/buffer/buffer_1.19-12.debian.tar.xz
tar -xf buffer_1.19.orig.tar.gz
tar -xf buffer_1.19-12.debian.tar.xz
(cd buffer-1.19 && for p in $(cat ../debian/patches/series); do patch -p1 < 
../debian/patches/$p; done)
cd buffer-1.19/
patch -p1 < ../buffer-min_percent.diff
make
./buffer ...

Note I've attached the diff again to this email
as the previous one erroneously included ansi coloring sequences.

> Even `parted' and `gparted' are no good! The first one doesn't support 
> moving, the
> 2nd one has 1M granilarity and doesn't show/allow to set actual sector 
> numbers.

thanks for testing all the options,
Pádraig.

buffer-min_percent.diff
Description: Text Data