[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
bug#6131: [PATCH]: fiemap support for efficient sparse file copy
|
From: |
jeff.liu |
|
Subject: |
bug#6131: [PATCH]: fiemap support for efficient sparse file copy |
|
Date: |
Thu, 27 May 2010 22:08:43 +0800 |
|
User-agent: |
Thunderbird 2.0.0.14 (X11/20080505) |
Jim Meyering wrote:
> jeff.liu wrote:
>> This is the revised version, it fixed the fiemap-start offset calculation
>> approach to remove it out
>> of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop.
>
> Hi Jeff,
>
> I've included below the state of my local changes.
> Unfortunately, with that 5-patch series, there is always a test failure
> on F13/ext4. Maybe someone who knows more about extents can provide an
> explanation?
>
> Here's a small example to demonstrate:
>
> Create a file with many extents:
>
> perl -e 'BEGIN { $n = 19 * 1024; *F = *STDOUT }' \
> -e 'for (1..100) { sysseek (*F, $n, 1)' \
> -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1
>
> Using the patched "cp", repeat the following 10 or 20 times:
>
> ./cp --sparse=always j1 j2; sync
> filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1
> filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1
> diff -u ff1 ff2 || fail=1
>
> Usually there is no diff output, but occasionally it'll print this:
> [hmm... today it consistently prints these differences every other time.]
Woo!!!
I just run this test on btrfs/ext4/ocfs2 against mainline kernel(Linux
jeff-laptop
2.6.33-rc5-00238-gb04da8b-dirty) on my laptop.
Only btrfs always works well for me, Ext4 has the same issue like yours.
address@hidden:/ext4/test$ for ((i=0; i < 100; i++)); do ./1.sh; done
--- ff1 2010-05-27 22:03:25.263480260 +0800
+++ ff2 2010-05-27 22:03:25.315476210 +0800
@@ -26,77 +26,78 @@
24 232
25 242
26 251
-27 261
-28 270
-29 280
-30 289
-31 299
-32 308
-33 318
-34 327
-35 337
-36 346
-37 356
-38 365
-39 375
-40 384
-41 394
-42 403
-43 413
-44 422
-45 432
-46 441
-47 451
-48 460
-49 470
-50 479
-51 489
-52 498
-53 508
-54 512
-55 517
-56 527
-57 536
-58 546
-59 555
-60 565
-61 574
-62 584
-63 593
-64 603
-65 612
-66 622
-67 631
-68 641
-69 650
-70 660
-71 669
-72 679
-73 688
-74 698
-75 707
-76 717
-77 726
-78 736
-79 745
-80 755
-81 764
-82 774
-83 783
-84 793
-85 802
-86 812
-87 821
-88 831
-89 840
-90 850
-91 859
-92 869
-93 878
-94 888
-95 897
-96 907
-97 916
-98 926
-99 935
-100 945
+27 256
+28 261
+29 270
+30 280
+31 289
+32 299
+33 308
+34 318
+35 327
+36 337
+37 346
+38 356
+39 365
+40 375
+41 384
+42 394
+43 403
+44 413
+45 422
+46 432
+47 441
+48 451
+49 460
+50 470
+51 479
+52 489
+53 498
+54 508
+55 512
+56 517
+57 527
+58 536
+59 546
+60 555
+61 565
+62 574
+63 584
+64 593
+65 603
+66 612
+67 622
+68 631
+69 641
+70 650
+71 660
+72 669
+73 679
+74 688
+75 698
+76 707
+77 717
+78 726
+79 736
+80 745
+81 755
+82 764
+83 774
+84 783
+85 793
+86 802
+87 812
+88 821
+89 831
+90 840
+91 850
+92 859
+93 869
+94 878
+95 888
+96 897
+97 907
+98 916
+99 926
+100 935
+101 945
OCFS2's show different for many times(repeat 100 times).
But I cannot figure out this issue from the patch at the moment. :(
HEROS, who can give some hints from the kernel's point of view?
address@hidden:/ocfs2/test$ for ((i=0; i < 100; i++)); do ./1.sh; done
--- ff1 2010-05-27 21:49:58.288483759 +0800
+++ ff2 2010-05-27 21:49:58.304545778 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 519
--- ff1 2010-05-27 21:50:05.260477055 +0800
+++ ff2 2010-05-27 21:50:05.276532299 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 815
--- ff1 2010-05-27 21:50:13.048484948 +0800
+++ ff2 2010-05-27 21:50:13.076486205 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 161
--- ff1 2010-05-27 21:50:19.748471818 +0800
+++ ff2 2010-05-27 21:50:19.764488230 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 457
--- ff1 2010-05-27 21:50:25.384540820 +0800
+++ ff2 2010-05-27 21:50:25.396508694 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 753
--- ff1 2010-05-27 21:50:33.504500173 +0800
+++ ff2 2010-05-27 21:50:33.536480827 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 99
--- ff1 2010-05-27 21:50:40.012512743 +0800
+++ ff2 2010-05-27 21:50:40.028511067 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 395
--- ff1 2010-05-27 21:50:46.680505132 +0800
+++ ff2 2010-05-27 21:50:46.708501290 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 691
--- ff1 2010-05-27 21:50:55.576480477 +0800
+++ ff2 2010-05-27 21:50:55.596475937 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 37
--- ff1 2010-05-27 21:51:03.491482451 +0800
+++ ff2 2010-05-27 21:51:03.515503404 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 333
--- ff1 2010-05-27 21:51:11.567497188 +0800
+++ ff2 2010-05-27 21:51:11.603479937 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 629
--- ff1 2010-05-27 21:51:20.983471486 +0800
+++ ff2 2010-05-27 21:51:21.011472813 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 925
--- ff1 2010-05-27 21:51:29.548508415 +0800
+++ ff2 2010-05-27 21:51:29.588486206 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 271
--- ff1 2010-05-27 21:51:35.927492089 +0800
+++ ff2 2010-05-27 21:51:35.959488178 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 567
--- ff1 2010-05-27 21:51:40.200480478 +0800
+++ ff2 2010-05-27 21:51:40.240471818 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 863
--- ff1 2010-05-27 21:51:45.772499475 +0800
+++ ff2 2010-05-27 21:51:45.820498986 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 209
--- ff1 2010-05-27 21:51:49.919519607 +0800
+++ ff2 2010-05-27 21:51:49.947475956 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 505
--- ff1 2010-05-27 21:51:55.195482312 +0800
+++ ff2 2010-05-27 21:51:55.231483778 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 801
--- ff1 2010-05-27 21:52:00.911479588 +0800
+++ ff2 2010-05-27 21:52:00.939473232 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 147
--- ff1 2010-05-27 21:52:06.208488091 +0800
+++ ff2 2010-05-27 21:52:06.252482433 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 443
....
...
>
> $ diff -u ff1 ff2 || fail=1
> --- ff1 2010-05-22 18:42:26.943501382 +0200
> +++ ff2 2010-05-22 18:42:27.020876155 +0200
> @@ -53,49 +53,50 @@ ext logical
> 51 489
> 52 498
> 53 508
> -54 517
> -55 527
> -56 536
> -57 546
> -58 555
> -59 565
> -60 574
> -61 584
> -62 593
> -63 603
> -64 612
> -65 622
> -66 631
> -67 641
> -68 650
> -69 660
> -70 669
> -71 679
> -72 688
> -73 698
> -74 707
> -75 717
> -76 726
> -77 736
> -78 745
> -79 755
> -80 764
> -81 774
> -82 783
> -83 793
> -84 802
> -85 812
> -86 821
> -87 831
> -88 840
> -89 850
> -90 859
> -91 869
> -92 878
> -93 888
> -94 897
> -95 907
> -96 916
> -97 926
> -98 935
> -99 945
> +54 512
> +55 517
> +56 527
> +57 536
> +58 546
> +59 555
> +60 565
> +61 574
> +62 584
> +63 593
> +64 603
> +65 612
> +66 622
> +67 631
> +68 641
> +69 650
> +70 660
> +71 669
> +72 679
> +73 688
> +74 698
> +75 707
> +76 717
> +77 726
> +78 736
> +79 745
> +80 755
> +81 764
> +82 774
> +83 783
> +84 793
> +85 802
> +86 812
> +87 821
> +88 831
> +89 840
> +90 850
> +91 859
> +92 869
> +93 878
> +94 888
> +95 897
> +96 907
> +97 916
> +98 926
> +99 935
> +100 945
>
>
> From 2ef44bcb0bbb38cca738aa90e067caee312da939 Mon Sep 17 00:00:00 2001
> From: Jie Liu <address@hidden>
> Date: Thu, 13 May 2010 22:09:30 +0800
> Subject: [PATCH 1/5] cp: Add FIEMAP support for efficient sparse file copy
>
> * src/fiemap.h: Add fiemap.h for fiemap ioctl(2) support.
> Copied from linux's include/linux/fiemap.h, with minor formatting changes.
> * src/copy.c (copy_reg): Now, when `cp' invoked with --sparse=[WHEN] option,
> we
> will try to do FIEMAP-copy if the underlaying file system support it, fall
> back
> to a normal copy if it fails.
> ---
> src/copy.c | 159
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> src/fiemap.h | 102 +++++++++++++++++++++++++++++++++++++
> 2 files changed, 261 insertions(+), 0 deletions(-)
> create mode 100644 src/fiemap.h
>
> diff --git a/src/copy.c b/src/copy.c
> index c16cef6..0e54729 100644
> --- a/src/copy.c
> +++ b/src/copy.c
> @@ -63,6 +63,10 @@
>
> #include <sys/ioctl.h>
>
> +#ifndef HAVE_FIEMAP
> +# include "fiemap.h"
> +#endif
> +
> #ifndef HAVE_FCHOWN
> # define HAVE_FCHOWN false
> # define fchown(fd, uid, gid) (-1)
> @@ -149,6 +153,141 @@ clone_file (int dest_fd, int src_fd)
> #endif
> }
>
> +#ifdef __linux__
> +# ifndef FS_IOC_FIEMAP
> +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap)
> +# endif
> +/* Perform FIEMAP(available in mainline 2.6.27) copy if possible.
> + Call ioctl(2) with FS_IOC_FIEMAP to efficiently map file allocation
> + excepts holes. So the overhead to deal with holes with lseek(2) in
> + normal copy could be saved. This would result in much faster backups
> + for any kind of sparse file. */
> +static bool
> +fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size,
> + off_t src_total_size, char const *src_name,
> + char const *dst_name, bool *normal_copy_required)
> +{
> + bool fail = false;
> + bool last = false;
> + char fiemap_buf[4096];
> + struct fiemap *fiemap = (struct fiemap *)fiemap_buf;
> + struct fiemap_extent *fm_ext = &fiemap->fm_extents[0];
> + uint32_t count = (sizeof (fiemap_buf) - sizeof (*fiemap)) /
> + sizeof (struct fiemap_extent);
> + off_t last_ext_logical = 0;
> + uint64_t last_ext_len = 0;
> + uint64_t last_read_size = 0;
> + unsigned int i = 0;
> +
> + /* This is required at least to initialize fiemap->fm_start,
> + but also serves (in May 2010) to appease valgrind, which
> + appears not to know the semantics of the FIEMAP ioctl. */
> + memset (fiemap_buf, 0, sizeof fiemap_buf);
> +
> + do
> + {
> + fiemap->fm_length = FIEMAP_MAX_OFFSET;
> + fiemap->fm_extent_count = count;
> +
> + /* When ioctl(2) fails, fall back to the normal copy only if it
> + is the first time we met. */
> + if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0)
> + {
> + /* If `i > 0', then at least one ioctl(2) has been performed
> before. */
> + if (i == 0)
> + *normal_copy_required = true;
> + return false;
> + }
> +
> + /* If 0 extents are returned, then more ioctls are not needed. */
> + if (fiemap->fm_mapped_extents == 0)
> + break;
> +
> + for (i = 0; i < fiemap->fm_mapped_extents; i++)
> + {
> + assert (fm_ext[i].fe_logical <= OFF_T_MAX);
> +
> + off_t ext_logical = fm_ext[i].fe_logical;
> + uint64_t ext_len = fm_ext[i].fe_length;
> +
> + if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL)
> + {
> + error (0, errno, _("cannot lseek %s"), quote (src_name));
> + return fail;
> + }
> +
> + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL)
> + {
> + error (0, errno, _("cannot lseek %s"), quote (dst_name));
> + return fail;
> + }
> +
> + if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST)
> + {
> + last_ext_logical = ext_logical;
> + last_ext_len = ext_len;
> + last = true;
> + }
> +
> + while (0 < ext_len)
> + {
> + char buf[buf_size];
> +
> + /* Avoid reading into the holes if the left extent
> + length is shorter than the buffer size. */
> + if (ext_len < buf_size)
> + buf_size = ext_len;
> +
> + ssize_t n_read = read (src_fd, buf, buf_size);
> + if (n_read < 0)
> + {
> +#ifdef EINTR
> + if (errno == EINTR)
> + continue;
> +#endif
> + error (0, errno, _("reading %s"), quote (src_name));
> + return fail;
> + }
> +
> + if (n_read == 0)
> + {
> + /* Figure out how many bytes read from the last extent. */
> + last_read_size = last_ext_len - ext_len;
> + break;
> + }
> +
> + if (full_write (dest_fd, buf, n_read) != n_read)
> + {
> + error (0, errno, _("writing %s"), quote (dst_name));
> + return fail;
> + }
> +
> + ext_len -= n_read;
> + }
> + }
> +
> + fiemap->fm_start = fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_length;
> +
> + } while (! last);
> +
> + /* If a file ends up with holes, the sum of the last extent logical offset
> + and the read-returned size will be shorter than the actual size of the
> + file. Use ftruncate to extend the length of the destination file. */
> + if (last_ext_logical + last_read_size < src_total_size)
> + {
> + if (ftruncate (dest_fd, src_total_size) < 0)
> + {
> + error (0, errno, _("extending %s"), quote (dst_name));
> + return fail;
> + }
> + }
> +
> + return ! fail;
> +}
> +#else
> +static bool fiemap_copy_ok (ignored) { errno == ENOTSUP; return false; }
> +#endif
> +
> /* FIXME: describe */
> /* FIXME: rewrite this to use a hash table so we avoid the quadratic
> performance hit that's probably noticeable only on trees deeper
> @@ -679,6 +818,25 @@ copy_reg (char const *src_name, char const *dst_name,
> #endif
> }
>
> + if (make_holes)
> + {
> + bool require_normal_copy = false;
> + /* Perform efficient FIEMAP copy for sparse files, fall back to the
> + standard copy only if the ioctl(2) fails. */
> + if (fiemap_copy_ok (source_desc, dest_desc, buf_size,
> + src_open_sb.st_size, src_name,
> + dst_name, &require_normal_copy))
> + goto preserve_metadata;
> + else
> + {
> + if (! require_normal_copy)
> + {
> + return_val = false;
> + goto close_src_and_dst_desc;
> + }
> + }
> + }
> +
> /* If not making a sparse file, try to use a more-efficient
> buffer size. */
> if (! make_holes)
> @@ -807,6 +965,7 @@ copy_reg (char const *src_name, char const *dst_name,
> }
> }
>
> +preserve_metadata:
> if (x->preserve_timestamps)
> {
> struct timespec timespec[2];
> diff --git a/src/fiemap.h b/src/fiemap.h
> new file mode 100644
> index 0000000..d33293b
> --- /dev/null
> +++ b/src/fiemap.h
> @@ -0,0 +1,102 @@
> +/* FS_IOC_FIEMAP ioctl infrastructure.
> + Some portions copyright (C) 2007 Cluster File Systems, Inc
> + Authors: Mark Fasheh <address@hidden>
> + Kalpak Shah <address@hidden>
> + Andreas Dilger <address@hidden>. */
> +
> +/* Copy from kernel, modified to respect GNU code style by Jie Liu. */
> +
> +#ifndef _LINUX_FIEMAP_H
> +# define _LINUX_FIEMAP_H
> +
> +# include <linux/types.h>
> +
> +struct fiemap_extent
> +{
> + /* Logical offset in bytes for the start of the extent
> + from the beginning of the file. */
> + uint64_t fe_logical;
> +
> + /* Physical offset in bytes for the start of the extent
> + from the beginning of the disk. */
> + uint64_t fe_physical;
> +
> + /* Length in bytes for this extent. */
> + uint64_t fe_length;
> +
> + uint64_t fe_reserved64[2];
> +
> + /* FIEMAP_EXTENT_* flags for this extent. */
> + uint32_t fe_flags;
> +
> + uint32_t fe_reserved[3];
> +};
> +
> +struct fiemap
> +{
> + /* Logical offset(inclusive) at which to start mapping(in). */
> + uint64_t fm_start;
> +
> + /* Logical length of mapping which userspace wants(in). */
> + uint64_t fm_length;
> +
> + /* FIEMAP_FLAG_* flags for request(in/out). */
> + uint32_t fm_flags;
> +
> + /* Number of extents that were mapped(out). */
> + uint32_t fm_mapped_extents;
> +
> + /* Size of fm_extents array(in). */
> + uint32_t fm_extent_count;
> +
> + uint32_t fm_reserved;
> +
> + /* Array of mapped extents(out). */
> + struct fiemap_extent fm_extents[0];
> +};
> +
> +/* The maximum offset can be mapped for a file. */
> +# define FIEMAP_MAX_OFFSET (~0ULL)
> +
> +/* Sync file data before map. */
> +# define FIEMAP_FLAG_SYNC 0x00000001
> +
> +/* Map extented attribute tree. */
> +# define FIEMAP_FLAG_XATTR 0x00000002
> +
> +# define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR)
> +
> +/* Last extent in file. */
> +# define FIEMAP_EXTENT_LAST 0x00000001
> +
> +/* Data location unknown. */
> +# define FIEMAP_EXTENT_UNKNOWN 0x00000002
> +
> +/* Location still pending, Sets EXTENT_UNKNOWN. */
> +# define FIEMAP_EXTENT_DELALLOC 0x00000004
> +
> +/* Data can not be read while fs is unmounted. */
> +# define FIEMAP_EXTENT_ENCODED 0x00000008
> +
> +/* Data is encrypted by fs. Sets EXTENT_NO_BYPASS. */
> +# define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080
> +
> +/* Extent offsets may not be block aligned. */
> +# define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100
> +
> +/* Data mixed with metadata. Sets EXTENT_NOT_ALIGNED. */
> +# define FIEMAP_EXTENT_DATA_INLINE 0x00000200
> +
> +/* Multiple files in block. Set EXTENT_NOT_ALIGNED. */
> +# define FIEMAP_EXTENT_DATA_TAIL 0x00000400
> +
> +/* Space allocated, but not data (i.e. zero). */
> +# define FIEMAP_EXTENT_UNWRITTEN 0x00000800
> +
> +/* File does not natively support extents. Result merged for efficiency. */
> +# define FIEMAP_EXTENT_MERGED 0x00001000
> +
> +/* Space shared with other files. */
> +# define FIEMAP_EXTENT_SHARED 0x00002000
> +
> +#endif
> --
> 1.7.1.348.gb26ba
>
>
> From 260b5b89e33da2b9a5ea5bcd9dba874f503d2937 Mon Sep 17 00:00:00 2001
> From: Jie Liu <address@hidden>
> Date: Thu, 13 May 2010 22:17:53 +0800
> Subject: [PATCH 2/5] tests: add a new test for FIEMAP-copy
>
> * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a
> loopbacked ext4 partition.
> * tests/Makefile.am (sparse-fiemap): Reference the new test.
> ---
> tests/Makefile.am | 1 +
> tests/cp/sparse-fiemap | 56
> ++++++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 57 insertions(+), 0 deletions(-)
> create mode 100755 tests/cp/sparse-fiemap
>
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index c458574..f7840c8 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -25,6 +25,7 @@ root_tests = \
> cp/special-bits \
> cp/cp-mv-enotsup-xattr \
> cp/capability \
> + cp/sparse-fiemap \
> dd/skip-seek-past-dev \
> install/install-C-root \
> ls/capability \
> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap
> new file mode 100755
> index 0000000..945c94b
> --- /dev/null
> +++ b/tests/cp/sparse-fiemap
> @@ -0,0 +1,56 @@
> +#!/bin/sh
> +# Test cp --sparse=always through fiemap copy
> +
> +# Copyright (C) 2006-2010 Free Software Foundation, Inc.
> +
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +if test "$VERBOSE" = yes; then
> + set -x
> + cp --version
> +fi
> +
> +. $srcdir/test-lib.sh
> +require_root_
> +
> +cwd=`pwd`
> +cleanup_() { cd /; umount "$cwd/mnt"; }
> +
> +skip=0
> +# Create an ext4 loopback file system
> +dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1
> +mkdir mnt
> +mkfs -t ext4 -F blob ||
> + skip_test_ "failed to create ext4 file system"
> +mount -oloop blob mnt || skip=1
> +echo test > mnt/f || skip=1
> +test -s mnt/f || skip=1
> +
> +test $skip = 1 &&
> + skip_test_ "insufficient mount/ext4 support"
> +
> +# Create a 1TiB sparse file
> +dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure
> +
> +cd mnt || fail=1
> +
> +# It takes many minutes to copy this sparse file using the old method.
> +# By contrast, it takes far less than 1 second using FIEMAP-copy.
> +timeout 10 cp --sparse=always sparse fiemap || fail=1
> +
> +# Ensure that the sparse file copied through fiemap has the same size
> +# in bytes as the original.
> +test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1
> +
> +Exit $fail
> --
> 1.7.1.348.gb26ba
>
>
> From 738031aed78b6323969f799d3d5fbf19e0cfc91a Mon Sep 17 00:00:00 2001
> From: Jim Meyering <address@hidden>
> Date: Fri, 21 May 2010 18:28:42 +0200
> Subject: [PATCH 3/5] tests: exercise more of the new FIEMAP copying code
>
> * tests/cp/sparse-fiemap: Ensure that a file with many extents (more
> than fit in copy.c's internal 4KiB buffer) is copied properly.
> ---
> tests/cp/sparse-fiemap | 38 ++++++++++++++++++++++++++++++++++++++
> 1 files changed, 38 insertions(+), 0 deletions(-)
>
> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap
> index 945c94b..b1643be 100755
> --- a/tests/cp/sparse-fiemap
> +++ b/tests/cp/sparse-fiemap
> @@ -53,4 +53,42 @@ timeout 10 cp --sparse=always sparse fiemap || fail=1
> # in bytes as the original.
> test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1
>
> +# =================================================
> +# Ensure that we exercise the FIEMAP-copying code enough
> +# to provoke at least two iterations of the do...while loop
> +# in which it calls ioctl (fd, FS_IOC_FIEMAP,...
> +# This also verifies that non-trivial extents are preserved.
> +
> +$PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl'
> +
> +$PERL -e 'BEGIN { $n = 16 * 1024; *F = *STDOUT }' \
> + -e 'for (1..100) { sysseek (*F, $n, 1)' \
> + -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1
> +
> +cp --sparse=always j1 j2 || fail=1
> +cmp j1 j2 || fail=1
> +
> +filefrag j1 | grep extent \
> + || skip_test_ 'skipping part of this test; you lack filefrag'
> +
> +# Here is sample filefrag output:
> +# $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \
> +# -e 'for (1..5) { sysseek(*F,$n,1)' \
> +# -e '&& syswrite *F,"."x$n or die "$!"}' > j
> +# $ filefrag -v j
> +# Filesystem type is: ef53
> +# File size of j is 163840 (40 blocks, blocksize 4096)
> +# ext logical physical expected length flags
> +# 0 4 6258884 4
> +# 1 12 6258892 6258887 4
> +# 2 20 6258900 6258895 4
> +# 3 28 6258908 6258903 4
> +# 4 36 6258916 6258911 4 eof
> +# j: 6 extents found
> +
> +# exclude the physical block numbers; they always differ
> +filefrag -v j1 | awk '/^ / {print $1,$2,$NF}' > ff1 || fail=1
> +filefrag -v j2 | awk '/^ / {print $1,$2,$NF}' > ff2 || fail=1
> +compare ff1 ff2 || fail=1
> +
> Exit $fail
> --
> 1.7.1.348.gb26ba
>
>
> From e84f881cb5c2eb92f3b6d5bddaef50c3e811bc30 Mon Sep 17 00:00:00 2001
> From: Jim Meyering <address@hidden>
> Date: Sat, 22 May 2010 10:22:58 +0200
> Subject: [PATCH 4/5] tests: require root access only if current partition is
> not ext4
>
> * tests/cp/sparse-fiemap: Don't require root access if current
> partition is ext4.
> ---
> tests/cp/sparse-fiemap | 44 +++++++++++++++++++++++++-------------------
> 1 files changed, 25 insertions(+), 19 deletions(-)
>
> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap
> index b1643be..371bced 100755
> --- a/tests/cp/sparse-fiemap
> +++ b/tests/cp/sparse-fiemap
> @@ -22,28 +22,34 @@ if test "$VERBOSE" = yes; then
> fi
>
> . $srcdir/test-lib.sh
> -require_root_
>
> -cwd=`pwd`
> -cleanup_() { cd /; umount "$cwd/mnt"; }
> -
> -skip=0
> -# Create an ext4 loopback file system
> -dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1
> -mkdir mnt
> -mkfs -t ext4 -F blob ||
> - skip_test_ "failed to create ext4 file system"
> -mount -oloop blob mnt || skip=1
> -echo test > mnt/f || skip=1
> -test -s mnt/f || skip=1
> -
> -test $skip = 1 &&
> - skip_test_ "insufficient mount/ext4 support"
> +if df -T -t ext4 . ; then
> + : # Current dir is on an ext4 partition. Good!
> +else
> + # It's not; we need to create one, hence we need root access.
> + require_root_
> +
> + cwd=$PWD
> + cleanup_() { cd /; umount "$cwd/mnt"; }
> +
> + skip=0
> + # Create an ext4 loopback file system
> + dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1
> + mkdir mnt
> + mkfs -t ext4 -F blob ||
> + skip_test_ "failed to create ext4 file system"
> + mount -oloop blob mnt || skip=1
> + echo test > mnt/f || skip=1
> + test -s mnt/f || skip=1
> +
> + test $skip = 1 &&
> + skip_test_ "insufficient mount/ext4 support"
> +
> + cd mnt || fail=1
> +fi
>
> # Create a 1TiB sparse file
> -dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure
> -
> -cd mnt || fail=1
> +dd if=/dev/zero of=sparse bs=1k count=1 seek=1G || framework_failure
>
> # It takes many minutes to copy this sparse file using the old method.
> # By contrast, it takes far less than 1 second using FIEMAP-copy.
> --
> 1.7.1.348.gb26ba
>
>
> From 213c9247a406b267bc5819ccbe3b729e9557c675 Mon Sep 17 00:00:00 2001
> From: Jim Meyering <address@hidden>
> Date: Sat, 22 May 2010 10:21:46 +0200
> Subject: [PATCH 5/5] tests: fiemap test improvement
>
> * tests/cp/sparse-fiemap: More tests.
> ---
> tests/cp/sparse-fiemap | 69
> +++++++++++++++++++++++++-----------------------
> 1 files changed, 36 insertions(+), 33 deletions(-)
>
> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap
> index 371bced..ef3742e 100755
> --- a/tests/cp/sparse-fiemap
> +++ b/tests/cp/sparse-fiemap
> @@ -38,14 +38,14 @@ else
> mkdir mnt
> mkfs -t ext4 -F blob ||
> skip_test_ "failed to create ext4 file system"
> - mount -oloop blob mnt || skip=1
> - echo test > mnt/f || skip=1
> - test -s mnt/f || skip=1
> + mount -oloop blob mnt || skip=1
> + cd mnt || skip=1
> + echo test > f || skip=1
> + test -s f || skip=1
>
> test $skip = 1 &&
> skip_test_ "insufficient mount/ext4 support"
>
> - cd mnt || fail=1
> fi
>
> # Create a 1TiB sparse file
> @@ -67,34 +67,37 @@ test $(stat --printf %s sparse) = $(stat --printf %s
> fiemap) || fail=1
>
> $PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl'
>
> -$PERL -e 'BEGIN { $n = 16 * 1024; *F = *STDOUT }' \
> - -e 'for (1..100) { sysseek (*F, $n, 1)' \
> - -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1
> -
> -cp --sparse=always j1 j2 || fail=1
> -cmp j1 j2 || fail=1
> -
> -filefrag j1 | grep extent \
> - || skip_test_ 'skipping part of this test; you lack filefrag'
> -
> -# Here is sample filefrag output:
> -# $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \
> -# -e 'for (1..5) { sysseek(*F,$n,1)' \
> -# -e '&& syswrite *F,"."x$n or die "$!"}' > j
> -# $ filefrag -v j
> -# Filesystem type is: ef53
> -# File size of j is 163840 (40 blocks, blocksize 4096)
> -# ext logical physical expected length flags
> -# 0 4 6258884 4
> -# 1 12 6258892 6258887 4
> -# 2 20 6258900 6258895 4
> -# 3 28 6258908 6258903 4
> -# 4 36 6258916 6258911 4 eof
> -# j: 6 extents found
> -
> -# exclude the physical block numbers; they always differ
> -filefrag -v j1 | awk '/^ / {print $1,$2,$NF}' > ff1 || fail=1
> -filefrag -v j2 | awk '/^ / {print $1,$2,$NF}' > ff2 || fail=1
> -compare ff1 ff2 || fail=1
> +for i in $(seq 20); do
> + for j in 1 2 31 100; do
> + $PERL -e 'BEGIN { $n = '$i' * 1024; *F = *STDOUT }' \
> + -e 'for (1..'$j') { sysseek (*F, $n, 1)' \
> + -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1
> +
> + cp --sparse=always j1 j2 || fail=1
> + cmp j1 j2 || fail=1
> + filefrag -v j1 | grep extent \
> + || skip_test_ 'skipping part of this test; you lack filefrag'
> +
> + # Here is sample filefrag output:
> + # $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \
> + # -e 'for (1..5) { sysseek(*F,$n,1)' \
> + # -e '&& syswrite *F,"."x$n or die "$!"}' > j
> + # $ filefrag -v j
> + # Filesystem type is: ef53
> + # File size of j is 163840 (40 blocks, blocksize 4096)
> + # ext logical physical expected length flags
> + # 0 4 6258884 4
> + # 1 12 6258892 6258887 4
> + # 2 20 6258900 6258895 4
> + # 3 28 6258908 6258903 4
> + # 4 36 6258916 6258911 4 eof
> + # j: 6 extents found
> +
> + # exclude the physical block numbers; they always differ
> + filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1
> + filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1
> + compare ff1 ff2 || fail=1
> + done
> +done
>
> Exit $fail
> --
> 1.7.1.348.gb26ba
>
>
>
--
With Windows 7, Microsoft is asserting legal control over your computer and is
using this power to
abuse computer users.
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, (continued)
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Jim Meyering, 2010/05/21
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Jim Meyering, 2010/05/21
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, jeff.liu, 2010/05/21
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Jim Meyering, 2010/05/24
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, jeff.liu, 2010/05/24
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, jeff.liu, 2010/05/24
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Jim Meyering, 2010/05/25
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, jeff.liu, 2010/05/25
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, jeff.liu, 2010/05/21
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Jim Meyering, 2010/05/27
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy,
jeff.liu <=
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Sunil Mushran, 2010/05/27
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Jim Meyering, 2010/05/27
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Sunil Mushran, 2010/05/27
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Jim Meyering, 2010/05/27
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Jim Meyering, 2010/05/27
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Tao Ma, 2010/05/28
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Jim Meyering, 2010/05/28
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Tao Ma, 2010/05/30
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, Jim Meyering, 2010/05/30
- bug#6131: [PATCH]: fiemap support for efficient sparse file copy, jeff.liu, 2010/05/30