Re: [Bug-wget] [PATCH] Keep fetched URLs in POSIX extended attributes

[Tim Rühsen]

Thanks, applied & pushed.

Tim

On Wednesday, July 27, 2016 9:39:53 AM CEST Sean Burford wrote:
> Hi,
> 
> Sorry, my previous replies to various concerns were all direct rather than
> to the list.  In summary, I promised to fix the style issues and I also
> agreed that DEBUG on failure to set xattr was a good approach.  Also, I
> confirmed that I work for Google.
> 
> Attached is a patch for the style issues and DEBUG.
> 
> On Fri, Jul 22, 2016 at 10:39 PM, Ander Juaristi <address@hidden> wrote:
> > Also, shouldn't we print something when xattrs are not supported? In
> > Linux fsetxattr() returns -1 when that is the case.
> > 
> > <pedantic_comment>
> > Extended attributes are not supported by all filesystems, eg:
> > 
> > dd if=/dev/zero bs=64M count=1 of=./minix.img
> > mkfs.minix ./minix.img
> > mount -o loop -t minix minix.img /tmp/minix-test
> > 
> > And then:
> > wget --xattr -O /tmp/minix-test/archive.html http://archive.org
> > getfattr -d /tmp/minix-test/archive.html
> > <empty>
> > 
> > Yeah, Minix is not used in the real world, but NTFS is (sometimes in the
> > form of portable USB drives) and I think it also does not support
> > extended attributes. Didn't try it - more hassle to set up in 5 minutes.
> > 
> > If we trust Wikipedia [1], other filesystems from the MS world (such as
> > FAT) do not support xattrs either.
> > </pedantic_comment>
> > 
> > [1] https://en.wikipedia.org/wiki/Extended_file_attributes#Linux
> > 
> > On 21/07/16 06:33, Sean Burford wrote:
> > > Hi,
> > > 
> > > I find it useful to keep track of where files are downloaded from. 
> > > POSIX
> > > extended attributes provide a lightweight portable method of keeping
> > > this
> > > information across Linux, OS/X, FreeBSD and many other platforms.
> > > 
> > > This compliments wget's existing WARC support, which serves a related
> > > but
> > > different use case closer to tcpdump or tar for web pages.  Extended
> > > attributes can provide a quick answer to "where did I get this file from
> > > again?"
> > > 
> > > This patch changes:
> > > *   autoconf detects whether extended attributes are available and
> > 
> > enables
> > 
> > > the code if they are.
> > > *   The new flags --xattr and --no-xattr control whether xattr is
> > 
> > enabled.
> > 
> > > *   The new command "xattr = (on|off)" can be used in ~/.wgetrc or
> > > /etc/wgetrc
> > > *   The original and redirected URLs are recorded as shown below.
> > > *   This works for both single fetches and recursive mode.
> > > 
> > > Here is an example, where http://archive.org redirects to
> > > https://archive.org:
> > > $ wget --xattr http://archive.org
> > > ...
> > > $ getfattr -d index.html
> > > user.xdg.origin.url="https://archive.org/";
> > > user.xdg.referrer.url="http://archive.org/";
> > > 
> > > These attributes were chosen based on those stored by Google Chrome (
> > > https://bugs.chromium.org/p/chromium/issues/detail?id=45903) and curl (
> > > https://github.com/curl/curl/blob/master/src/tool_xattr.c)

-- 
OMS Open Media System GmbH
Holzdamm 40
20099 Hamburg
Fon +49-40-238878-40
Fax +49-40-238878-99
Email address@hidden
Sitz und Registergericht Hamburg
HRB 57616