Re: [Fab-user] rsync_project

[Christian Vest Hansen]

I think you misunderstood me!

I had already pushed a change. Or maybe you're building on top of that?

On Fri, Apr 3, 2009 at 10:34 PM, Paul Baumgart <address@hidden> wrote:
> Ok, fair enough. :-)
>
> How's this then:
>
> @operation
> def rsync_project(remotedir, exclude=[], delete=False, extra_opts='',
> **kwargs):
>     """
>     Uploads the current project directory using rsync.
>     By using rsync, only changes since the last upload are actually sent
> over
>     the wire, rather than the whole directory like when using
> upload_project.
>
>     Requires the rsync command-line utility to be available both on the
> local
>     and the remote machine.
>
>     Parameters are:
>         remotedir:          the directory on the remote machine to which to
>                             rsync the current project. The project directory
>                             becomes a subdirectory of the remotedir.
>         exclude (optional): values passed to rsync's --exclude option.
>                             If the parameter object is iterable, (that is,
> it
>                             defines an __iter__ method), each of its
> elements is
>                             passed to a separate --exclude option.
> Otherwise,
>                             the object is passed as a string to a single
>                             --exclude option.
>                             See the rsync manpage for details on specifying
>                             filter rule arguments for --exclude.
>         delete (optional):  True or False, whether to delete remote files
> that
>                             don't exist locally. Defaults to False.
>         extra_opts (optional): Additional command-line options to set for
> rsync.
>
>     The rsync command is built from the options as follows:
>         rsync [--delete] [--exclude exclude[0][, --exclude[1][, ...]]] \\
>             -pthrvz [extra_opts] ../<project dir>
> <fab_user>@<host>:<remotedir>
>     """
>     username = ENV.get('fab_user')
>
>     if not hasattr(exclude, '__iter__'):
>         exclude = [exclude]
>
>     exclude_opts = ' --exclude "%s"' * len(exclude)
>     exclusions = tuple([str(s).replace('"', '\\\\"') for s in exclude])
>
>     options_map = {
>         "delete"  : '--delete' if delete else '',
>         "exclude" : exclude_opts % exclusions,
>         "extra"   : extra_opts
>     }
>     options = "%(delete)s%(exclude)s -pthrvz %(extra)s" % options_map
>     cwd = '../' + os.getcwd().split(os.sep)[-1]
>     userhost = "$(fab_user)@$(fab_host)"
>     rdir = _lazy_format(remotedir, ENV)
>
>     cmd = "rsync %s %s %s:%s" % (options, cwd, userhost, rdir)
>     local_per_host(cmd, **kwargs)
>
>
> Besides reducing the length of lines, I also made it more pythonic by using
> duck typing to determine how to handle the exclude parameter. So now you can
> pass integers, complex numbers, or anything else with a __str__ method and
> it will work.
>
> I also cleaned up the documentation a bit and made some parts of it more
> precise.
>
> Paul
>
>
>
>
>
> On Fri, Apr 3, 2009 at 10:50 AM, Christian Vest Hansen
> <address@hidden> wrote:
>>
>> Good idea. But as you may have guessed, I'm not fond of long lines :)
>>
>> On Fri, Apr 3, 2009 at 5:21 AM, Paul Baumgart <address@hidden> wrote:
>> > Hi Christian,
>> >
>> > Thanks, your version looks a lot cleaner.
>> >
>> > I would like to add one thing, because I think it makes the exclude
>> > parameter much more useful:
>> >
>> > @operation
>> > def rsync_project(remotepath, exclude=False, delete=False,
>> > extra_opts='',
>> > **kwargs):
>> >    """
>> >    Uploads the current project directory using rsync.
>> >    By using rsync, only changes since last upload are actually sent over
>> >    the wire, rather than the whole directory like using upload_project.
>> >
>> >    Requires the rsync command-line utility to be available both on the
>> > local
>> >    and the remote machine.
>> >
>> >    Parameters are:
>> >        remotepath:         the path on the remote machine to which to
>> > rsync
>> > the
>> >                            current project
>> >        exclude (optional): values passed to rsync's --exclude option.
>> >                            If the parameter is iterable, each of its
>> > elements
>> >                            is passed to a separate --exclude argument.
>> > If
>> > the
>> >                            parameter is a string, that string is the
>> > value
>> >                            passed to the single --exclude argument.
>> >                            See the rsync manpage for details on using
>> > the
>> >                            --exclude option.
>> >        delete (optional):  True or False, whether to delete remote files
>> > that
>> >                            don't exist locally.
>> >        extra_opts (optional): Additional command-line options to set for
>> > rsync.
>> >
>> >    The rsync command is built from the options as follows:
>> >        rsync [--delete] [--exclude exclude] -pthrvz [extra_opts] \\
>> >            <project dir> <fab_user>@<host>:<remotepath>
>> >    """
>> >    username = ENV.get('fab_user')
>> >
>> >    if isinstance(exclude, basestring):
>> >        exclude = [exclude]
>> >
>> >    options_map = {
>> >        "delete" : '--delete' if delete else '',
>> >        "exclude" : exclude and ' '.join(['--exclude "%s"' %
>> > opt.replace('"',
>> > '\\\\"') for opt in exclude]) or '',
>> >        "extra" : extra_opts
>> >    }
>> >    options = "%(delete)s %(exclude)s -pthrvz %(extra)s" % options_map
>> >    cwd = '../' + os.getcwd().split(os.sep)[-1]
>> >    userhost = "$(fab_user)@$(fab_host)"
>> >    rpath = _lazy_format(remotepath, ENV)
>> >
>> >    cmd = "rsync %s %s %s:%s" % (options, cwd, userhost, rpath)
>> >    local_per_host(cmd, **kwargs)
>> >
>> > So, instead of taking just a string, it can take either a string or a
>> > list,
>> > and multiple --exclude options are generated from a list. A string
>> > parameter
>> > is treated just as it is currently.
>> >
>> > Paul
>> >
>> > On Thu, Apr 2, 2009 at 12:38 PM, Christian Vest Hansen
>> > <address@hidden> wrote:
>> >> Ok. I implemented this change, but I modified it a little bit.
>> >>
>> >> On Thu, Apr 2, 2009 at 9:17 PM, Christian Vest Hansen
>> >> <address@hidden> wrote:
>> >>> On Sun, Mar 22, 2009 at 12:51 AM, Paul Baumgart <address@hidden>
>> >>> wrote:
>> >>>> A different topic this time:
>> >>>>
>> >>>> The current upload_project() function is a neat feature, but I have
>> >>>> two issues with it:
>> >>>>
>> >>>> 1) It doesn't allow me to exclude certain files/directories (like
>> >>>> .git* for example)
>> >>>> 2) It uploads the entire project every time, which can be irritating
>> >>>> if the change is small but the project is large.
>> >>>>
>> >>>> So, I made this function, which uses rsync to only upload the
>> >>>> differences between the local and remote copies.
>> >>>>
>> >>>> I would like to get feedback on it, primarily as to whether this
>> >>>> would
>> >>>> be useful in the mainline Fabric code, and if not, if there is any
>> >>>> way
>> >>>> to make it be useful.
>> >>>>
>> >>>> @operation
>> >>>> @connects
>> >>>> def rsync_project(host, client, env, remotepath, exclude=False,
>> >>>> delete=False, extra_opts='', **kwargs):
>> >>>>    """
>> >>>>    Uploads the current project directory using rsync, so only changes
>> >>>> are
>> >>>>    uploaded rather than the whole directory like using
>> >>>> upload_project.
>> >>>>
>> >>>>    Requires the rsync command-line utility to be available both on
>> >>>> the
>> >>>> local
>> >>>>    and the remote machine.
>> >>>>
>> >>>>    Parameters are:
>> >>>>        remotepath:         the path on the remote machine to which to
>> >>>> rsync the
>> >>>>                            current project
>> >>>>        exclude (optional): the string passed to rsync's --exclude
>> >>>> option.
>> >>>>                            See the rsync manpage for details.
>> >>>>        delete (optional):  True or False, whether to delete remote
>> >>>> files
>> >>>> that
>> >>>>                            don't exist locally.
>> >>>>        extra_opts (optional): Additional command-line options to set
>> >>>> for
>> >>>> rsync.
>> >>>>
>> >>>>    The rsync command is built from the options as follows:
>> >>>>        rsync [--delete] [--exclude exclude] -pthrvz [extra_opts]
>> >>>> <project dir> <fab_user>@<host>:<remotepath>
>> >>>>    """
>> >>>>
>> >>>>    remotepath = _lazy_format(remotepath, env)
>> >>>>
>> >>>>    username = ENV.get('fab_user')
>> >>>>    username = username + '@' if username else username
>> >>>>
>> >>>>    cwd_name = '../' + os.getcwd().split(os.sep)[-1]
>> >>>
>> >>> This line right here. Why do you go through all this trouble to build
>> >>> a relative CWD?
>> >>
>> >> Elementary, dear Watson. To get a prettier output when the assembled
>> >> command is printed to the console.
>> >>
>> >>>
>> >>>
>> >>>>
>> >>>>    delete_opt = '--delete' if delete else ''
>> >>>>
>> >>>>    exclude_opt = '--exclude' if exclude else ''
>> >>>>    exclude = '"' + exclude.strip('"') + '"' if exclude else ''
>> >>>>
>> >>>>    return local('rsync %(delete_opt)s %(exclude_opt)s %(exclude)s
>> >>>> -pthrvz %(extra_opts)s %(cwd_name)s
>> >>>> %(username)s%(host)s:%(remotepath)s'
>> >>>>        % locals(), **kwargs) == 0
>> >>>>
>> >>>>
>> >>>> Note that it requires adding a line to the bottom of local():
>> >>>>
>> >>>> return retcode
>> >>>>
>> >>>> Regards,
>> >>>> Paul
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Fab-user mailing list
>> >>>> address@hidden
>> >>>> http://lists.nongnu.org/mailman/listinfo/fab-user
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Venlig hilsen / Kind regards,
>> >>> Christian Vest Hansen.
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Venlig hilsen / Kind regards,
>> >> Christian Vest Hansen.
>> >>
>> >
>> >
>>
>>
>>
>> --
>> Venlig hilsen / Kind regards,
>> Christian Vest Hansen.
>
>



-- 
Venlig hilsen / Kind regards,
Christian Vest Hansen.