Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?

[Brandon Whaley]

I think I see what the biggest point of confusion here is.  When fab
executes a task but no -H parameters are set and the decorator
specifies no host, the Connection object is instead an invoke Context
object, which has no .local.  I agree that that's not intuitive, but
I'm not sure where that choice is made in the fabric codebase.

On Tue, Apr 2, 2019 at 2:50 PM Michel Albert <address@hidden> wrote:
>
> Creating a new Context from the connection config did indeed work. So I am 
> planning - for now - to go ahead with this.
>
> I hear you on the point of separating invoke from fabric tasks.
>
> To elaborate, and please don't read the following as a kind of rant, just as 
> an elaboration where the need to mix remote & local tasks comes from.
>
> My current aim is to migrate an old, rather large and complex collection of 
> fabric-1.x tasks to fabric-2.x And many of those tasks mix both remote and 
> local commands. My aim is to keep everything available under the "umbrella" 
> "fab" executable. Many tasks are shared among an ever growing team via a 
> library made available to them. I really want to avoid having two executables 
> where it may not be clear from the outside which task needs to be executed 
> using invoke and which one needs fabric. The end-goal is to have a task 
> runner which has the same core tasks for all of our projects. Namely:
>
> develop - to create a new development environment
> build - To build and bundle whatever is necessary for deployment
> test - To run unit-tests
> deploy - To deploy to a targeted environment
>
> We chose fabric in the past because it is really flexible, and lets us use it 
> even for non-Python projects. And most of all it is extensible. So most 
> projects have more than these 4 core tasks. Not being able to determine 
> whether to use "inv" or "fab" would quickly become quite annoying.
>
> The library has grown over the past and it helped us a lot in our day-to-day 
> business. Splitting the tasks now into an invoke- and fabric-library is 
> definitely something I will take a closer look at, but I am afraid that some 
> of the more complex tasks might lose in readability when doing so. I will see.
>
> In any case, your feedback helped me a lot. I wonder if this would be a 
> useful example to put into the docs (that is,calling an invoke task from a 
> fabric task). It does not seem to be something very common, but I had a lot 
> of trouble finding a solution without falling back to this mailing list.
>
> On Tue, Apr 2, 2019 at 5:35 PM Brandon Whaley <address@hidden> wrote:
>>
>> Have you tried instantiating a context with the config from Fabric's
>> connection?  I haven't mixed invoke tasks and fabric tasks in the same
>> module before, I'm surprised the same task decorator will work for
>> both using `inv` on the command line and `fab` as well.  If you have a
>> task that only does local work inside your fabfile module, I'd
>> recommend using conn.local instead of making it an invoke task.  If
>> you have an existing invoke task codebase, I would import that module
>> from my fabfile module and run tasks with a locally created context,
>> since you're not calling the task from the invoke cli.
>>
>> ```
>> tasks.py
>> ====
>> from invoke.tasks import task
>>
>> @task
>> def build(ctx):
>>    ctx.run('make artifact')
>>
>> fabfile.py
>> ====
>> from fabric.tasks import task
>> from invoke.context import Context
>> from tasks import build
>>
>> @task(hosts=PROD)
>> def deploy(conn):
>>     ctx = Context(config=conn.config)
>>     build(ctx)
>>     conn.put('artifact', '/app')
>> ```
>>
>> On Tue, Apr 2, 2019 at 2:04 AM Michel Albert <address@hidden> wrote:
>> >
>> > I understand the differences. The only traceback I  got was the one I sent 
>> > earlier. This traceback is correct, only maybe a bit cryptic.
>> >
>> > So far, I managed to get some basic tasks ported to fabric2.
>> >
>> > However, this morning I ran into a new problem:
>> >
>> > I have a remote task (deploy) which depends on a local task (build) and I 
>> > want to be able to call the build step separately. But now I don't know 
>> > how to call "build" from the "deploy" task. The docs state to simply call 
>> > the function and forward the connection/context. But when calling from the 
>> > "deploy" task, I have a "Connection" instance, but I need a "Context" 
>> > instance to pass into the "build" task. Is there a way to get the Invoke 
>> > context from the Connection instance? Example:
>> >
>> > @task
>> > def build(ctx):
>> >    ctx.run('make artifact')
>> >
>> > @task(hosts=PROD)
>> > def deploy(conn):
>> >     build(?)  # <- what do I need to pass onto build? "conn" won't work
>> >     conn.put('artifact', '/app')
>> >
>> > On Mon, Apr 1, 2019 at 9:33 PM Brandon Whaley <address@hidden> wrote:
>> >>
>> >> There is no local attribute on invoke contexts, only on fabric
>> >> connections (which inherits from invoke contexts).  If this isn't what
>> >> you're trying to do, could you provide some code examples that give
>> >> you the traceback you're seeing?
>> >>
>> >> On Mon, Apr 1, 2019 at 1:07 PM Michel Albert <address@hidden> wrote:
>> >> >
>> >> > Ok. I think I will get it to work like that.
>> >> >
>> >> > But after fiddling around with it I found out that the type of the 
>> >> > first argument depends on whether a "hosts" argument was passed into 
>> >> > the task decorator or not. This was quite confusing. And the error is a 
>> >> > bit cryptic. When calling "local" on an invoke context, the following 
>> >> > AttributeError is thrown:
>> >> >
>> >> > Traceback (most recent call last):
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/config.py", line 113, 
>> >> > in __getattr__
>> >> >     return self._get(key)
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/config.py", line 178, 
>> >> > in _get
>> >> >     value = self._config[key]
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/config.py", line 169, 
>> >> > in __getitem__
>> >> >     return self._get(key)
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/config.py", line 178, 
>> >> > in _get
>> >> >     value = self._config[key]
>> >> > KeyError: 'local'
>> >> >
>> >> > During handling of the above exception, another exception occurred:
>> >> >
>> >> > Traceback (most recent call last):
>> >> >   File "/home/exhuma/.local/bin/fab", line 10, in <module>
>> >> >     sys.exit(program.run())
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/program.py", line 363, 
>> >> > in run
>> >> >     self.execute()
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/program.py", line 532, 
>> >> > in execute
>> >> >     executor.execute(*self.tasks)
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/executor.py", line 129, 
>> >> > in execute
>> >> >     result = call.task(*args, **call.kwargs)
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/tasks.py", line 128, in 
>> >> > __call__
>> >> >     result = self.body(*args, **kwargs)
>> >> >   File "/home/exhuma/tmp/fabfile.py", line 8, in get_version
>> >> >     version = ctx.local('python setup.py --version').strip()
>> >> >   File "/usr/lib/python3.7/site-packages/invoke/config.py", line 125, 
>> >> > in __getattr__
>> >> >     raise AttributeError(err)
>> >> > AttributeError: No attribute or config key found for 'local'
>> >> >
>> >> > Valid keys: ['connect_kwargs', 'forward_agent', 'gateway', 
>> >> > 'inline_ssh_env', 'load_ssh_configs', 'port', 'run', 'runners', 
>> >> > 'ssh_config_path', 'sudo', 'tasks', 'timeouts', 'user']
>> >> >
>> >> > Valid real attributes: ['cd', 'clear', 'config', 'cwd', 'from_data', 
>> >> > 'pop', 'popitem', 'prefix', 'run', 'setdefault', 'sudo', 'update']
>> >> >
>> >> >
>> >> > On Mon, Apr 1, 2019 at 5:37 PM Brandon Whaley <address@hidden> wrote:
>> >> >>
>> >> >> Hi Mich,
>> >> >>
>> >> >> The connection object (you're using ctx in your examples) has a .local
>> >> >> method that is just a pass-through to invoke.run.  It's documented on
>> >> >> the connection object's page:
>> >> >> http://docs.fabfile.org/en/2.4/api/connection.html?highlight=local#fabric.connection.Connection.local
>> >> >>
>> >> >> On Mon, Apr 1, 2019 at 5:55 AM Michel Albert <address@hidden> wrote:
>> >> >> >
>> >> >> > Hi,
>> >> >> >
>> >> >> >
>> >> >> > Consider the following fabric-1 task. For illustration I kept it 
>> >> >> > really short:
>> >> >> >
>> >> >> > @fab.task
>> >> >> > def sample():
>> >> >> >     version = fab.local('python setup.py --version')
>> >> >> >     fab.run('mkdir -p /snapshots/%s' % version.strip())
>> >> >> >
>> >> >> > This task needs to run a local and remote command. I am now trying 
>> >> >> > to port this to fabric-2, and I can't figure out how I can implement 
>> >> >> > this. If I define the "hosts" variable in the task, then the first 
>> >> >> > line will be executed on the remote host as well, which I don't 
>> >> >> > want. A naive aproach which won't work:
>> >> >> >
>> >> >> > @task(hosts=PROD)
>> >> >> > def sample(ctx):
>> >> >> >     version = ctx.run('python setup.py --version').strip()   # <- 
>> >> >> > this won't work
>> >> >> >     ctx.run('mkdir -p /snapshots/%s' % version)
>> >> >> >
>> >> >> > At first I thought I would split the task into two, one for just 
>> >> >> > local commands and one for remote tasks, but then I am forced to 
>> >> >> > pass in the context, which will in turn cause it again to be run 
>> >> >> > remotely:
>> >> >> >
>> >> >> > @task
>> >> >> > def get_version(ctx):
>> >> >> >     version = ctx.run('python setup.py --version').strip()
>> >> >> >     return version
>> >> >> >
>> >> >> > @task(hosts=PROD)
>> >> >> > def sample(ctx):
>> >> >> >     version = get_version(ctx)  # <- this won't work
>> >> >> >     ctx.run('mkdir -p /snapshots/%s' % version)
>> >> >> >
>> >> >> > How can I accomplish something like this? And where is it noted in 
>> >> >> > the docs? In the current example on the "Upgrading from 1.x" page 
>> >> >> > does not have a single task mixing local with remote commands in any 
>> >> >> > way.
>> >> >> >
>> >> >> >
>> >> >> > Regards,
>> >> >> >
>> >> >> >
>> >> >> > Mich
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > Fab-user mailing list
>> >> >> > address@hidden
>> >> >> > https://lists.nongnu.org/mailman/listinfo/fab-user