Re: Possible speedup?

[Kai Grossjohann]

Skip Montanaro <address@hidden> writes:

>     Kai> Hm.  Tramp is being invoked with file-attributes only, so it would
>     Kai> have to figure out how to remember the right calls.  Hm.  Oh, if it
>     Kai> really reads bottom to top then that means that Tramp starts with
>     Kai> /a, followed by /a/b, then /a/b/c, and so on.  So it can't know on
>     Kai> the first call which subsequent calls will be needed.
>
> Bummer.

Yeah...

>     Kai> The whole thing probably comes from the *-truename variables.  You
>     Kai> could try setting them to nil to see if that changes something.
>
> They are all nil already.

But the log tells me it's finding the true name anyway.  Interesting.

Could you invoke file-truename on that file to find out if it perhaps
produces similar sequences of file-attributes?

>     >> I'm sure there are other ways of accomplishing this.  I don't
>     >> understand the tramp code well enough to speculate, but it seems like
>     >> it would be a big improvement over the status quo.
>
>     Kai> If my idea about *-truename is correct, then it might help to
>     Kai> rework tramp-handle-file-truename.  Are those calls to
>     Kai> file-attributes coming from tramp-handle-file-truename?
>
> I'm not sure.  I tried stepping through things in edebug, but it didn't stop
> many places.  I did set a breakpoint in tramp-handle-file-attributes and
> noticed that it gets called several times for the file being visited, always
> with the same filename arg.  If each of those is going to the remote
> machine, simply caching file attribute information for a few seconds would
> speed things up significantly without seriously compromising the validity of
> the attribute checks.
>
> I counted one instance.  While calling find-alternate-file with
>
>     /[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py
>
> as the new file, tramp-handle-file-attributes was called seven times.  The
> filename arg was always the full path.  Here's the relevant section of the
> *debug* buffer:
>
>     $ ( test -d /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
> tramp_exit_status $? )
>     tramp_exit_status 1
>     # Finding true name for 
> `/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py'

Ah, it's finding the true name.  I think that msg comes from
tramp-handle-file-truename, so changing that function to kind of
precompute all those file-attribute calls would help.

Hm.  But that would require rewriting the file-truename logic as a
shell script to be sent to the remote host.  What it does is this:
Check if /home is a symlink elsewhere.  If so, redo from start with
the symlink target instead of /home.  If not, look if /home/skip is a
symlink elsewhere.  And so on.

At least that's what my foggy memory tells me.

> As you can see, the remote activity was made worse by the fact that the file
> is in a remote CVS repository, so XEmacs did a little bit of poking around
> there.

Hm.

>     Kai> fstat program, which could be used.
>
> Doesn't seem to be a significant difference between the two:
>
>     % time perl tramp_file_attrs.pl ~/src/projects/www/ctlevent.py 
>     ...
>     0.01user 0.00system 0:00.01elapsed 90%CPU (0avgtext+0avgdata 
> 0maxresident)k
>     0inputs+0outputs (313major+47minor)pagefaults 0swaps
>
>     % time stat ~/src/projects/www/ctlevent.py 
>     ...
>     0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k
>     0inputs+0outputs (140major+24minor)pagefaults 0swaps
>
> so I doubt an ls-based option would improve things.  The main culprit has to
> be the network overhead, which will be approximately the same no matter how
> you read the file attributes.  My apologies for suggesting differently in my
> earlier message.

Well, I thought it possible that the Perl startup times are the
problem.  But you're right; I also think that the network overhead is
the prime contender for performance bottlenecks.

>     Kai> I asked around for different stat programs and how to parse their
>     Kai> output, but nothing ever came out of it.  That is, people responded
>     Kai> but I didn't do anything with that information.
>
> I wouldn't worry about it.  Avoiding trips to the remote machine looks like
> it will be a much more important source of time savings.

Heh ;-)

>     Kai> Does anyone want to work on this?
>
> If it was written in Python, I'd have it done already. ;-)

Well, you just said you didn't think parsing the output from fstat
makes sense ;-)

But maybe something that does the logic of file-truename in sh (in a
portable way!) would be cool.

Kai