Re: Possible speedup?

[Skip Montanaro]

    Kai> Hm.  Tramp is being invoked with file-attributes only, so it would
    Kai> have to figure out how to remember the right calls.  Hm.  Oh, if it
    Kai> really reads bottom to top then that means that Tramp starts with
    Kai> /a, followed by /a/b, then /a/b/c, and so on.  So it can't know on
    Kai> the first call which subsequent calls will be needed.

Bummer.

    Kai> The whole thing probably comes from the *-truename variables.  You
    Kai> could try setting them to nil to see if that changes something.

They are all nil already.

    >> I'm sure there are other ways of accomplishing this.  I don't
    >> understand the tramp code well enough to speculate, but it seems like
    >> it would be a big improvement over the status quo.

    Kai> If my idea about *-truename is correct, then it might help to
    Kai> rework tramp-handle-file-truename.  Are those calls to
    Kai> file-attributes coming from tramp-handle-file-truename?

I'm not sure.  I tried stepping through things in edebug, but it didn't stop
many places.  I did set a breakpoint in tramp-handle-file-attributes and
noticed that it gets called several times for the file being visited, always
with the same filename arg.  If each of those is going to the remote
machine, simply caching file attribute information for a few seconds would
speed things up significantly without seriously compromising the validity of
the attribute checks.

I counted one instance.  While calling find-alternate-file with

    /[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py

as the new file, tramp-handle-file-attributes was called seven times.  The
filename arg was always the full path.  Here's the relevant section of the
*debug* buffer:

    $ ( test -d /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? )
    tramp_exit_status 1
    # Finding true name for 
`/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py'
    # Check /home
    $ test -e /home 2>/dev/null; echo tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: /[rsync/manatee.mojam.com]/home
    $ tramp_file_attributes /home
    (t 23 0 0 (16316 37202) (16196 63486) (16196 63486) 1024 16877 t (0 . 2) (0 
773))
    # Check /home/skip
    $ test -e /home/skip 2>/dev/null; echo tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: /[rsync/manatee.mojam.com]/home/skip
    $ tramp_file_attributes /home/skip
    (t 33 501 501 (16316 37154) (16313 22045) (16313 22045) 4096 16877 t (0 . 
53249) (0 773))
    # Check /home/skip/src
    $ test -e /home/skip/src 2>/dev/null; echo tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: /[rsync/manatee.mojam.com]/home/skip/src
    $ tramp_file_attributes /home/skip/src
    (t 44 501 501 (16316 37129) (16202 16222) (16202 16222) 2048 16877 t (0 . 
12289) (0 773))
    # Check /home/skip/src/projects
    $ test -e /home/skip/src/projects 2>/dev/null; echo tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: 
/[rsync/manatee.mojam.com]/home/skip/src/projects
    $ tramp_file_attributes /home/skip/src/projects
    (t 8 501 501 (16316 37090) (15906 3340) (15906 3340) 1024 16877 t (0 . 
14337) (0 773))
    # Check /home/skip/src/projects/www
    $ test -e /home/skip/src/projects/www 2>/dev/null; echo tramp_exit_status 
$? 
    tramp_exit_status 0
    # file attributes with perl: 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www
    $ tramp_file_attributes /home/skip/src/projects/www
    (t 3 501 501 (16316 37089) (16292 18773) (16292 18773) 12288 16877 t (0 . 
16385) (0 773))
    # Check /home/skip/src/projects/www/ctlevent.py
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py
    $ tramp_file_attributes /home/skip/src/projects/www/ctlevent.py
    (nil 1 501 100 (16317 11096) (16289 6785) (16289 6785) 11436 33188 t (0 . 
16433) (0 773))
    # True name of 
`/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py' is 
`/home/skip/src/projects/www/ctlevent.py'
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py
    $ tramp_file_attributes /home/skip/src/projects/www/ctlevent.py
    (nil 1 501 100 (16317 11096) (16289 6785) (16289 6785) 11436 33188 t (0 . 
16433) (0 773))
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    # Fetching 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py to tmp file 
/tmp/skip/tramp.YxBWCj...
    # Fetching 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py to tmp file 
/tmp/skip/tramp.YxBWCj...done
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py
    $ tramp_file_attributes /home/skip/src/projects/www/ctlevent.py
    (nil 1 501 100 (16317 11620) (16289 6785) (16289 6785) 11436 33188 t (0 . 
16433) (0 773))
    # Inserting local temp file `/tmp/skip/tramp.YxBWCj'...
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    $ test -w /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    $ test -e /home/skip/src/projects/www/RCS/ctlevent.py\,v 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 1
    $ test -e /home/skip/src/projects/www/ctlevent.py\,v 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 1
    $ test -e /home/skip/src/projects/www/RCS/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 1
    $ test -e /home/skip/src/projects/www/SCCS/s.ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 1
    $ test -e /home/skip/src/projects/www/s.ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 1
    $ ( test -d /home/skip/src/projects/www/CVS 2>/dev/null; echo 
tramp_exit_status $? )
    tramp_exit_status 0
    $ test -r /home/skip/src/projects/www/CVS/Entries 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    $ test -e /home/skip/src/projects/www/CVS/Entries 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    $ test -e /home/skip/src/projects/www/CVS/Entries 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    $ test -e /home/skip/src/projects/www/CVS/Entries 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    # Fetching 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/CVS/Entries to tmp file 
/tmp/skip/tramp.YxBjMp...
    # Fetching 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/CVS/Entries to tmp file 
/tmp/skip/tramp.YxBjMp...done
    # Inserting local temp file `/tmp/skip/tramp.YxBjMp'...
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py
    $ tramp_file_attributes /home/skip/src/projects/www/ctlevent.py
    (nil 1 501 100 (16317 11620) (16289 6785) (16289 6785) 11436 33188 t (0 . 
16433) (0 773))
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py
    $ tramp_file_attributes /home/skip/src/projects/www/ctlevent.py
    (nil 1 501 100 (16317 11620) (16289 6785) (16289 6785) 11436 33188 t (0 . 
16433) (0 773))
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py
    $ tramp_file_attributes /home/skip/src/projects/www/ctlevent.py
    (nil 1 501 100 (16317 11620) (16289 6785) (16289 6785) 11436 33188 t (0 . 
16433) (0 773))
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py
    $ tramp_file_attributes /home/skip/src/projects/www/ctlevent.py
    (nil 1 501 100 (16317 11620) (16289 6785) (16289 6785) 11436 33188 t (0 . 
16433) (0 773))
    $ test -e /home/skip/src/projects/www/ctlevent.py 2>/dev/null; echo 
tramp_exit_status $? 
    tramp_exit_status 0
    # file attributes with perl: 
/[rsync/manatee.mojam.com]/home/skip/src/projects/www/ctlevent.py
    $ tramp_file_attributes /home/skip/src/projects/www/ctlevent.py
    (nil 1 501 100 (16317 11620) (16289 6785) (16289 6785) 11436 33188 t (0 . 
16433) (0 773))

As you can see, the remote activity was made worse by the fact that the file
is in a remote CVS repository, so XEmacs did a little bit of poking around
there.

    >> On a related note, why is using ls to fetch file attributes not
    >> preferred over using perl?  I suspect it would be faster.

    Kai> The Perl script knows how to find out the exact timestamp.  Since
    Kai> the timestamps returned by ls -l are so imprecise, I currently told
    Kai> Tramp to ignore them.

Ah, okay.

    Kai> I guess that there are quite a number of machines with a stat or
    Kai> fstat program, which could be used.

Doesn't seem to be a significant difference between the two:

    % time perl tramp_file_attrs.pl ~/src/projects/www/ctlevent.py 
    ...
    0.01user 0.00system 0:00.01elapsed 90%CPU (0avgtext+0avgdata 0maxresident)k
    0inputs+0outputs (313major+47minor)pagefaults 0swaps

    % time stat ~/src/projects/www/ctlevent.py 
    ...
    0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k
    0inputs+0outputs (140major+24minor)pagefaults 0swaps

so I doubt an ls-based option would improve things.  The main culprit has to
be the network overhead, which will be approximately the same no matter how
you read the file attributes.  My apologies for suggesting differently in my
earlier message.

    Kai> I asked around for different stat programs and how to parse their
    Kai> output, but nothing ever came out of it.  That is, people responded
    Kai> but I didn't do anything with that information.

I wouldn't worry about it.  Avoiding trips to the remote machine looks like
it will be a much more important source of time savings.

    Kai> Does anyone want to work on this?

If it was written in Python, I'd have it done already. ;-)

Skip