Some thoughts on making tramp faster

[CeleritasCelery]

I found this project on reddit and I am excited that you are working on it. I have recently been working from a position where I use tramp all day for work and hence run into the performance issue with it first hand. As I understand it, you are using libssh to create an Emacs module to use as a tramp handler.

What actually makes this faster? As I understand it, you still need to send shell commands to the remote host and read the replies. I can see that setting up the commands and parsing the output is faster in Rust than lisp. Is that essentially what is making the difference? Do you have a sense of how much faster it is?

I have been doing some profiling of tramp to to understand where time is being spent. Here is a couple of my findings:

Case 1: Copying 27 files from one directory on the remote host to another directory using dired

90% of the time is spent in tramp-send-command waiting for output.

We end up with 350 commands being sent across SSH.

88  file-directory-p
55  file-exists-p (one for source and dest + directory)
55 file-truename (one for source and dest + directory)
27 copy
27 file-readable-p
27 calls to ls. I think this is for file-modes?
57 file-attributes (one for source and dest + directory)
27 calls to cd (I think this is for the call to ls)
1 call to tramp_perl_file_name_all_completions

Note that each command is sent serially. Meaning that it can't start the next command until it get the result back from the current one. Even if tramp-libssh was able to have multiple threads with new connections, it wouldn't be able to use them because we are getting the commands one at a time from elisp, which is single threaded.

Case 2: Opening a file over tramp

Believe it or not, actually opening the file is not the slow part. There are a bunch of auxilary commands that come with it.

41 calls to tramp-send-command

16 file-directory-p
6 file-exists-p
2 file-readable-p
5 calls to git for `magit-toplevel`
4 file-attributes
1 file-readable-p
1 file-symlink-p
1 call to open the file
1 call to `tramp_bundle_read_file_names` to check for `.git` in each parent directory
1 call to `tramp_perl_file_name_all_completions`

there are six calls to check for set_system_mode exists in both the current directory and the root. This is part of ffap-guess-file-name-at-point in file-name-at-point-functions.

It has to check the directories of each level of hierarchy going all the way to the top. This is for locate-dominating-file as part of vc-find-root.

Checks for git directories going all the way to the top as well. This is part of forge-bug-reference-setup in find-file-hook calling vc-registered. But there is a tramp specific backend for vc-registered that handles it with a "bundled" command that checks them all at once.

How can we reduce the number of calls?

Making the calls the faster is certainly an improvement, but I wonder if we can't reduce the number of total calls. My approach is to have a client-side and sever-side helper programs. The server-side one is essentially a file watcher. it looks at directories that tramp has interacted with in the past and looks for any changes (modified files, new files, deleted files, changes in permission etc). If it find any it sends them to the client side program. When tramp is going to send a command, it will send it to the client-side program (which could just be a Emacs dynamic module). If it has the data cached, it just returns the value. If not, it sends a command to the server-side to get the data. This would essentially make tramp as fast as working only client side once you have opened a directory.

Curious your thoughts.

[jsadusk]

So the reason this is hopefully faster is because I'm precisely not going to use the shell parsing of tramp. My goal is to implement all tramp handling calls using the features of the ssh protocol. Directory traversal and handling, and file reading/writing are done using sftp. Some of those operations work already (though nowhere near enough to replace the shell parser). When done using sftp, these operations are much more lightweight, because they translate into simple system calls on the remote side, instead of starting a whole process to do something like a directory listing. In addition, I am aggressively reusing sessions and channels to minimize connection startup cost.

For async processes, which tramp is especially slow with, my goal is to open up exec channels using a shared session, then use ssh's channel select feature to forward std[in|out|err] to emacs processes objects. Async sessions are handled on a separate thread, with buffering between emacs and the ssh session, to prevent one from deadlocking the other. This feature is not done yet, I've only started on it.

One of my theories on why this will make the whole system faster is that async processes will be on a separate thread from sync operations. Emacs, being single threaded, currently won't process the IO from an async process if its waiting on the result of a sync operation through the shell. Having async stuff running independently on a thread, using a separate session from the sync stuff, should remove ordering issues. In theory. Again, I don't have POC code for this yet.

[jsadusk]

Also, I have thought about the remote side server idea. It would definitely save round trips. But the power of tramp is that it can operate anywhere without a server install. I'm hoping that I can use ssh itself as the server, if we use all the features of the ssh protocol.

That said, there is a command already on most linux machines that can act as a file watch server, inotifywait. I do plan to add support to handle directory notifications via inotifywait running over an ssh channel.

[CeleritasCelery]

For async processes, which tramp is especially slow with

Have you tried using direct async processes? They made async processes much faster for me.

But the power of tramp is that it can operate anywhere without a server install. I'm hoping that I can use ssh itself as the server, if we use all the features of the ssh protocol.

I will admit that I am quite ignorant about the possibilities of the SSH protocol. So there might be a lot that can be done with API that you can't do with the binary.

As far as "running without a server install" Tramp right now will sometimes copy a perl program over to the server to do some processing. Vscode does the same thing, but with it's binary blob. So you don't need to manually install something, the handshaking takes care of that for you.

[jsadusk]

My experience is that direct async processes we're not reliable, some features like magit and eglot wouldn't work properly with that feature enabled.

That said, it's still a later of indirection and if anything this is an experiment to see how many layers of indirection I can remove. It might not be as beneficial as I hope, if so it was a fun learning experience.

That said, another idea I had was to use an emacs instance on the remote side as a remote execution server. I can imagine a local emacs instance sending elisp bytecode over the wire as a form of rpc. It could even initialize functions defined in the local context on the remote side by sending over the bytecode of the function definition. I'm not sure how you'd manage dynamic scope variables or lexical captures, however.

[CeleritasCelery]

To fix the issues with magit you need to set

(setq magit-tramp-pipe-stty-settings 'pty)

I have not used eglot before though.

But I agree that removing layers of indirection would certainly be beneficial.

Are you interested in contributors to this project? I know both Rust and Emacs well. I am not very familiar with the SSH protocol, but can learn it.

[jsadusk]

Happy to have contributors. I fear the state of the project is too early for anyone to build off of my work, and I'm not able to commit to consistent contributions myself (demanding job, two small kids, I work on it when I can which is not that often). I'm looking at your work with rune, and yes you're more than capable of doing this project. I haven't completely flushed out my approach for async processes, I'd love to run the architecture by you when we have the chance. In the meantime, I should clean up and document the code that's here, so its usable to anyone but me. In particular I need to delete the old C code I abandoned, and I need to port the tests I wrote as ad-hoc elisp functions to something like ert.