1) mosh lowers the average latency, but it causes the perceived latency to be highly variable.
The issue is that there are various moments where the algorithm somewhat-correctly decides it cannot predict what will happen due to new incoming keystrokes. An example of this is after you hit enter: you might be running an entirely different program at this point.
In essence, the algorithm "warms up" as it makes correct decisions, and once it decides "ok, my decisions seem accurate: when he types a character they seem to echo back to the terminal" it makes a sudden transition from "slightly slower than raw SSH" to "much faster than SSH and only slightly slower than a local screen session".
These sudden transitions are jarring: when they happen to you with normal connections the expectation is "oh, my connection is failing", a feeling that with mosh you have every time you type a command.
Meanwhile, for the interactive-shell or complex-text-editor use case, these "unpredictable events" always happen at the worst possible moment: when you first start doing something; the result is that the perceived incidence of latency is actually worse than with normal SSH, as every "new event" has a higher latency... by the time it starts prediction, you are probably "in the flow" enough to not even be paying attention.
2) it relies on a specific range of UDP ports
Because it operates over a custom supposedly-mobile-optimized protocol, it tends to end up being inaccessible when you actually need it due to pervasive modern firewalls. You'll start a session at home, go to the office, and... nothing... because you can't communicate over that UDP port. This happened to me quite often while evaluating mosh.
3) it doesn't have a good way to reconnect later
While it supports sitting down at a different computer and taking over your session, it requires you to drop to a rather low-level mechanism based on knowing the secret session key that was established at the time the server was setup. This actually undermines the "mobile" use case, as it only lets the connection be mobile, not the session.
(I also have some specific issues with a library it relies on for spoofing hostnames in wtmp: just having that library installed on your system opens a vulnerability where any non-privileged user can spoof their hostname as anyone else... their opinion differs, but it seems to mostly be due to a knee-jerk reaction of "you don't get fired for avoiding setuid" than "this is actually secure".)
The result is that after spending a bunch of time testing it, working with the codebase, getting frustrated at a bunch of the design decisions, and then having it both totally fail to work in real-world network settings and become yet another source of infuriating latency, I pretty much dropped all interest in the project and replaced it with autossh, which worked great.
The primary downside of this is that it doesn't handle the packet loss problem well, as you are really just still using normal SSH to do the connections. The session might get weirdly slow and even lock up in these packet-loss situations.
However: it doesn't require any special code on the server, which is a downside of mosh I didn't even list above (as it is fairly obvious and something you know going in): this means you get to use it for every connection to every server you might ever care about.
Otherwise, you get everything you expect from normal ssh and screen: it works everywhere you might theoretically have been able to connect to the server anyway (so no ending up at the client site and finding out that they firewall UDP), it is obvious how to reconnect to (or even share) the screen session from a different computer or a different client without special software, and you don't need to even contemplate "what if this introduces a security vulnerability by running a new daemon on my server?".
The other annoyance of using autossh is it is pretty adamant about doing exponential back-off on reconnect, and in the naive way I have it setup in that function it is also going to be doing slow-to-fail DNS requests as it attempts to do the reconnection sequence, so the result is that the "open laptop, wait for all terminals to connect" sequence doesn't always click perfectly.
However, that is easily fixable by using a simpler wrapper (to be clear: use autossh when autossh is appropriate; it is designed, however, for server-to-server backend links to maintain persistent port forwarding), so I bring you cysh.sh, which is what I actually use (and which I've done some testing with others to verify works on random platforms):
(Yes: you are now downloading a random shell script from someone on a web forum. However, it's like 35 lines of very easily readable sh: you should just read it; hell, after reading it, you might consider it so simple you just want to rewrite it for yourself. ;P)
You use this in the same way as the function, which I didn't even document earlier anyway as I knew I'd just be replacing it with this shell script by the end of the comment ;P.
This will reconnect to the existing session named stuff on server.saurik.com with the username saurik. It will disconnect other clients when it connects: if you don't want that, change -dRS to -xRS. It also disables normal screen control with ctrl-A (so it feels more like "just a console"): if you don't want that, remove -e'\0\0'. If you want some other screen setting: just add it. ;P
(BTW, if you are curious why I have it set to do -dRS "by default" as opposed to -xRS, is that if you have multiple clients associated with the session the size of window can't be automatically updated; the result is that you often find yourself connecting back to copies of things like irssi or even bash prompts that are now the wrong width; -dRS works perfectly out of the box. Many people, however, know enough about screen that they love -x: if that includes you, you should definitely just change that.)
So, if you don't need the "severe packet loss" protection, I really think you are better off just passing on mosh and using something infinitely simpler cobbled over ssh. The latency mitigation feature are overrated or even annoying, and it seems many people actually just want mosh for "easily auto-reconnecting mobile shell", something you seriously can solve very well with a very short shell script and no special server-side logic.
(If anyone tries using that script, it doesn't work, and you want help debugging it, I'm happy to do so; that said, it is purposely not "a product" right now because different people seem to want different things, and it is frankly really simple to write this from scratch if you need it, or just kind of pull it apart and fix/change whatever you don't like about it: just attempting to get an automatically-reconnecting shell is a fairly simple problem.)
If you're willing to sell "just use screen" as an answer for auto-reconnecting, why not "just use screen" with mosh? That's what I do, and I don't have any trouble with point 3 -- I just open a new mosh connection, and reattach my existing screen session.
(Points 1 and 2, for what they're worth, seem pretty specific to your environment. I've never run into them. So, please bill those as anecdotes.)
While I agree with all your points to some extent, and I don't actually like the way mosh has been made, I don't think your conclusion is fair.
When used on slow mobile networks, where I actually keep the same incoming IP (thus there is no reconnection), mosh works better than raw SSH for me and here's why:
- when switching cellphone towers, mosh's UDP catches up instantly. SSH's TCP waits for some retransmissions so that it knows if it's going to timeout or resume sending packets, and it takes _several_ seconds!
- prediction actually works well most of the time.
I've spent the past year using Mosh from high latency, unreliable satellite connections in Afghanistan. I would have been pulling my hair out without it. My only complaint is the scrollback buffer often gets jumbled.
While a nice idea, licensing issues are a problem for this project... I don't understand why they didn't use the standard MIT license, but instead went for GPL3, but still chose a proprietary encryption method.
iSSH on iOS now also experimentally supports it with an own independent implementation, with an in-app purchase, since the encryption uses a proprietary encryption method only freely licensed for opensource projects.
OCB has a clean, short, embeddable reference implemention, and there's a patent grant permitting usage in GPL projects. (Calling it "proprietary" is a stretch, but I understand what you mean.)
GCM didn't have any great implementations. There's starting to be an implementation in OpenSSL 1.0, but at the time mosh was written, that was difficult to count on, and even now that's still shaky.
It's important that the crypto algorithm implementation be known to be correct and safe (avoid side channels, etc.).
As far as I know there's no particular reason that iSSH does not use the standard mosh implementation. One thing to note is that iSSH is not free software (everything it embeds -- PuTTY, Xorg, dwm, etc. -- is under an MIT-style license), so embedding a GPL project would have forced them to open up the iSSH source code.
It would be great if someone wanted to write a free software terminal emulator for iOS, and at that point you can absolutely include mosh. (I'd probably encourage _building_ on mosh, which happens to include a rigorous terminal implementation.) But it's true that you can't embed GPL software inside proprietary software.
I just installed this on my Chromebook with Ubuntu, and on my EC2 instance. I've been very frustrated with the lag I was experiencing while typing in the shell, due to high/variable latency. In my very preliminary testing over the last hour, this has solved my problems almost completely.
Security is less of a concern because I'm running OpenVPN to my instances, so the UDP traffic mosh uses is encrypted anyway, and not open to the public.
One issue I did notice is that irssi isn't refreshing quite as quickly. Seems to lag behind the conversation a bit. I'll have to troubleshoot. I wonder if running byobu/screen is part of the problem.
I also run a similar setup and am trying out mosh now. I agree re: the lag in irssi. Additionally, if I start typing into irssi and then rapidly delete the text, I can delete the channel name temporarily:
After using mosh on and off for a year, I realized that what they wrote in their PDF is absolutely true:
mosh latency isn't much different from SSH latency, which is a common misconception. What mosh adds is:
- decent prediction (this is what mitigate most of the latency)
- integrated auto-reconnect (even thus auto-ssh and others do the same thing, they're not built-in)
- mostly asynchronous keystrokes thanks to UDP+stream protocol, so if you hit ^C it will abort stuff even if the screen is scrolling. Note that the way keystrokes are handled makes it susceptible to a timing attack, and there might have been work to mitigate this that i'm unaware of.
> so if you hit ^C it will abort stuff even if the screen is scrolling
I might be wrong, but AFAIK the fact that your terminal isn't handling this correctly already is due to a flow control setup problem somewhere in the chain (probably the terminal itself, as usually the server-side components are the most ancient): if your local console is only able to download and display and scroll data at a certain rate, it should "push back" on the server to get it to send you data at a slower rate, which in turn will push back on the terminal. When setup correctly, this flows all the way back to the original application, which will itself slow down whatever it is that it is doing.
One thing that's distinctly better is that the backoff/recovery behavior of SSP (mosh's custom encrypted protocol, over UDP) works a lot better than TCP's. If I'm typing, and there's a network blip for e.g. thirty seconds (cell handoff), SSH will literally take minutes to get back to normal, and it'll be easier for me to kill SSH and restart the connection. mosh will recover much more quickly. TCP was written in the days when congestion on a shared Ethernet segment was a thing people actually ran into; with wireless and cell networks, that's not actually your protocol's biggest concern.
The common-case latency, when there's a good network connection, shouldn't differ much.
Most network programs lose their connections after roaming, including SSH and Web apps like Gmail.
Gmail.com has been a roaming-safe/disconnect-tolerant web app for quite some time now. It is even tolerant of huge disparities in time, as far as SSL will allow it to be.
I wonder what Gmail client they're talking about...
I would love to see mosh-like solution, but client only.
Pretty much a wraper to ssh which will use screen plus some predictions. I guess it could get 80% of mosh benefits without the hassle of deploying new software on servers.
Mosh has been brilliant so far. I travel a lot by train but even SSH'ing over the trains WiFi can be problematic. Connecting through Mosh solves all the typical issues and with local echo you can continue to type as if the connection was still up.
I loved Mosh... until I started using it. Latency is definitely a big problem for me - even on fast connections. Often keystrokes arrive in the wrong order if you type too fast, making it useless.
Hi, thanks for your feedback! I have heard of this happening with iSSH's clean-room reimplementation of the protocol, but never with Mosh. Please file a bug report at https://github.com/keithw/mosh because we are eager to trace this down. (We have never heard of this happening to anybody, and it should not be possible if all is well.)
I simply could not work without mosh. I do not experience significant lag and it allows me to have connections open even if I switch from WiFi over to 3G and back which is often the case.
I know this sounds not very secure, but since one of the best features of mosh is that you needn't install it, why not provide a place to download binaries for all platforms?
I was sad because it was disturbed when I try to login my VPS(Burst) from mainland China with Mosh, after I was so happy to find it and installed it on all my Linux OS.
iSSH has an independent proprietary implementation of the mosh protocol as an in-app purchase. (mosh itself has license terms explicitly permitting redistribution on the App Store, but nobody has done the port yet.)
[+] [-] saurik|13 years ago|reply
1) mosh lowers the average latency, but it causes the perceived latency to be highly variable.
The issue is that there are various moments where the algorithm somewhat-correctly decides it cannot predict what will happen due to new incoming keystrokes. An example of this is after you hit enter: you might be running an entirely different program at this point.
In essence, the algorithm "warms up" as it makes correct decisions, and once it decides "ok, my decisions seem accurate: when he types a character they seem to echo back to the terminal" it makes a sudden transition from "slightly slower than raw SSH" to "much faster than SSH and only slightly slower than a local screen session".
These sudden transitions are jarring: when they happen to you with normal connections the expectation is "oh, my connection is failing", a feeling that with mosh you have every time you type a command.
Meanwhile, for the interactive-shell or complex-text-editor use case, these "unpredictable events" always happen at the worst possible moment: when you first start doing something; the result is that the perceived incidence of latency is actually worse than with normal SSH, as every "new event" has a higher latency... by the time it starts prediction, you are probably "in the flow" enough to not even be paying attention.
2) it relies on a specific range of UDP ports
Because it operates over a custom supposedly-mobile-optimized protocol, it tends to end up being inaccessible when you actually need it due to pervasive modern firewalls. You'll start a session at home, go to the office, and... nothing... because you can't communicate over that UDP port. This happened to me quite often while evaluating mosh.
3) it doesn't have a good way to reconnect later
While it supports sitting down at a different computer and taking over your session, it requires you to drop to a rather low-level mechanism based on knowing the secret session key that was established at the time the server was setup. This actually undermines the "mobile" use case, as it only lets the connection be mobile, not the session.
(I also have some specific issues with a library it relies on for spoofing hostnames in wtmp: just having that library installed on your system opens a vulnerability where any non-privileged user can spoof their hostname as anyone else... their opinion differs, but it seems to mostly be due to a knee-jerk reaction of "you don't get fired for avoiding setuid" than "this is actually secure".)
The result is that after spending a bunch of time testing it, working with the codebase, getting frustrated at a bunch of the design decisions, and then having it both totally fail to work in real-world network settings and become yet another source of infuriating latency, I pretty much dropped all interest in the project and replaced it with autossh, which worked great.
The primary downside of this is that it doesn't handle the packet loss problem well, as you are really just still using normal SSH to do the connections. The session might get weirdly slow and even lock up in these packet-loss situations.However: it doesn't require any special code on the server, which is a downside of mosh I didn't even list above (as it is fairly obvious and something you know going in): this means you get to use it for every connection to every server you might ever care about.
Otherwise, you get everything you expect from normal ssh and screen: it works everywhere you might theoretically have been able to connect to the server anyway (so no ending up at the client site and finding out that they firewall UDP), it is obvious how to reconnect to (or even share) the screen session from a different computer or a different client without special software, and you don't need to even contemplate "what if this introduces a security vulnerability by running a new daemon on my server?".
The other annoyance of using autossh is it is pretty adamant about doing exponential back-off on reconnect, and in the naive way I have it setup in that function it is also going to be doing slow-to-fail DNS requests as it attempts to do the reconnection sequence, so the result is that the "open laptop, wait for all terminals to connect" sequence doesn't always click perfectly.
However, that is easily fixable by using a simpler wrapper (to be clear: use autossh when autossh is appropriate; it is designed, however, for server-to-server backend links to maintain persistent port forwarding), so I bring you cysh.sh, which is what I actually use (and which I've done some testing with others to verify works on random platforms):
http://test.saurik.com/hackernews/cysh.sh
(Yes: you are now downloading a random shell script from someone on a web forum. However, it's like 35 lines of very easily readable sh: you should just read it; hell, after reading it, you might consider it so simple you just want to rewrite it for yourself. ;P)
You use this in the same way as the function, which I didn't even document earlier anyway as I knew I'd just be replacing it with this shell script by the end of the comment ;P.
This will reconnect to the existing session named stuff on server.saurik.com with the username saurik. It will disconnect other clients when it connects: if you don't want that, change -dRS to -xRS. It also disables normal screen control with ctrl-A (so it feels more like "just a console"): if you don't want that, remove -e'\0\0'. If you want some other screen setting: just add it. ;P(BTW, if you are curious why I have it set to do -dRS "by default" as opposed to -xRS, is that if you have multiple clients associated with the session the size of window can't be automatically updated; the result is that you often find yourself connecting back to copies of things like irssi or even bash prompts that are now the wrong width; -dRS works perfectly out of the box. Many people, however, know enough about screen that they love -x: if that includes you, you should definitely just change that.)
So, if you don't need the "severe packet loss" protection, I really think you are better off just passing on mosh and using something infinitely simpler cobbled over ssh. The latency mitigation feature are overrated or even annoying, and it seems many people actually just want mosh for "easily auto-reconnecting mobile shell", something you seriously can solve very well with a very short shell script and no special server-side logic.
(If anyone tries using that script, it doesn't work, and you want help debugging it, I'm happy to do so; that said, it is purposely not "a product" right now because different people seem to want different things, and it is frankly really simple to write this from scratch if you need it, or just kind of pull it apart and fix/change whatever you don't like about it: just attempting to get an automatically-reconnecting shell is a fairly simple problem.)
[+] [-] geofft|13 years ago|reply
(Points 1 and 2, for what they're worth, seem pretty specific to your environment. I've never run into them. So, please bill those as anecdotes.)
[+] [-] zobzu|13 years ago|reply
When used on slow mobile networks, where I actually keep the same incoming IP (thus there is no reconnection), mosh works better than raw SSH for me and here's why:
- when switching cellphone towers, mosh's UDP catches up instantly. SSH's TCP waits for some retransmissions so that it knows if it's going to timeout or resume sending packets, and it takes _several_ seconds!
- prediction actually works well most of the time.
[+] [-] moe|13 years ago|reply
[+] [-] jread|13 years ago|reply
[+] [-] rshm|13 years ago|reply
[+] [-] koffiezet|13 years ago|reply
iSSH on iOS now also experimentally supports it with an own independent implementation, with an in-app purchase, since the encryption uses a proprietary encryption method only freely licensed for opensource projects.
[+] [-] geofft|13 years ago|reply
GCM didn't have any great implementations. There's starting to be an implementation in OpenSSL 1.0, but at the time mosh was written, that was difficult to count on, and even now that's still shaky.
It's important that the crypto algorithm implementation be known to be correct and safe (avoid side channels, etc.).
As far as I know there's no particular reason that iSSH does not use the standard mosh implementation. One thing to note is that iSSH is not free software (everything it embeds -- PuTTY, Xorg, dwm, etc. -- is under an MIT-style license), so embedding a GPL project would have forced them to open up the iSSH source code.
It would be great if someone wanted to write a free software terminal emulator for iOS, and at that point you can absolutely include mosh. (I'd probably encourage _building_ on mosh, which happens to include a rigorous terminal implementation.) But it's true that you can't embed GPL software inside proprietary software.
[+] [-] eli|13 years ago|reply
[+] [-] HorizonXP|13 years ago|reply
Security is less of a concern because I'm running OpenVPN to my instances, so the UDP traffic mosh uses is encrypted anyway, and not open to the public.
One issue I did notice is that irssi isn't refreshing quite as quickly. Seems to lag behind the conversation a bit. I'll have to troubleshoot. I wonder if running byobu/screen is part of the problem.
[+] [-] carbocation|13 years ago|reply
[+] [-] zobzu|13 years ago|reply
mosh latency isn't much different from SSH latency, which is a common misconception. What mosh adds is:
- decent prediction (this is what mitigate most of the latency)
- integrated auto-reconnect (even thus auto-ssh and others do the same thing, they're not built-in)
- mostly asynchronous keystrokes thanks to UDP+stream protocol, so if you hit ^C it will abort stuff even if the screen is scrolling. Note that the way keystrokes are handled makes it susceptible to a timing attack, and there might have been work to mitigate this that i'm unaware of.
[+] [-] saurik|13 years ago|reply
I might be wrong, but AFAIK the fact that your terminal isn't handling this correctly already is due to a flow control setup problem somewhere in the chain (probably the terminal itself, as usually the server-side components are the most ancient): if your local console is only able to download and display and scroll data at a certain rate, it should "push back" on the server to get it to send you data at a slower rate, which in turn will push back on the terminal. When setup correctly, this flows all the way back to the original application, which will itself slow down whatever it is that it is doing.
[+] [-] geofft|13 years ago|reply
The common-case latency, when there's a good network connection, shouldn't differ much.
[+] [-] doctorpangloss|13 years ago|reply
Gmail.com has been a roaming-safe/disconnect-tolerant web app for quite some time now. It is even tolerant of huge disparities in time, as far as SSL will allow it to be.
I wonder what Gmail client they're talking about...
[+] [-] lwf|13 years ago|reply
[+] [-] niggler|13 years ago|reply
[+] [-] kami8845|13 years ago|reply
[+] [-] nicoritschel|13 years ago|reply
iSSH includes the mosh protocol as a 99c in-app purchase. Use mosh on your mobile device, just where it should be!
[+] [-] jakozaur|13 years ago|reply
Pretty much a wraper to ssh which will use screen plus some predictions. I guess it could get 80% of mosh benefits without the hassle of deploying new software on servers.
[+] [-] Karunamon|13 years ago|reply
[+] [-] daenney|13 years ago|reply
[+] [-] kolev|13 years ago|reply
[+] [-] keithwinstein|13 years ago|reply
[+] [-] DoubleMalt|13 years ago|reply
[+] [-] wangweij|13 years ago|reply
[+] [-] jiyinyiyong|13 years ago|reply
[+] [-] visarga|13 years ago|reply
[+] [-] geofft|13 years ago|reply
[+] [-] misleading_name|13 years ago|reply
[+] [-] rckrd|13 years ago|reply
Can anyone briefly rehash some of the security issues that were brought up with mosh?
[+] [-] geofft|13 years ago|reply
[+] [-] leh0n|13 years ago|reply