top | item 42535217

Fish 4.0: The Fish of Theseus

906 points| jdxcode | 1 year ago |fishshell.com | reply

198 comments

order
[+] chubot|1 year ago|reply
Congrats to the fish team! Great writeup with lots of interesting detail.

I wonder if this is the biggest project that has moved from C++ entirely to Rust (or maybe even C to Rust?) It probably has useful lessons for other projects.

If I'm reading this right, it looks like fish was not released as a hybrid C++ / Rust program, with the autocxx-generated bindings. There was a release during that time, but it says "fish 3.7 remains a C++ program" [1]

It sounds like they could have released if they wanted to, but there was a last stage of testing that didn't happen until the end.

Some people didn't quite get the motivation for adding C++ features to Rust [2], and vice versa, to enable inter-op. But perhaps this is a good case study.

It would be nice if you could just write new Rust code in a C++ codebase, without writing/generating bindings, and then throwing them away, which is mentioned in this post.

---

Also the #1 gripe with Rust seems to be that it supports version detection, not feature detection.

But feature detection is better for distros, web browsers, and compilers:

Feature Detection Is Better than Version Detection - https://github.com/oils-for-unix/oils/wiki/Feature-Detection...

Version/name detection is why Chrome and IE pretend to be Mozilla, and why Clang pretends to be GCC. Feature detection (e.g. ./configure and eval() ) doesn't cause this problem!

[1] https://github.com/fish-shell/fish-shell/releases

[2] e.g. https://news.ycombinator.com/from?site=safecpp.org

[+] ComputerGuru|1 year ago|reply
To clarify, work on the rust rewrite started after 3.7.0, but the C++ code remained in a working branch on the git repo. Midway through the rewrite, we backported additions and improvements to fish scripts (most observable being new and improved completions) and a couple of important bugfixes from the rust-containing `master` branch to the C++ branch and released that as 3.7.1.

We never considered releasing anything with a hybrid codebase; aside from the philosophical purity of fully making the switch to rust, it would have been a complete distribution nightmare (we take package maintainer requirements very seriously). Moreover, the code itself was not in a very pretty state - the port was very much like trying to undo a knot: you had to make it much uglier in order to get it properly undone. There were proverbial tons of SLoC that were introduced only for transitional interop purposes that were later removed, this code was never held to the same quality standards (in terms of maintainability; it was still intended to be bug-free and required to pass all our unit and integration tests, however).

As mentioned in the article, we prefer to do feature detection when and where needed/possible. The old codebase was purely feature-detected via the CMake build system but we ended up writing our own feature detection crate for rust invoked via build.rs (maintained here [0]) though we just defer to libc on a lot (which doesn't do that yet). One side effect of the libc issue is that we're beholden to their minimum supported targets (though I'm not sure if that's strictly the case if we don't use the specific apis that cause that restriction?), which are higher than what we would have liked because we were fine with feature detecting and implementing using both older and newer apis where needed.

[0]: https://github.com/mqudsi/rsconf

[+] boris|1 year ago|reply
> Feature Detection Is Better than Version Detection

The problem with feature detection (normally referred to as configuration probing), at least the way it's done in ./configure and similar, is that it relies on compiling and potentially linking (and sometimes even running, which doesn't work when cross-compiling) of a test program and then assuming that if compilation/linking fails, then the feature is not available.

But the compilation/linking can fail for a myriad of other reasons: misconfigured toolchain, bug in test, etc. For example, there were a bunch of recent threads on this website where both GCC and Clang stopped accepting certain invalid C constructs which in turn broke a bunch of ./configure tests. And "broke" doesn't mean you get an error, it means your build now thinks the latest Fedora and Ubuntu all of a sudden don't have strlen().

[+] Conscat|1 year ago|reply
Although Clang does set the `__GNUC__` macro and you have to distinguish it using the `__clang__` macro, Clang and GCC also both have very fine-grained feature detection features as well, both at the CLI level and in the preprocessor (such as the `__has_feature` family of builtins).
[+] scop|1 year ago|reply
I remember switching from bash to zsh a few years back and thinking I was the bees knees. After the switch trying other shells seemed like bike-shedding because, I mean, what more could a shell? Then I got a new computer and decided to start from scratch with my tooling and downloaded fish. I was shocked how it instantly made zsh feel cumbersome and ancient.

Heartily recommend others give it a try as a daily driver for a couple of weeks. I liken it to Sublime Text: an excellent “out of the box” tool. Just the right amount of features, with the option to add more if you want. But you also don’t feel like your missing out if you keep it bare bones. A great tool in and of itself.

[+] kstrauser|1 year ago|reply
Same here. I used it for about 3 days before I installed it on all my systems and permanently switched. For me, it was like the first time I learned a non-Latin language, and my eyes were opened to how much stuff I took for granted was completely arbitrary.

For example, here's how you write an autoloaded function "foo" in Fish: you make a file called "foo.fish" in its config directory. Inside that, you write "function foo ..." to implement it. There's no step 3. That's it.

Want to customize your shell prompt? Follow the process above to write a function called "fish_prompt" that uses normal scripting things like echo, pwd, git, or whatever to write your prompt to the screen. There's no step 2. That's it.

Fish was revelatory. Other shells of the same vintage feel hopelessly outdated to me now. For context, I was the maintainer of FreeBSD's "bash-completion" port for a few years way back when. It's not that I don't have experience with other shells. I have plenty. I just don't want to use any of the others now.

[+] pzmarzly|1 year ago|reply
Interesting, I went the other way about 7 years ago - switched from fish to zsh (initially with oh-my-zsh). The interactive experience was similar enough on both shells, and the performance was great on fish and okay-ish on zsh, but two things won me over:

1. With zsh, I can copy-paste some bash snippet and in 99% of cases it will just work. Aside of copy-pasting from StackExchange, I also know a lot of bash syntax by heart by now, and can write some clever one-liners. With zsh, I didn't need to learn everything from scratch. (I guess this matters less now that you can ask AI to convert a bash one-liner into fish one-liner?)

2. For standalone scripts... well, I think it's best to reach for a proper programming language (e.g. Python) instead of any shell language, but if I had to use one, I would pick bash. Sure, it has many footguns, but I know them pretty well. And fish language is also not ideal - e.g. IIRC it doesn't have an equivalent of `set -e`, you have to add `; or return 1` to each line.

[+] thayne|1 year ago|reply
I went bash -> fish -> zsh.

The main reason I switched is because zsh can (often) source bash scripts and can use bash completion scripts (usually), and I was tired of having to translate things from bash to fish. I also ran into a few things where something that was relatively easy to do in bash was impossible to do with fish. But that was years ago so maybe that is less of an issue now, and I don't remember exactly what it was.

Having used zsh, I think a big advantage it has over fish is the completions. There are completions available for more programs for zsh, and the zsh completions are sometimes higher quality in zsh.

But I do generally like the syntax, and good out of the box experience of fish. I wish it had a bash or even posix compatibility mode and more available completions.

[+] BeetleB|1 year ago|reply
Also, do consider xonsh.[1]

It's a Bash-like shell written in Python. It has significant overlap with the awesomeness of fish, and has the advantage of being able to write your shell scripts in a Python dialect. So if you know Python, the mental burden is much lower.

On top of that, it's cross platform, since Python is. No WSL needed.

I switched to it in 2018 and haven't looked back. Originally it was just because I wanted a better command prompt environment in Windows for work, but I liked it so much I switched to it in Linux as well.

(And yes, you can type any Python statement right in the command prompt).

[1] https://xon.sh/

[+] junek|1 year ago|reply
I know it's a typo but this:

> what more could a shell?

Is quite good. It could almost be the tag line for fish shell.

[+] bravura|1 year ago|reply
Do you mind sharing what you think are the killer features of fish?
[+] ramon156|1 year ago|reply
We had the exact same experience, still in love with fish!
[+] giancarlostoro|1 year ago|reply
My only issue with Fish is when pasting things from the web that assume Bash, a lot of the time it just works, then now and then I get screwed. I don't know nearly enough Fish or Bash to switch. Still though, I prefer Fish ultimately.
[+] freedomben|1 year ago|reply
> The one platform we care about a bit that it does not currently seem to have enough support for is Cygwin, which is sad, but we have to make a cut somewhere.

> We’re also losing Cygwin as a supported platform for the time being, because there is no Rust target for Cygwin and so no way to build binaries targeting it. We hope that this situation changes in future, but we had also hoped it would improve during the almost two years of the port. For now, the only way to run fish on Windows is to use WSL.

I understand, but this is indeed incredibly sad. To this day I still use Cygwin, and in fact prefer it to WSL depending on what I'm doing. Cygwin is an incredible project that is borderline miraculous for what it accomplished and provides. Without Cygwin I may not have any sanity left. I can't exude enough love for the Cygwin team.

Hopefully rust will support cygwin as a build target in the future!

[+] _zagj|1 year ago|reply
It's strange how the article starts off complaining about C++'s platform "issues":

> We’ve experienced some pain with C++. In short:

> tools and compiler/platform differences

before conceding that, because of Rust, they 1) are actually dropping support for a platform they previously supported and 2) can only support (in theory) a small fraction of those platforms supported by g++, but that that's OK because those are the only platforms which really matter. I get that it's a trade-off, but it would have been more intellectually honest to just admit this is one area (portability, backwards compatibility, and ABI stability) where C++ mops the floor with Rust, instead of pretending it's a another paintpoint Rust avoids.

[+] bloppe|1 year ago|reply
genuinely curious: with so much love for cygwin, why not just run Linux? possibly with a dual boot?
[+] epage|1 year ago|reply
> The one goal of the port we did not succeed in was removing CMake.

> That’s because, while cargo is great at building things, it is very simplistic at installing them. Cargo wants everything in a few neat binaries, and that isn’t our use case. Fish has about 1200 .fish scripts (961 completions, 217 associated functions), as well as about 130 pages of documentation (as html and man pages), and the web-config tool and the man page generator (both written in python).

Our issue for this is https://github.com/rust-lang/cargo/issues/2729

Personally, I lean away from Cargo expanding into these use cases and prefer another tool being implemented on top. I've written more about this at https://epage.github.io/blog/2023/08/are-we-gui-build-yet/

[+] sunshowers|1 year ago|reply
(hi Ed!)

I would definitely love to see Cargo have the ability to do this -- it means that `cargo install --locked` stays as a viable approach. It probably won't apply to fish, but I think being able to run a post-install command from the binary you just installed would suffice for my needs.

[+] qalmakka|1 year ago|reply
As a decade-long user and as a professional C++ developer, I'm so happy they've managed to successfully port the shell to Rust. While I have a lot of fun writing C++ (and Rust), I must admit that Rust is vastly nicer to use.

People can complain as much as they want about the borrow checker, but you basically have to be as strict as Rust is in C++ if you want to really avoid use-after-free issues, ... I've been writing "Rusty C++" since before Rust was a thing, because that's the only sane approach to memory safety. I'd rather have a program check that I don't fumble up instead of running sanitizers when things go awry (often years later). The best bug is a bug that can't happen at all.

Static analyzers are sadly too limited compared to what a borrow checker can do in my experience. Some bad stuff will always slip in in C/C++.

[+] bad_username|1 year ago|reply
What is you "rusty C++"? The only thing I can think of is strict adherence to RAII.
[+] Too|1 year ago|reply
Surprised to see the line count go up so much, 56K LOC of C++ to 75K of Rust. The blog attributes it to rustfmt using less oneliners. Even so, i would believe that should be a small factor compared to the heaps of duplicate code you get from c++ header files and all the other syntax ergonomics rust gives you.

Is this typical for such a translation. They also mention addition of new features contributing to more code, how much of the addition was new features vs pure translation?

Would be interesting to see the line count of the c++ version if it was run through a formater with similar configuration.

[+] pornel|1 year ago|reply
Rust is denser than C, but both Rust and C++ can work on similarly high level of abstraction.

It may be just down to rustfmt. It really adds a lot of vertical sprawl. I personally can't stand how much rustfmt makes multi-line code explode.

[+] underdeserver|1 year ago|reply
The tone in the "The Timeline" section seems apologetic:

> The initial PR had a timeline of “handwaving, half a year”. It was clear to all of us that it might very well be entirely off, and we’re not disappointed that it was.

I'm amazed that you estimated it at so little time originally, and I'm amazed you shipped it in full in just 2 years. Congrats!

[+] runiq|1 year ago|reply
Absolutely. Staying within an order of magnitude for a project of this size is just really good eyeballing. :)
[+] nasretdinov|1 year ago|reply
Very nice too see Rust being used where it is actually appropriate! Hopefully Rust "easy" multi-threading will allow more parts of fish to be async, even though it's already much better in that regard than bash (or any other shell I've seen).

One weird thing I'd also like to see is more bash integration, as others pointed out that being their primary motivation against switching to fish full-time. My use case is mostly sourcing bashrc/bashevv, and theoretically it should be possible in fish if I understand correctly: you need to be able to import e.g. every new env variable that changed before and after sourcing a bash script via real bash.

[+] gorgoiler|1 year ago|reply
I try not to post unsubstantive comments here but I’m just so moved by this success that I have to say an enormous Congratulations!
[+] WD-42|1 year ago|reply
Congrats to the Fish team. The best shell just got better.

How about updating the project tagline to: "Finally, a shell for the 00s!"

[+] ComputerGuru|1 year ago|reply
Thanks but one cannot be too ambitious like that! '00s would mean the end of zip drives, dealing with unstandardized flash drives flakier than the floppy disks of old, and supporting point-and-shoot digital cameras!
[+] OptionOfT|1 year ago|reply
> it is often better to use if cfg!(...) instead of #[cfg(...)] because code behind the latter is eliminated very early

My experience with this is the other way around, especially if you have crates tied to that feature.

The cfg! is a marco that compiles to true/false, so whatever is inside of the if guard needs to compile regardless.

E.g.:

Cargo.toml

    [features]
    default = []
    my_feature = ["deps:feature_dependency"]

    [dependencies]
    feature_dependency = "1.0.0"
And in code:

    if cfg!(feature = "my_feature") {
        feature_dependency::something::Something::invoke();
    }
This will fail if you compile without `my_feature`.
[+] CGamesPlay|1 year ago|reply
That was the point. The paragraph is talking about how errors only show up in some configurations, leading to “works for me” behavior for some of the devs. When you can get away with cfg!, you are more confident that it will at least compile regardless of the config being checked.
[+] dajonker|1 year ago|reply
I might be wrong but most optimizing compilers will treat "if false" and the following code as dead and remove it.
[+] outlore|1 year ago|reply
I am curious to ask others here, are there other low-config alternative tools like Fish that, looking back, now seem like a no brainer? Ghostty is a recent example, Helix seems like another. I’d love to know about other tools people are using that have improved or simplified their lives.
[+] memco|1 year ago|reply
Thought for a second that this was a 4.0 release announcement but this is just about the rewrite in rust. Any fish users wanting release notes of what to look forward to can look here: https://fishshell.com/docs/4.0b1/relnotes.html. Glad the rewrite is helping the dev team make improvements, but I’m more excited for the actual new features (except the new alt-backspace behavior which I’m sure I’ll get used to).
[+] petepete|1 year ago|reply
Really happy to see this, such a mammoth effort by the team and everyone else involved.

I switched over from zsh about four years ago and my config went from several hundred lines to a handful with just one plugin (fzf.fish).

It just works how I expect it to and I can't imagine changing again any time soon.

[+] Spoof7726|1 year ago|reply
> Fish also uses threads for its award-winning (note to editor: find an actual award) autosuggestions and syntax highlighting, and one long-term project is to add concurrency to the language.

(note to editor: find an actual award)

[+] syhol|1 year ago|reply
The two most popular zsh plugins are total clones of this at 31k and 20k gh stars respectively. Not an award but certainly an indication of its success.
[+] dwattttt|1 year ago|reply
Seriously, can someone find them an award? I think they've earned it.
[+] akdor1154|1 year ago|reply
I'd be really interested to hear from distro packagers how this is going - how amenable is rust-fish to being packaged following e.g. Debian guidelines?
[+] ComputerGuru|1 year ago|reply
We took an incredible amount of care to consider the package maintainer requirements for our the most popular distributions using/distributing fish. One of our maintainers is very careful about letting us know when we're doing something that might upset distro packagers, and we're constantly letting package maintainer guidelines and requirements influence how we structure fish itself and which dependencies we pull in.
[+] zanchey|1 year ago|reply
It's hopefully not too tricky - it can't be packaged as a crate using (say) debcargo, as the install path still requires CMake. The Debian experimental package changes are mostly about pulling in the right dependencies (including some internal mangling to support some policy choices).
[+] abbefaria27|1 year ago|reply
Amazing write up! Everyone at work is itching to try Rust, but I think what’s killing adoption is that it’s not very clear how to gradually transition a code base. We have a few million lines of C++, some of it written 25 years ago. A full rewrite is just out of the question, at best we could use it for new sections. This is super common in the c++ world, so it’s a pity that porting wasn’t a first class concern in rust considering C++ devs are the target audience. It sounds like it was a challenge even at 57k LOC. Congrats to the fish team though, great accomplishment!
[+] jpc0|1 year ago|reply
If you codebase isn't somewhat modern C++(C++11) I would start there before concidering a port to rust. It will be a significantly easier upgrade in safety even if not going all the way to rusts level of safety.

Generally code that has been running for years is unlikely to have too many bugs since they have been shaken out, "rewrite it in Rust" as a fad just ignores the decades of work already put into the codebase and for large codebases likely eont succeed.

As you mentioned, write new modules with rust. That means likely needing to export a C API for your libraries but there's a good chance you were already doing that. There was also a rust crate that tried to automate most of the c++ rust interop for you but not sure about how good it is in reality.

[+] LAC-Tech|1 year ago|reply
I use the shell a lot every day, mainly bash and some ash (alpine).

Does something like fish make the experience a bit smoother? is it pretty easy to get into?

[+] naurupatel|1 year ago|reply
We're flush with new and awesome terminals lately, Ghostty public launch now a huge upgrade to fish.

I've tried Fish a few times but hard to migrate over from bash/zsh. Does anyone have tips on how to port over a bunch of aliases/scripts/etc. easily?

[+] andrewshadura|1 year ago|reply
You don't need to port your scripts. Migrating aliases shouldn't be too difficult.