top | item 19475218

On Learning Rust and Go: Migrating Away from Python

240 points| edward | 7 years ago |blog.liw.fi

337 comments

order
[+] jacquesm|7 years ago|reply
15Kloc and trouble is indicative of other problems than a problem with the language. I wrote multiple 50Kloc and up pieces of software in GFA Basic, arguably a much more limiting and unsafe environment than Python ever was, and yet, that software worked well and was maintainable to the point that its descendants still run 3 decades later.

Python has all the bells and whistles you need to build large code bases, but you do need to get organized, no language can really do that for you and attempts at forcing you to do it give rise to 'factory factory' nonsense. I'm no huge fan of Python, I use it when I have to but Rust is from a bells-and-whistles point of view a step down from Python, it is more a better 'C' than a better Python and Go has it's own share of issues that has kept me away from it so far.

If you want to crank out some research project or application quickly building on the libraries that others provide (especially in machine learning or other data intensive fields) then Python would be my first choice, and something like 'R' probably my second (though I see 'R' mostly as a data scientist's workbench that has a programming language bolted on).

[+] gameswithgo|7 years ago|reply
Most of my serious work has been done with statically typed languages, though of course I have worked with dynamically typed ones as well. There are a few things about dynamically typed languages that seem to productivity-negative, but it may just be my lack of understanding of how people expert at a dynamic languages work with them.

1 - When working with a large enough code base that you don't remember exactly what every function you have written needs, or using third party libraries, function signatures lacking type specifications make it harder to know what is expected. if there is documentation that is good, you can of course load that up, but this slows you down. IDEs today can usually show you the function signature as you type it, but with dynamic types you get less information.

2 - When refactoring something in a statically typed language, changing a data structure or type name will cause the compiler to error every single place that you need to adjust for your change. With a dynamic language you need to rely on unit tests or you will be finding places you missed during runtime for a long time afterwards. If you do rely on unit tests for this kind of thing, then you have sort of built an effectively statically typed environment, at some large degree of effort, with worse runtime performance to show for it.

I understand some of the workarounds people use to deal with these problem but a lot of them seem to push you to the point that you might as well use static typing and get better runtime performance for it.

[+] littlestymaar|7 years ago|reply
I've worked on a 100kloc statically-typed project yet I can definitely see how static typing helps even with a much smaller code-base: I can't write a thousand line of JavaScript without encountering at least one “undefined is not a function” or equivalent error. Call me a bad programmer if you want, but I do find static typing helpful (and Rust's one is really good compared to say Java, C# or Go).
[+] pytester|7 years ago|reply
From glancing at the obnam code it looks relatively okay: https://github.com/lukipuki/obnam/tree/master/obnamlib

The one thing that stands out to me is that it's heavy on the unit tests and light on integration tests which would make an application like this (at least for me) more difficult to handle. I'd go heavier on integration testing and lighter on unit testing.

[+] IshKebab|7 years ago|reply
I disagree. 15k lines of code is a lot of code to keep entirely in your head all at once, which is basically what you have to do if you're using a language as dynamic as Python. In static languages like Java you can easily use tools to check types are correct, find usages of variables, jump to definitions and so on. And you get compile time errors if you screw up, rather than runtime errors.
[+] computerex|7 years ago|reply
> Python has all the bells and whistles you need to build large code bases, but you do need to get organized, no language can really do that for you and attempts at forcing you to do it give rise to 'factory factory' nonsense

This is almost satirical.

> Assembler has all the bells and whistles you need to build large code bases, but you do need to get organized, no language can really do that for you and attempts at forcing you to do it give rise to 'factory factory' nonsense.

I can definitely see someone replacing python with any X language.

Just because you can do something, doesn't mean you should. I personally like opinionated languages that have made some decisions for you based on best practices and research.

[+] lordnacho|7 years ago|reply
I'm going to have a go with Rust and Go too, but for different reasons.

IMO Python is still the best language to go with when you are writing anything "scripty":

- Stuff that's not expected to be long. Do one thing well, let some other program do another, compose the higher order functions from functions that work. Of course this isn't always possible.

- Stuff where the types are not going to confuse you. Often you just have a few types, and it's fairly obvious what you can do with a given object. If you get too many types, there's a good chance it will start to get confusing and then strong typing would help.

- Stuff where performance is not a problem. Don't write your HFT system in Python. You simulation though, might benefit from orchestration of some VMs.

The ultimate use case for Python is glue code. And guess what, a lot of code is just glue. A good glue language has a large ecosystem of libs, and some language features that make it easy to read. That's Python in a nutshell: whitespace formatting makes things easy to read, and you get built in list/dict syntax to keep everything short and sweet.

For me Rust and Go are interesting having come from C++. There's a lot of pitfalls in C++ that might be helped with Rust. At the moment I have a whole load of Valgrind suite tests that look for potential problems. Perhaps Rust can bake those into the language? I haven't had time to look at it yet.

Go seems to be interesting as a compromise between easy and fast, and by the sound of it things have turned out well. I have people on my team using it, and it seems to be the thing to do for web services that need to be faster than nodejs but you don't want to spend a load of time making sure a c++ version is correct.

[+] Annatar|7 years ago|reply
There is one and only one language that's both correct and the best tool for scripting: Bourne or its superset the Korn shell (not even bash).
[+] toyg|7 years ago|reply
> Note that I've not written any significant code in either language, so I'm just writing based on what I've learnt by reading.

So he’s basically comparing 20+ years of Python usage with marketing material from Go and Rust. Right.

I wish people could be honest with their motivations, instead of making up excuses to justify this sort of change. Here it’s a classic case of “I got bored and I don’t like the new features, want new shiny”, wrapped in a bit of whining about problems that have well-known solutions (use a good IDE, use type hints or one of the many libs that enforce types...). I have no problem with that, just don’t try to sell it to me on technical grounds, because it looks very much like you have none.

Python is still one of the best ways to get stuff done, quickly and with a minimum of boilerplate. Its limitations are well-known and have not really increased for about 15 years - if anything they have shrunk in many, many areas. Adoption rates keep exploding, despite a lot of doom forecasted by some alpha geeks about 10 years ago: “There is no mobile story! We will all be using Ruby/Go/NodeJS tomorrow! Nobody will ever go to Py3!” Etc etc etc.

You got bored by it? That’s fine, it happens, no harm done. But you don’t need to convince anyone, including yourself, that you’re moving on because of this or that flimsy excuse.

[+] AsyncAwait|7 years ago|reply
> I wish people could be honest with their motivations, instead of making up excuses to justify this sort of change

Why do you assume motivations other than what the author claims? I very much started on Python, but as my career progressed, I got tired of being frustrated with poor library documentation and having no static types, making it extremely difficult to figure out even the parameter types to pass in, especially after it's been a while.

After switching to static typing, things have been a lot less brittle and even faster to develop. It is often said that Python is fast to develop in, but this is really only true for prototyping. Making a program and making a robust python program are entirely different, whereas they're much less separate in a statically typed language.

Also, statically typed does not imply FactoryFactory or no type inference, especially for languages like Rust, Swift and to a certain extent Go as well. The code written in Rust, Swift etc. could often be as elegant & short as one in Python, without loosing the benefits of static typing.

[+] computerex|7 years ago|reply
Wow. It's amazing the lengths people go to to maintain their delusions. Show me one python IDE in action doing effective refactoring across a large, nontrivial python code base. And you didn't even mention concurrency and skipped over it altogether!

> use type hints or one of the many libs that enforce types...

As if that's pythonic. Heard of duck typing? And those are things you have to do yourself, as additional work and maintenance rather than getting it free because it's a feature of the language. And you don't always have full control over the stack.

Is there an IDE that will guarantee that no runtime errors will take place due to typing?

> Adoption rates keep exploding

Because python made its way into the rapidly growing machine learning market. Python is great for writing glue code/scripts because it's easy to extend with C. Everyone and their grandma getting into data science use python.

> I have no problem with that, just don’t try to sell it to me on technical grounds, because it looks very much like you have none.

Looks like you are just not willing to accept or even recognize any valid criticism of Python. This type of fanaticism towards the language is not the first time I am seeing it.

[+] pagefault123|7 years ago|reply
Perhaps there should be apostasy punishments for Python defectors. /s

Python's typing story is far from perfect and rather annoying compared to other languages. Python is good for interacting with the operating system, networking and other things.

But wanting a proper compiler is an honest motivation.

[+] acqq|7 years ago|reply
See the details here about his project that he maintained in Python for years (since 2006) in Python:

https://blog.liw.fi/posts/2017/08/13/retiring_obnam/

I can really understand his:

"Obnam has not turned out well, from a maintainability point of view. It seems that every time I try to fix something, I break something else. Usually what breaks is speed or memory use: Obnam gets slower or starts using even more memory."

Also from his current post:

"I could perhaps have been more diligent in how I used Python, and more careful in how I structured my code, but that's my point: a language like Python requires so much self-discipline that at some point it gets too much."

[+] jjoonathan|7 years ago|reply
You don't need to convince anyone, including yourself, that python's well-known limitations couldn't possibly have inspired this guy to look at other languages.
[+] atoav|7 years ago|reply
I was a year long Python user when I stared learning Rust. Reason: I knew just enough C/C++ that I knew I wouldn’t be able to pull through learning it properly from a perspective of time. But I wanted a strongly typed fast alternative to Python without garbage collection.

I had read some stuff about Rust and it seemed like a interesting language, mature enough for my applications. I tried it out and came to like it more than I would’ve expected. In fact I tend to use it over python for many things nowadays. One of these things is definitly how the module system works, but I agree with you: if you fuck up in Python with your code structure, the problem is definitly between chair and desk.

However: Rust won’t let you get away for cheap if you don’t think about structure, and this could certainly help some to adopt better practise (which in the end is good for all of us).

Interestingly enough my appreciation for and interest in languages like C and C++ grew a lot, after I had learned Rust. It gave me a good new perspective on C++ and certainly learnd me ton of good patterns and concepts that will definitly end up beeing useful in other languages. Rust certainly is well thought out and good ideas are never bad to look at.

[+] 0815test|7 years ago|reply
> Here it’s a classic case of “I got bored and I don’t like the new features, want new shiny”, wrapped in a bit of whining

This seems to be a rather shallow reading of the blogpost, and the way you phrased your reaction is not exactly polite either. If anything, the "adoption rates" of Python (like those of ECMAscript, a language of a similar vintage all-things-considered) are most easily explained by non-technical factors, so I have quite a bit of trouble making sense of your comment.

[+] Waterluvian|7 years ago|reply
Wanting new shiny is a helpful healthy way to keep exploring and growing.

I've worked with the opposite who absolutely refuse to ever learn python/go/whatever, producing these awful thousand line bash scripts.

[+] empath75|7 years ago|reply
Type checking as an add-on is okay but it is not the same thing as a statically typed language.
[+] mykowebhn|7 years ago|reply
I feel exactly the same way with all the Rubyists moving to Elixir.
[+] buro9|7 years ago|reply
I'm currently migrating a similarly sized Python and Django application to Go but for different reasons.

For me it's bit rot. My Python is approaching EOL on this version and needs upgrading to Python 3, and I have a lot of 3rd party code in here that made things simpler to create but over time they've changed enough or disappeared that I now have to take on work there too. Then Django and it's internals have changed a lot in the last 6 years.

My Go code written at the same time (6+ years ago) hasn't had any of it's libraries change so much that I've needed to do anything. Every now and then I pull the next major version of Go and recompile and it's done.

This is all side project stuff but a lot of people use it (monthly average visitors being +250k)... it's non-profit and I don't wish to have to maintain it much.

It worked once, if nothing has changed in the functionality it should keep working... but this isn't as true in non-compiled languages where you need to depend on so much of the environment, let alone build culture that doesn't fully vendor dependencies.

I like Python, but prefer to write code and move forward with other projects.

As to Rust vs Go. I code in both and only picked Go here because a lot of the code in another project I worked on can be used here. I'd probably try Rust if I was starting from scratch now. But I'd be a little concerned bit rot would affect Rust too, it's still quite early for the wider ecosystem.

[+] bayesian_horse|7 years ago|reply
I'd chose Python 3 migration and updating/removing/exchanging dependencies over rewriting in a different language I know less well any time.

And good look finding replacements for Python/Django packages in Go or Rust.

Don't get me wrong. You'll find a lot of libraries for those languages. But probably not everything...

[+] FraaJad|7 years ago|reply
If it is stability you are after, choosing Java on JVM will get you there much better. Java written 20 ago still compiles and works fine today.
[+] ris|7 years ago|reply
What you're actually saying here is "Using Go I'm going to leave the application with unmaintained versions of its libraries", which is a bit of an alarm-bells statement from a security point of view.

It's perfectly possible to build a python application that is extremely reliably tied to its dependencies, you just have to avoid the default python package "management" solutions that get shoved down everyone's throat (pip, virtualenv, pipfile, tox and the like). I've been building python projects on top of Nix for years and have managed to maintain stable and reliable results.

[+] Animats|7 years ago|reply
Having used all three of those languages, I've commented on this before.

Go is mediocre as a a language, but it has one big edge - if you're doing web backend stuff, the libraries for that are the ones used internally by Google. So they're being executed billions of times per second on Google servers and tend to be well behaved. Python tends to have multiple libraries for basic functions like the database interface, all with different bugs. If you write web backend stuff in Go, it will probably work.

Having suffered through the Python 2 to 3 transition, I'm less of a Python fan than I used to be. At some point, they need to stop adding features. Currently, Python is acquiring unchecked "type hints", which look like a terrible idea. Optional typing, sure, but unchecked type hints? If the compiler enforced them, they might be worth something. (And no, running a separate type checking tool is not the answer. Especially since no such tool exists that's production grade and in sync with the language.)

Rust has no idea when to stop adding features. Rust started as an imperative language and then morphed into a semi-functional language. Mostly to get a sane error handling model. The borrow checker was a brilliant innovation, though. I used Rust for a while, but quit about two years ago in disgust. I never want to hear "that only compiles in nightly" again.

[+] tomjakubowski|7 years ago|reply
> Rust has no idea when to stop adding features. > I used Rust for a while, but quit about two years ago in disgust. I never want to hear "that only compiles in nightly" again.

I don't understand how you can reconcile these beliefs. They stabilized too many features, but you also want more nightly-only features to be stabilized?

It's my experience as a Rust programmer since pre-1.0 that very few libraries require nightly; they tend to be niche things (e.g. Rocket) by authors that prefer to experiment on the bleeding edge of the language over having stability or many users.

[+] apta|7 years ago|reply
> So they're being executed billions of times per second on Google servers

So are Java and C++ programs at Google. golang is very mediocre for webdev, if anything, it can be used for small network tool kind of stuff. For anything that needs to be more complex, Java would be a better fit (e.g. golang doesn't have annotations that can be used for validations or automatic mapping between different internal and external types, which come in very handy and prevent bugs).

[+] paulvorobyev|7 years ago|reply
>And no, running a separate type checking tool is not the answer. Especially since no such tool exists that's production grade and in sync with the language.

What does "production grade" mean w/r/t such a tool?

[+] LawnboyMax|7 years ago|reply
Strongly agree with author. I really like Python and at one point thought that it makes sense to write almost any project in it, since it's so versatile and I know it pretty well.

After working on a quite large application and having to use lots of assert(isinstance(arg, type) at the beginning of almost every function, I began to think that a strong type system is very much needed for large projects. I believe this was one of the reasons why Twitter moved from Ruby to Scala.

I still love to use Python for hacking something quick for myself. But I also look at some popular strongly-typed languages now and hope to get better at one of them soon.

[+] jsjohnst|7 years ago|reply
> I began to think that a strong type system

You’re confusing static/dynamic and strong/weak. Python is a strongly typed language, but also is dynamicly typed language.

Need proof? Try doing this:

a = 3

b = “3”

c = a + b

> having to use lots of assert(isinstance(arg, type))

Python has type hinting now, try using a current version of Python and this isn’t needed.

[+] vonseel|7 years ago|reply
> having to use lots of assert(isinstance(arg, type) at the beginning of almost every function

Python should not be written like this; assertions can be ignored at runtime: https://docs.python.org/3/using/cmdline.html#cmdoption-o

A much better solution is to use qualified try/except blocks to handle potential errors.

[+] joshvm|7 years ago|reply
> assert(isinstance(arg, type))

The most irritating thing I've found is trying to write a function that accepts either a path or the raw contents of a file.

Python 3 makes this easy, you can check against string. Python 2 of course treats everything as a byte string.

In the end I just check the length of the input if the interpret identifies as python 2. If it's a short byte string, I assume it's a file name. Otherwise the minimum expected file size is almost certainly larger than the maximum path length.

[+] ris|7 years ago|reply
You're confusing static typing with strong typing. Python is strongly-typed, but not statically typed.

Python's optional typing could also be used to get rid of a lot of your assertions.

[+] mark_l_watson|7 years ago|reply
Thanks for The writeup, I am going through a similar process. I just retired last Friday (I am an old guy) from managing a machine learning team and everything was done in Python.

For personal projects (and past consulting work) I have been loving Lisp languages since the 1980s but many languages have so many of the benefits of Lisp (thinking of Haskell and Julia) and there is definitely a ‘lisp tax’ on deploying and shipping products.

Anyway, for writing new books, writing open source projects, and perhaps some consulting, I would like to cut back to a single language, instead of the 4 or 5 I use now.

EDIT: I did some Go Lang codelabs when I worked at Google and have played with Rust (principally spending some evenings with an open source blockchain project) and neither language will be my home base language.

[+] pdimitar|7 years ago|reply
If I retired today I'd try really hard to only use Elixir and OCaml.

Elixir is perfect for servers and fault-tolerance. Also scales quite well to several tens of machines before needing any special tooling.

For a huge percent of most projects today Elixir would be ideal.

OCaml I want to use more for its really concise and strict (but extensible) syntax. And mostly for its strong typing and really well-optimized compiled native code.

Congratulations on retirement, relax and tinker at your own pace. ^_^

[+] dangoor|7 years ago|reply
Congrats on your retirement!

And enjoy your opportunity to work with whatever languages you like. The variety of tools available today is so much greater than it was in the 80s.

[+] siteshwar|7 years ago|reply
> Rust is developed by a community, and was started by Mozilla. Go development seems to be de facto controlled by Google, who originated the language. I'd rather bet my non-work future on a language that isn't controlled by a huge corporation, especially one of the main players in today's surveillance economy.

Anyone else agree with this view ? Programming languages should be choosen based on technical merits, rather than who is behind it. Is there a lesson from history that I am missing ?

[+] xyproto|7 years ago|reply
The strengths of Go become apparent when working in teams and when deploying, not when reading blogs about the language before trying to write code in it.
[+] xedrac|7 years ago|reply
I'm a long time C++ developer, and recently took a job where I write a lot of Python in a large code base. Initially it seemed great, but I quickly realized how uncomfortable it is to have no safety net up front. The interesting thing is Python eliminates a lot of programming errors you may encounter in C++. But it introduces a whole different set of programming errors you would never see in C++. I don't feel comfortable using Python for any project where the code must be correct. I would use Rust every time for that.
[+] 0815test|7 years ago|reply
Not trying to dismiss this person's work in any way, but is there any reason why this blog post should be considered notable? They clearly say at the end that they've yet to write any real-world code in either Rust or Go, and so far have just been learning about either language - so how come that this shoots up to the top of HN? Is it just the magic keywords "Rust" and "Go" that do that?
[+] kumarvvr|7 years ago|reply
At 15000 loc, if the author found it difficult to organize python code, then it's not python's fault.

I find it a pleasure to organize code in python.

What am I missing here?

[+] sidcool|7 years ago|reply
I have a genuine question. One of Rust's biggest plus points is having no GC and the related overhead at runtime. Go, on the other hand does use GC. Then how come so many successful Systems programs (e.g. Kubernetes) are written in Go. Anything I am missing here?
[+] zachruss92|7 years ago|reply
For me I think it comes down to what is the right tool for the job. I have been using Python for years for web scraping & data analysis (via Scrapy and SciPy) and I feel that they are the best tools for what I am doing.

Now that a lot of my scraping needs require me to render the DOM, I have had to run a headless web browser (previously PhantomJS/Splash and now Chrome). Chrome released the Puppeteer library which is the most straight forward way to interact with Headless Chrome. This is in Node so that is what i’m using.

On my last project we needed to parse through a large amount of HTML files to extract relevant information (about 70,000,000) and put it in a DB for further analyses. I have done similar things in Python before, but there are blocking/concurrency issues that can make it slow and frustrating to build.

My solution was to just do this in Go. I can then run my extraction in a goroutine. Go’s selling point for me was speed (relative to Python) and dead simple concurrency - which made it the right tool for the job.

Edit: Spelling, iOS’s autocorrect can be so annoying sometimes :)

[+] ChrisSD|7 years ago|reply
This is more of a general observation but why are Rust and Go so often mentioned in the same sentence? They are designed around very different use cases such that a direct comparison doesn't seem to warrant as much attention as it seems to get.
[+] therealmarv|7 years ago|reply
15K LOC in Python. Be prepared to increase your LOCs to at least 33-100% when writing in Rust or Go, hopefully your error rate by line of code will decrease more than your overall LOC. There are not only benefits.
[+] yakshaving_jgt|7 years ago|reply
I share the author's sentiment, and it's what lead me to Haskell. It's faster and cheaper for me to write prototypes in Haskell than it ever was in Ruby.

And at least in startup-world which is the bulk of my career, the speed at which you can build a reasonably robust prototype can be the life or death of a business, and this prototype usually ends up being the production product anyway. Prototypes are seldom rewritten "properly".