A Python Interpreter Written in Rust

[hathawsh]

This is wonderful. This could become the best way to move Python projects to Rust: initially just run on the RustPython interpreter, but then optimize low level routines in Rust. In 15 years I wouldn't be surprised if this or something like it surpasses CPython in popularity.

Still, no discussion about Python implementations is complete without some mention of the infamous GIL (global interpreter lock). :-) CPython has it, Pypy doesn't (I think) (EDIT: yes it does), Jython doesn't, etc. What is the GIL plan for RustPython?

[std_throwawayay]

The GIL comes with great convenience as you don't have to worry about a whole host of data races. It's no silver bullet but it's mighty convenient if you only do some multi-threading like in a web-server.

Many libraries are not prepared for the disappearance of the GIL and while it's not a general problem for python per se it will be a great amount of work to make every library compatible with GILless python.

Therefore I think that you must always provide an option for the GIL that is enabled by default in order to provide backward compatibility.

[schmichael]

PyPy does have a GIL: http://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil-...

Jython, as you state, does not: http://www.jython.org/jythonbook/en/1.0/Concurrency.html

[amelius]

Writing a concurrent runtime system including garbage collector is a serious effort, and that's why all those other versions of Python don't support it and are stuck with a GIL. Hence, I highly doubt that this Rust version of Python has gotten rid of the GIL.

I'd love to see a better separation of language and VMs. I think it's a bit sad that a language designer has to either implement their runtime system from scratch, or has to run it on top of a VM that was designed for another language (Java in the case of Jython).

Therefore, the thing I'm looking forward to most is a concurrent, generic and portable VM written in Rust.

[tanilama]

If we are ever going to get rid of GIL, we need to get rid of Python's C extensions all together.

It is not a real Python implementation if not compatible with C extension, it is just embedded DSL that has Python flavor syntax.

[sametmax]

Indeed, I'm very excited about this.

Rust + Python seems a natural combinaison to me, and being able to have one single dev env (and maybe in the end, one single deployment mechanism) to do both is a killer feature.

And actually, I think having Python written in Rust would provide some other very nice properties:

- limit the number of bug you can introduce in the implementation because of the rust safety nets;

- can still expose a C compatible ABI and hence be compatible with existing extensions;

- the rust toolchain being awesome, it may inspire people to make it easy to compile a python program. Right now I use nuikta, which is great, but has to convert to C then compile, which make it a complex toolchain.

[antpls]

> This is wonderful. This could become the best way to move Python projects to Rust: initially just run on the RustPython interpreter, but then optimize low level routines in Rust. In 15 years I wouldn't be surprised if this or something like it surpasses CPython in popularity.

What you are describing is simply a JIT compiler. Maybe are you suggesting to rewrite PyPy (its C part) in Rust?

[tln]

Agreed, this looks like a great project!

Grumpy was supposed to accomplish the same for Python->Go, and although now abandoned, probably holds some lessons in how to design a platform to help Python projects get Rusted.

Grumpy compiled python code to fairly unreadable Go, and then quickly compiled the result. One effect of this is that a programmer could theoretically refactor the resulting Go code gradually.

[woolvalley]

I would personally try to move away from python at this point for greenfield projects. The GIL is so baked into the language, if you removed it a bunch of current python code will probably break in subtle ways.

Modern languages need proper multithreading support, static types and fast compile speeds. Use golang, use kotlin, use dart, use anything but python & javascript.

[loeg]

It's basically impossible to implement a CPython-compatible language without a GIL (or you lose single thread performance by using very fine-grained atomics/locking). Python has very specific multithreading semantics that are a function of the CPython bytecode and the GIL, and programs rely on this.

[lykr0n]

PyPy has the GIL.

[sametmax]

I want it to have the GIL, because I want it to maintain compat with C extensions.

We already have a plan to bypass the GIL: multi interpreters.

Having an implementation in Rust may make future improvement to Python easier, so it's better to have something exactly similar first, then start to hack it.

[lykr0n]

Neat. Was able to clone the repo, run cargo run, and drop into a python shell. Doesn't seem like can do much right now, but I really like the idea.

  >>>>> a = [1,2,3]
  >>>>> a[2:]
  [3]
  >>>>> a[1:]
  [2, 3]
  >>>>> fh = open('~/.ssh/id_rsa.pub', 'r')
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: RefCell { value: [PyObj instance] }', src/libcore/result.rs:999:5
  note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

What would be really cool if this could one day be like Nuitka- but in rust. Write in python, compile into Rust. Maybe even support inline Rust like cPython supports inline C.

[ciupicri]

While your code was valid, open('~/.ssh/id_rsa.pub'...) won't work on any Python other interpreter, you need to expand ~ from the path. For example you can use:

  open(os.path.expanduser('~/.ssh/id_rsa.pub'), 'r')

[Ralfp]

> cPython supports inline C

First time I am hearing this. Can you share an example?

[Animats]

It's a naive interpreter. The source is parsed into a tree and then flattened into byte code. All values are stored in PyObjectPayload structures. Each operation has a function. Here's "hex":

    fn builtin_hex(vm: &mut VirtualMachine, args: PyFuncArgs) -> PyResult {
        arg_check!(vm, args, required = [(number, Some(vm.ctx.int_type()))]);

        let n = objint::get_value(number);
        let s = if n.is_negative() {
            format!("-0x{:x}", n.abs())
        } else {
            format!("0x{:x}", n)
        };

        Ok(vm.new_str(s))
    }

There's a big dispatch table, generated at compile time, and an interpreter loop. Just like you'd expect. It's useful if you happen to need a Python implementation to embed in something and want something cleaner than CPython. Probably slower, though.

[4gotunameagain]

Serious question with no irony at all. But why? How is this useful?

[orf]

If anyone is at FOSDEM this will be presented tomorrow in the rust room. Can't wait!

[ZeikJT]

I found the slides and info at [1] but does anyone have a recording of this presentation?

[1] https://fosdem.org/2019/schedule/event/rust_python/

[sametmax]

If anybody had any doubt about how welcome the project is...

Since this has been posted to HN, the repo got 8 new PR.

[gonational]

Now this is something I will donate money to…

Python with Rust as its foundation sounds like the best idea ever.

I’m curious to know whether or not it would be possible (or reasonable) to eventually get the same or better performance as CPython.

[Waterluvian]

CPython is really quite slow by design. The reference implementation is meant to be obvious and pretty easy to interoperate with C.

I think a RustPython implementation would be pretty cool. You could definitely take that opportunity to worry about performance more than CPython does while also worrying about interoperability more than PyPy does.

Or I'm missing your point and you're suggesting a drop-in replacement for CPython that supports all the same C-based libraries as CPython does.

[grok2]

> Python with Rust as its foundation sounds like the best idea ever.

Why do you think it's the "best idea ever"? What are the benefits over any other Python implementation?

[johnisgood]

I'm curious to know that why isn't it the case right now? Isn't that one of the selling points of Rust, being blazing fast? Or perhaps just this particular piece of software has been implemented poorly?

[systems]

I wish, I wish, I wish .. something like this is done for Tcl/Tk

I like Tcl the language a lot, and I love the idea of two language systems

One high level for scripting Tcl One low level for high performance commands and parts Rust

You can of course do that today, using Tcl and C But .. well C is no Rust

[MR4D]

Given all the great ideas here, maybe it’s time to fork Python. Ok, so maybe a close derivative as opposed to a true fork.

Given the language names involved (Rust & Python), I’d like to suggest “Copperhead” as a name for it.

[sametmax]

Forking implies keeping the C code.

I think a rewrite in rust is more future proof: we benefit from a safer, more modern language to implement it, which comes with cargo, and hence, the potential of an hybrid python/rust toolchain and dev plateform.

[bakery2k]

How would your Python derivative differ from RustPython?

[chadrs]

I stopped paying attention to the Rust parsing ecosystem for a while, curious how LALRPOP compares to nom/pest/combine and if something about python's grammar led to the choice.

[losvedir]

Totally different use cases. nom/pest/combine are parser combinators, where you stitch the functions together yourself. LALRPOP is more in the vein of yacc where you specify the grammar and it generates the Rust parsing code for you in a build step.

[cs702]

Yes. In particular, I like the potential for:

* using Rust's borrow-checking to develop new lightweight/shared-memory multiprocessing tools for Python (think "import SharedMemoryPool from multiprocessing") without having to mess with the GIL, so as to maintain compatibility with existing libraries;

* using Rust's type inference on Python code for applications in which type safety is highly desirable; and

* compiling Python code for speed, targeting all architectures and platforms supported by Rust (e.g., WebAssembly).

[yuchi]

I want to create the Ramon’s Law (as a corollary of Atwood’s Law):

> Anything that can be Written in Rust, will Eventually be Written in Rust

Please go on, cite me, consider it «Attribution, Share-alike»

[amelius]

Cool. Rust with a garbage collector :)

[nostrademons]

Curious what the general story on Rust <=> Python interop is like now (beyond writing a Python interpreter in Rust). I'd checked out rust-cpython a couple years ago, and I see there's now PyO3, but I just looked at some code samples and it seems like they're a long way away from the convenience that you get with CFFI + C code, boost::python, or even SWIG. Anyone have experience writing Python extensions in Rust or embedding a Python interpreter into a Rust program? Could you comment on how hard the process was and how robust the result is?

[bibyte]

This is awesome. I hope more features from CPython are ported. It looks a little bare bones right now. Is there any chance it will gain full Python compatibility ?

[FrankDixon]

How nice, even a wasm interpreter, that's dandy

[kodablah]

Neat. Now keep a hash of the types/info for instructions at runtime, and when a set is called enough times and appears reasonably pure/simple, JIT it with cranelift (akin to a tracing JIT).

[tyingq]

Is there anything about Rust that would make it hard to use existing python extensions that are written in C? I see the FFI docs, so it seems like it would be fine, but have no Rust experience.

[tasubotadas]

Does it have a GIL? If not, it's already better than CPython :-D

[fxfan]

I don't get the wasm craze in the rust crowd. There is no popular language that cannot be compiled to wasm. It just feels sad to spend time there.

[nicoburns]

Rust is particularly suited to WASM due to the lack of GC and runtime (which tend to be quite large, and have to be downloaded in a web context). And due to it being able to achieve very high performance (which is the whole point of WASM). The only other languages that really compete here are C and C++, and they don't have the great, easily installable library ecosystem that Rust has, and they aren't very accessible to JavaScript developers).

Personally I'm more excited about other uses of Rust, but I can see why people are excited about Rust and WASM.

[lukeqsee]

I think the point is Rust is a very good language in its own right. The fact it compiles to WASM is a great feather in the cap.

They are able to deliver a demo in the browser for something that would normally require downloading and compiling. I think that’s pretty cool to show for a project people wouldn’t normally be able to try out with such low barrier of entry.

[mjw1007]

If this ended up making it possible to run Python in the web browser, that might be interesting to a fair number of people.

(There are advantages to running the same language on the server and client, and there's plenty of server-side web application Python code out there.)

[geofft]

I think it's less "the Rust crowd is excited about wasm" and more as "there is a crowd which is excited about doing certain things with languages and software, and Rust and wasm are both exciting means for those ends." Perhaps a good approximation is, they're both tools for using high-level programming techniques with high performance for domains that were previously constrained in terms of what languages you could use.

[xkapastel]

Most of the popular languages are not making a serious effort to compile to WebAssembly, though. It's totally reasonable for Rust to focus on.