Writing Python inside Rust

[6gvONxR4sf7o]

I know this project may just be for fun, but with WASM targets for all sorts of languages, I'm hoping we get to a future where mixing and matching different languages for different parts of your program will be seamless. Imagine starting a project in an easy language, then migrating pieces to a faster "bare metal" language as needed in a super piecemeal way. Same with moving pieces to a safer language as the project grows, slowly expanding the boundaries of the safe bits as appropriate.

[pjmlp]

Just like the JVM and CLR I guess.

https://en.wikipedia.org/wiki/List_of_JVM_languages

https://en.wikipedia.org/wiki/List_of_CLI_languages

https://www.graalvm.org/docs/

[alfalfasprout]

The problem is that inherently a lot of the times when you want to drop down to C/C++, you're doing numerical code that would really benefit from SIMD operations and many of those tend to not be portable (as far as I'm aware, basic SIMD is just in the proposal stage for WASM).

[Kinrany]

I wonder if we'll ever get to a future where all code is write-only, immutable, and thrown away after compilation.

The choice of the language would become a matter of presenting the compiled code and writing replacements for new code.

The runtime and the type system can't be replaced this way though.

[marvy]

So, at first I thought this approach is doomed, because you lose comments, and what looks like a comment to Rust may not look like one to Python. Example:

  x = y // 2 # floor division

But then I decided to look at the docs:

https://doc.rust-lang.org/proc_macro/struct.Span.html

And I noticed source_text, which "preserves the original source code, including spaces and comments"!!!

Why not just use this from the start then?? Seems like the easy way out, no?

(Disclaimer: I don't know Rust, can't even write hello world.)

[uranusjr]

The author likely just barely missed its introduction. While the article is written recently, the implementation it talks about was published first in early April 2019, right about when source_text was first introduced into nightly.

[oon]

Using Span to get the token location is exactly what I needed! Thanks. I wrote a blog post[0] the other day about making a css macro that compiles into rust for the use in Yew, a front end react like framework.

> However, in rust, there’s no way to differentiate .a.b with .a .b

Now I know that the above is incorrect. I would have never thought of spans so I thank you again

[0] https://conradludgate.com/posts/yew-css/#what-are-the-downsi...

[rav]

The blog post shows two Python snippets starting with "if True:" and different indentation and says the snippets "have a different meaning". However, in this case the difference between the snippets is mainly in their syntax and not in their meaning. The example would have been better if "if True:" was replaced by "if False:" or "if foo:".

        if True:
            x()
        y()


        if True:
            x()
            y()

[m-ou-se]

Good point! Updated.

[ptato]

What is the use case for embedding Python code in a Rust program?

[charliesome]

There could be no reasonable use case for something like this, and yet it would still have artistic value and this would still be an interesting article.

It's a brilliant hack and we are on Hacker News after all.

[choward]

That's exactly what I thought when I read the title. It's very frustrating how many landing pages, project readmes, blog posts, etc that don't answer the question "why?". I usually need to know that before going further when I come across something that I've never heard of before. If they don't have the "why?" somewhere easy to find I just close the tab.

[memco]

I’m just throwing out ideas but what if I wanted to take working python code and convert it to rust? Could I use this to start as the baseline and start replacing bits of it and see that it still produces the same output?

[tedmielczarek]

A friend of mine has a similar project called PyOxidizer that's intended for building standalone executables with Python+Rust: https://pyoxidizer.readthedocs.io/en/stable/

His primary use for it is building distributable binaries for Mercurial, which is written primarily in Python.

[rtpg]

Plugins for example? Game scripts? Think of use cases for embedding Lua into a C program.

[djohnston]

Maybe interfacing w an ml lib like pytorch or something?

[amanzi]

I only had a quick scan through the article, but could this be used to create an executable for some existing Python code?

[granzymes]

If imports worked I could see it being very useful for making graphs. I've heard of people serializing data in other languages and then using Python to plot it.

[uijl]

At FOSDEM there was a talk about boosting Python with Rust [1]. Might be interesting for the people checking this.

[1] https://fosdem.org/2020/schedule/event/python2020_rust/

[pansa2]

Is it common to embed one language's code directly inside another language like this?

Lua is often used tightly-coupled to C, and it doesn't have the significant-whitespace issues that Python shows here. Even so, I've only ever seen it used in separate `.lua` files, never embedded directly within `.c` files.

[iddan]

I don’t think it’s common nor it’s a best practice as you can’t apply static analysis on your embedded code. But it’s really cool Rust allows that and it can be a ad-hoc solution when you need to write a piece of code fast inside of rust

[axegon_]

While I've seen people doing this, I've never done it. I've done the opposite in a small open source project of mine(arguably abandoned at this point due to lack of time and contributors). That said, there is a huge potential here - your code may require some small operation that is cheap to execute but would take you 2 minutes to write in python while it would take you an hour of doing the same in rust. I can think of some use cases for this. Nice article!

[hypewatch]

Why call python from Rust? I think calling rust from python would make more sense. Use it to optimize python functions like how python libraries do with with C (i.e. numpy)

[FridgeSeal]

Suppose I’m building an application in Rust that processes a lot of data, one of the steps in processing that data I maybe want to run through a Python tool like SpaCy or Flair.

How would I go about doing that? I could put the Python code behind a little http API and call it that way, but that’s a bunch of overhead and extra stuff to maintain just to analyse some text. If I embed said Python tools in my Rust code then I can call those tools with significantly less overhead and complexity.

[ComputerGuru]

I wrote this [0] up a couple of weeks ago, which is a hack to put shell code in your rust, with reasons why you would want to.

[0]: https://neosmart.net/blog/2020/self-compiling-rust-code/

[zengid]

I'm having issues seeing the whole post, when I scroll down i just see white, and when I click the subject headers from the menu it jumps down half the page and the mouse wheel scroll gets locked. I'm using Chrome on MacOS.

[kzrdude]

Lua is a better fit for Rust, because it doesn't have the threading restrictions of (C)Python. With Lua you can run a thousand parallel vms in the same process if you want.

But the macro hacks are impressive!

[amrx101]

So I am not the only one with questions regarding its use case? I cant seem to conjure any.

[random32840]

Quickly introducing battle-tested data science frameworks, snippets and patterns without having to replicate them in Rust.

[superdisk]

There's a typo where stringify! is referred to as strinfigy!

Cool article btw!

[m-ou-se]

Thanks! Updated.

[bsder]

Am I the only person having trouble reading this blog? That color scheme makes my eyes bleed.

It seems like it's a light gray background with a slightly (not much) darker gray text. The contrast and thin font weight is terrible.

And the pink is practically vibrating on the page.

At least the blue is okay.