Translating Quake 3 into Rust

[floatingatoll]

One highlight of this post is their work exposing a memory safety error in the original source code:

> The Rust compiler also flagged a similar but actually buggy example in G_TryPushingEntity where the conditional is >, not >=. The out of bounds pointer was then dereferenced after the conditional, which is an actual memory safety bug.

[Avamander]

Interestingly I've too found bugs in original source similar to this when rewriting in just TypeScript, so I suspect it's just someone taking a look that allows the bug to be discovered.

[ahomescu1]

Author here: I wanted to let everyone know we made some small edits to the text, and added two new branches to our repository: transpiled containing the raw transpiler output [1], and refactored containing the same code [2] after a few refactoring commands.

1. https://github.com/immunant/ioq3/tree/transpiled/quake3-rs

2. https://github.com/immunant/ioq3/tree/refactored/quake3-rs

[phkahler]

Is there any progress on C++ to Rust? I know the whole language is huge, so maybe even a little support could be opening a can of worms.

[wyldfire]

I recall seeing a presentation of c2rust by Andrei at a conference a while back. It's great to hear that it's making good progress. I'm curious at how good c2rust is at producing idiomatic rust code. It seems like that would be difficult and yet very valuable.

> we'd love to hear what you want to see translated next.

qemu [1] is all in C (probably C89 or "gnu89") and would make an interesting project, IMO.

[1] https://github.com/qemu/qemu

[K0nserv]

I looked for the transpiled source, but I couldn't find it. I suspect it's not particularly idomatic Rust since it has to maintain all details of the C code however I think it's a very promising project anyway.

The fact that they found a memory safety bug is excellent. Maybe translating old C projects to Rust via this method can be used to find memory issues as well as to aid rewriting in Rust projects.

EDIT: You can try the transpilation yourself on https://c2rust.com/

Here's an example of the output:

    #![allow(dead_code, mutable_transmutes, non_camel_case_types, non_snake_case,
             non_upper_case_globals, unused_assignments, unused_mut)]
    #![register_tool(c2rust)]
    #![feature(register_tool)]
    #[no_mangle]
    pub unsafe extern "C" fn insertion_sort(n: libc::c_int, p: *mut libc::c_int) {
        let mut i: libc::c_int = 1 as libc::c_int;
        while i < n {
            let tmp: libc::c_int = *p.offset(i as isize);
            let mut j: libc::c_int = i;
            while j > 0 as libc::c_int &&
                      *p.offset((j - 1 as libc::c_int) as isize) > tmp {
                *p.offset(j as isize) =
                    *p.offset((j - 1 as libc::c_int) as isize);
                j -= 1
            }
            *p.offset(j as isize) = tmp;
            i += 1
        };
    }

[smolder]

The long tail of patterns c2rust would need to recognize to reliably make idiomatic code seems like a much larger problem. With that as the goal, it might make sense to build a separate rust-to-rust "idiomaticizer" that can recognize opportunities to reduce code complexity through refactoring, and apply idiomatic patterns without changing application behavior.

[flohofwoe]

Here's an example from another code base, the Linux (GLX+X11+GL) extract of an 3D API wrapper around OpenGL:

https://github.com/not-fl3/miniquad/blob/c2rust/sokol-app-sy...

The generated code looks fairly straightforward, however I wouldn't call it "idiomatic".

What's exciting though is that (at least) both Rust and Zig get this ability to translate C code into the "native" language (and sometimes even finding bugs in the process). This basically turns C into a "meta-language" to create "bedrock" cross-platform API modules for a variety of modern languages, essentially the next step from "C as a cross-platform assembler".

And to be honest, this sort of "boring" low-level system API glue code which mainly talks to native C system APIs anyway isn't a joy to write in any language, and also doesn't benefit much from more modern language features, can just as well do it in C once, and then "transpile everywhere".

[masklinn]

> I'm curious at how good c2rust is at producing idiomatic rust code.

It’s not. It should eventually improve but currently the goal is to

> build a rock-solid translator that gets people up and running in Rust.

[ZiiS]

I would highly value something which turned code into idiomatic code; I don't care which language.

[giovannibajo1]

Out of curiosity, how do compilation times look, for a fresh build and for the edit-compile cycle?

[tombert]

Out of curiosity, has anyone here tried taking something like Quake 2 or 3 and porting it to run on the JVM?

This project kind of makes me want to try porting Quake 3 into Clojure, but I want to know the limitations in doing so.

[cartoonfoxes]

https://bytonic.de/html/jake2.html

[cs02rm0]

I'd imagine the graphics libraries is where the JVM falls down.

[arendtio]

I would love to see them translating the Linux kernel, but that might be a bit too ambitious.

[kstenerud]

That's super cool!

What I'd really like to see is benchmarks that compare the rust version to C.

[steveklabnik]

Benchmarks would be interesting, in the sense that it would help make sure that the translation is doing the same thing, but it wouldn't say a ton about Rust vs C in a more general case. This translates to unsafe code, which means it's not really representative of Rust generally.

[rinon]

We didn’t benchmark this project (yet) but previous c2rust translations had approximately equal performance to the original. This is to be expected since the transpiled output is unsafe Rust that is equivalent to the original C code. One caveat is checked va unchecked array indexing, but unless an array indexing operation into a statically sized array is hot _and_ the Rust compiler can’t elide the checks, that’s unlikely to make much of a difference.

[AdmiralAsshat]

What is the realistic best-case scenario? Has anyone demonstrated that well-written Rust code should run faster than C, or is the hope simply to get performance parity with C while maintaining memory safety?

[jtaft]

This is some awesome PR too! Didn't expect them to be a software security consultancy.

[steveklabnik]

Some of this work was even funded by DARPA...

[bbmario]

Is the final code readable/maintainable?

[ahomescu1]

We added the transpiled code in a new branch at https://github.com/immunant/ioq3/tree/transpiled/quake3-rs and also a more cleaned up refactored code base at https://github.com/immunant/ioq3/tree/refactored/quake3-rs

[steveklabnik]

There's an example of the output and a discussion of this elsewhere in this thread. TL;DR: not as much as it could be, yet. This is a work in progress.

[edflsafoiewq]

No.

[Waffle5]

Would be interested in a performance comparison - c (using opengl1, etc etc) vs c2rust.

[willvarfar]

It would be really cool to see snippets or even the whole original and transpiled code side-by-side.

When I was working on compilers I spent as much time making such a script in a web-browser with scrolling etc as I did studying the results.

[ahomescu1]

We added the transpiled code in a new branch at https://github.com/immunant/ioq3/tree/transpiled/quake3-rs and also a more cleaned up refactored code base at https://github.com/immunant/ioq3/tree/refactored/quake3-rs

[saldivarcher]

https://c2rust.com will show you side by side comparisons, although it isn't the best looking code; in terms of readability. But as mentioned in other threads, the refactoring tool (https://github.com/immunant/c2rust/tree/master/c2rust-refact...) aims to fix that.

[faebi]

Does it mean we could translate languages like Ruby and Python to Rust?

[K0nserv]

While it might be possible my guess is it would be much more difficult than C. AFAIK Rust is more or less a superset of C whereas Python and Ruby have numerous advanced features that aren't present in Rust and would likely require a bespoke Rust runtime to emulate the behaviour. Types is also a whole can of worms here.

I think in relation to Ruby, Python and other languages like them Rust's best application is leveraging the support for C FFIs to replace hot paths. While Rust has a significant learning curve it's probably much simpler, for say a ruby programmer, to pick it up and write safe performant code than C.

[johannes1234321]

Technically it is possible. The gain won't be there for most cases, though as the generated code would be complex and bad as variables often have to be carried as "variant" types and you have to plug in to the library (which has a specific C API) and so on.

Facebook for some time did this with their "HipHop for PHP" transpiler which translates PHP into C++ and then fed it to gcc. That worked, but due to all the type conversion and similar magic required on all calls this lead them to compilation times of 24 hours and interfaces which were hard to interact with for language bindings, thus they went back to write a scripting engine (HHVM) which now runs their forked PHP.

[lmm]

Not usefully, since those languages require an extreme level of dynamism (even if rarely used by practical programs). E.g. Python requires that an object's methods are a dictionary that can be accessed and manipulated as one, so you'll never be able to translate Python objects into Rust traits. You could of course write a Python interpreter in Rust.

[jashmatthews]

CRuby's JIT basically works by translating Ruby to C.

[johnklos]

Why is there such a push to use this new language that's so inaccessible to much of the world? Not everyone has a multi-gigahertz, multi-core, multi-gigabyte computer to develop in Rust. It seems like unnecessary stratification for nebulous gain.

Perhaps the Rust team should make their compiler work on reasonable resources, like 32 bit processors, before marching off to try to convert the world.

Or perhaps we should have rust2c instead, so people can take all this Rust code that everyone is pushing and compile it on modest machines.

[AceJohnny2]

> Perhaps the Rust team should make their compiler work on reasonable resources, like 32 bit processors

Rust compiles for and runs on STM32, a 32-bit microcontroller:

https://docs.rust-embedded.org/discovery/

And here's an index of supported embedded hardware:

https://github.com/rust-embedded/awesome-embedded-rust#drive...

[steveklabnik]

We distribute binaries of the compiler for a bunch of 32 bit targets. It will of course be slower on slower machines, but that’s just physics.

mrustc is "rust to C", though it's not quite ready for the general case yet. It's primarily for translating the compiler, to ease bootstrapping.

[smolder]

The compiler performance may be a perfectly valid reason not to use Rust. Don't mistake the frequent Rust articles for the whole world moving to Rust. It's more like a few excited pioneers exploring its use cases and doing foundational work. Besides, if it helps other people build more stable, secure, or better performing software tools for you to use, isn't that good, even if you don't use it directly yourself?

[p1necone]

> "multi-gigahertz, multi-core, multi-gigabyte"

Unless you're talking about people holding on to hardware from 20+ years ago then yes - basically everyone with a computer, or a mobile phone has something that fits that classification.

People complain about Rust having long compile times, but I doubt it would be unusably bad even on something like a core 2 duo.

[elihu]

Poor computer security poses an existential threat to modern civilization. Slow-ish compile times do not.

[zozbot234]

Even some large C++ projects are becoming hard or perhaps impossible to build using a 32-bit compiler. It's not something that only hits Rust.

[antpls]

> Not everyone has a multi-gigahertz, multi-core, multi-gigabyte computer to develop in Rust.

Even low-end Android phones have multi-gigahertz, multi-core, multi-gigabyte memory, so actually, many people have that in their pockets

[imtringued]

I'm not working with Rust but I just tried a quick benchmark by compiling a random project.

git clone https://github.com/hyperium/hyper

Full Build: time cargo build (1 min 42 seconds without download) (349K LOC total [0])

Recompile: time cargo build (2 seconds) (around 18k LOC)

I don't see the problem. The full build also includes all the 53 dependencies. So we are talking about a compilation speed of about 3.5k lines/s on a dual core 11" laptop from 2017 with turbo boost disabled.

I've personally worked on C++ projects where changing a single header can cause 5 minutes of compilation on an incremental build. Any slowdowns you have experienced may have been caused by a specific library or project.

[0] according to cloc ~/.cargo/registry/src/github.com-1ecc6299db9ec823/

[vmchale]

> Why is there such a push to use this new language that's so inaccessible to much of the world?

Eh?

[tracker1]

First, rust provides safety checks you don't get in C. Second, the only way you'd improve compiler performance via C conversion would bypass these checks. Third, transpiling through C without these safety checks loses the point of rust in the first place.

The fact is, that there are a great many bugs in the wild that never would have happened in idiomatic rust in the first place. Yes, there is additional overhead to compiling. By comparison, if you use static code analysis on your C/C++ code there is additional overhead, and even then a greater chance of security bugs slipping by compared to rust.

In the end, used server hardware is relatively cheap in most of the world, which does pretty well with this type of code. There are a lot of people using mid-upper range x86 hardware and even to cross-platform compile to lower end arm64 support. At some point, it's less worth maintaining legacy hardware either for pragmatic and/or practical reasons.

Rust was started well after x64 and other 64-bit hardware was in the wild, and 32-bit hardware was for the most part no longer being produced. Supporting such systems is work that practically speaking is less valuable to most people.

As someone that really likes the old DOS based BBS software and related tech, and even supports Telnet BBSes today, I don't always like seeing this left behind to some extent. Last year, a large portion of the community blew up when 32bit was going to be dropped from Ubuntu, mainly because of the need required by wine and the gaming community.

This isn't about supporting 10+ year old apps and programs in an emulated environment... this is about porting existing code in an automated enough way to improve stability and support-ability over time by leveraging a safer language.

[damnyou]

Rust trades off compile time performance for runtime correctness and performance. Far more people run code than compile it.

[twblalock]

> Perhaps the Rust team should make their compiler work on reasonable resources, like 32 bit processors, before marching off to try to convert the world.

It's time to let 32-bit processors go and move on.