Programming language semantics turn out to be far, far thornier than most people (even people who work on language committees!) give them credit for. And the memory model turns out to be the absolute worst part of it all: the hardware plays lots of fun games on you, the OS plays lots of fun games on you, and the compiler itself is playing lots of fun games on you.
Multithreaded--really, multiprocessor--memory models are really, really confusing. Hardware memory models are generally described in terms of what reorderings are possible (e.g., loads may be ordered before subsequent stores), which is easy to understand from a hardware perspective but less easy to deal with from a language perspective. There turns out to be a wonderful theorem, though, that lets you hide all of the hardware ugliness: if you properly synchronize your code, then you can't see any of the reordering, and the compiler merely has to guarantee that the synchronization is properly handled. This is the data-race-free programming model, and it is the primary basis for all modern memory models. Stray away from this model (e.g., relaxed atomics) and you reenter the world of pain, for which no satisfactory solution has yet been found.
Discovering the pains of pointer provenance is even more recent, arguably only about a decade old. For a long time, pointer provenance has been handwaved on a vague data-dependency basis (you see this in LLVM's LangRef, for example). Unfortunately, it turns out that data dependencies are not generally preserved by compilers (this also doomed the release/consume aspect of the C/C++ memory model). An escaping/lifetime-based pointer model is closer to the actual one used by compilers, but there remains tricky things about things like memcpy [1].
It turns out there are two things that make pointer provenance painful: integer-to-pointer (where does the provenance come from?), and memory (since this usually ends up creating implicit integer-to-pointer casts in most putative models that you really want to avoid). The lesson we know from pointer provenance is that pointers are not integers, and cannot be losslessly roundtripped via integers, and Rust being newer and with a heavier emphasis on safety gives it a better chance of adopting a model here that more forcefully breaks this mistaken link.
[1] One consequence is that most formal pointer provenance models actually don't let you legally write memcpy in C code.
Hmm, I am getting a sense that modern engineers shows a sign of losing touch to computer science theoretical foundation.
It's not funny game. It's part of abstraction hierarchy and physical computing model vs. programming language computing model.
It's only funny because one would presume that there ever was a time the hardware was not playing "funny" game.
And no, it always has a giant abstraction gap between hierarchy, and that's a fundamental part of modern electronic computing. One shall allow oneself to learn those and get used to how things are actually working.
I hope this new upper story of the tower is successful. I don't reckon I've actually needed anything weaker than the tower's ptr.addr() and ptr.with_addr(addr) in code I've actually shipped anywhere. Maybe people have examples of stuff (in Rust or otherwise) they've written where they're confident that would not be enough?
A possible non-Rust example: SpiderMonkey uses a trick we call NaN-boxing to represent JS values, which takes advantage of the fact that doubles have 2^52 different bit patterns for NaN to store 48-bit pointers with 4 bits leftover for a type tag. Provenance for the ptr->int->double->int->ptr loop might still be manageable using a "base of the GC heap" pointer and with_addr(), but there are also cases where built-in objects store arbitrary non-GC pointers in private slots (for example, a Map object has a raw pointer to its heap-allocated backing array). In that case, there isn't an obvious way to reestablish provenance (although we could hack it for e.g. Cheri by adding a layer of indirection through a GC allocation that stores the real pointer).
The only example I can think of I've seen released code is the "xor linked list" trick, where each node of a link list stores "neighbours = prev xor next" (rather than pointers to the 'next' and 'prev' thing in the list).
The idea is this lets you iterate up and down the linked list, as if you have prev, you can do "neighbours xor prev" to get next, and "neighbours xor next" to get prev.
However, while in that particular program that did create a notable speed improvement and memory saving, I think it's probably worth sacrificing to the greater good.
There are definitely cases where you are converting integers into pointers ex-nihilo. Memory-mapped I/O is one (write something special to memory location 0x04ff0030 and it turns on the light!).
My current Rust project would probably need to hit up the exposed_addr interface, but it's definitely in the twilight of "how to even memory model" since it involves trying to reason about the provenance of pointer values in the register array passed to you by the third argument of the signal handler.
A nice think about this work is that adopting this strict model in real life codebases is going to bring to light cases where it isn't enough, and hopefully more scrutiny can be directed towards those cases.
This is my question as well. I don't write a lot of C or Rust code, so I'm curious when int -> ptr and ptr -> int conversions are actually needed in practice. The only scenarios I can think of with my limited domain experience in are low-level things where you actually know the numeric addresses of certain important locations in memory. In which case, saying "yeah, that's a lower level in the weakenings tower" seems sensible.
I found this article hard to read. I was trying to understand what the current state of affairs is, and how these new abstractions improve the matter. E.g. the consistency properties, the weakenings, and when they help.
Instead it read more like an exasperated rant, that links to rust toolchain improvements. It does have an account of how to use the new features, which was good. But I couldn't glean when and how that would help me as a user
There's a prior post that this one is building off of, but basically:
1. Pointers are not addresses; they are pairs of (allocation, address). This is known as pointers having provenance.
2. Treating a pointer as an int-sized address is merely a hardware optimization; and there is already hardware that deliberately does not do this (e.g. CHERI and maybe ARM PAC).
3. The compiler needs to know the allocation part of the pointer (the one that gets lost at runtime) in order to determine if a pointer write will change a local variable. If this association is lost then you get miscompiles.
4. Converting an integer back into a pointer (e.g. usize as ptr) does not establish pointer provenance, will break CHERI, and will miscompile on other architectures.
5. Rust made the mistake of allowing #4 in unsafe code. This is entirely unsound.
6. The proposed strict-provenance APIs allows doing something like #4, but sound, by letting the user stitch an address onto a pointer with a compatible allocation. This re-establishes the chain of provenance and avoids the miscompile.
[+] [-] jcranmer|4 years ago|reply
Multithreaded--really, multiprocessor--memory models are really, really confusing. Hardware memory models are generally described in terms of what reorderings are possible (e.g., loads may be ordered before subsequent stores), which is easy to understand from a hardware perspective but less easy to deal with from a language perspective. There turns out to be a wonderful theorem, though, that lets you hide all of the hardware ugliness: if you properly synchronize your code, then you can't see any of the reordering, and the compiler merely has to guarantee that the synchronization is properly handled. This is the data-race-free programming model, and it is the primary basis for all modern memory models. Stray away from this model (e.g., relaxed atomics) and you reenter the world of pain, for which no satisfactory solution has yet been found.
Discovering the pains of pointer provenance is even more recent, arguably only about a decade old. For a long time, pointer provenance has been handwaved on a vague data-dependency basis (you see this in LLVM's LangRef, for example). Unfortunately, it turns out that data dependencies are not generally preserved by compilers (this also doomed the release/consume aspect of the C/C++ memory model). An escaping/lifetime-based pointer model is closer to the actual one used by compilers, but there remains tricky things about things like memcpy [1].
It turns out there are two things that make pointer provenance painful: integer-to-pointer (where does the provenance come from?), and memory (since this usually ends up creating implicit integer-to-pointer casts in most putative models that you really want to avoid). The lesson we know from pointer provenance is that pointers are not integers, and cannot be losslessly roundtripped via integers, and Rust being newer and with a heavier emphasis on safety gives it a better chance of adopting a model here that more forcefully breaks this mistaken link.
[1] One consequence is that most formal pointer provenance models actually don't let you legally write memcpy in C code.
[+] [-] fbkr|4 years ago|reply
Relaxed atomics are still data-race free, did you mean non-atomic accesses?
[+] [-] bigcat123|4 years ago|reply
It's not funny game. It's part of abstraction hierarchy and physical computing model vs. programming language computing model.
It's only funny because one would presume that there ever was a time the hardware was not playing "funny" game.
And no, it always has a giant abstraction gap between hierarchy, and that's a fundamental part of modern electronic computing. One shall allow oneself to learn those and get used to how things are actually working.
[+] [-] tialaramex|4 years ago|reply
[+] [-] IainIreland|4 years ago|reply
[+] [-] CJefferson|4 years ago|reply
The idea is this lets you iterate up and down the linked list, as if you have prev, you can do "neighbours xor prev" to get next, and "neighbours xor next" to get prev.
However, while in that particular program that did create a notable speed improvement and memory saving, I think it's probably worth sacrificing to the greater good.
[+] [-] jcranmer|4 years ago|reply
My current Rust project would probably need to hit up the exposed_addr interface, but it's definitely in the twilight of "how to even memory model" since it involves trying to reason about the provenance of pointer values in the register array passed to you by the third argument of the signal handler.
[+] [-] GolDDranks|4 years ago|reply
[+] [-] seo-speedwagon|4 years ago|reply
[+] [-] tooltower|4 years ago|reply
Instead it read more like an exasperated rant, that links to rust toolchain improvements. It does have an account of how to use the new features, which was good. But I couldn't glean when and how that would help me as a user
[+] [-] kmeisthax|4 years ago|reply
1. Pointers are not addresses; they are pairs of (allocation, address). This is known as pointers having provenance.
2. Treating a pointer as an int-sized address is merely a hardware optimization; and there is already hardware that deliberately does not do this (e.g. CHERI and maybe ARM PAC).
3. The compiler needs to know the allocation part of the pointer (the one that gets lost at runtime) in order to determine if a pointer write will change a local variable. If this association is lost then you get miscompiles.
4. Converting an integer back into a pointer (e.g. usize as ptr) does not establish pointer provenance, will break CHERI, and will miscompile on other architectures.
5. Rust made the mistake of allowing #4 in unsafe code. This is entirely unsound.
6. The proposed strict-provenance APIs allows doing something like #4, but sound, by letting the user stitch an address onto a pointer with a compatible allocation. This re-establishes the chain of provenance and avoids the miscompile.