Freestanding, ABI-level interoperability with C is planned for Rust, but is not yet implemented. There are two main reasons why being able to be called with dlsym() was less important for us than the current setup:
1. M:N scheduling. We wanted high scalability with hundreds of thousands or millions of tasks, which means we need the capability to have our own scheduler. OS threads aren't lightweight enough for this, especially on Windows, where over 80% of our Firefox users are. Having our own scheduler means we need our own ABI. We would like to make the scheduler optional, but getting it working was a higher priority than making it optional.
2. Safety. On many platforms (basically, anything but MSVC on Windows), the C stack is not bounds-checked; either there is no guard page or the guard page is unreliable because you can overshoot it. This is not acceptable for our Rust code, since stack overflows are a frequent source of exploitable security vulnerabilities for us. So we do our own stack management with a custom ABI that performs stack checks at every function entry, freeing us from this source of pain on all operating systems. Again, this is planned to be optional, but it was more important at this early stage for us to demonstrate advantages over C than for us to duplicate its ABI exactly.
Many libraries, however, would prefer direct interoperability with C to M:N scheduling or stack overflow safety. So we plan to provide a mechanism for Rust code to be called from C without having to set up a scheduler or perform stack checks. It's not implemented yet, but it's very much part of the current design, and I would be thrilled to see it happen.
Now technically you can compile a special python which can handle Go extensions, which is the wrapper mentioned in the second answer, but you can't distribute python modules that ask people to install "this special python" to run (which will reliably work with all their other installed code..)
I would like to see a "go init" function in a C API, that would let you interop the two whichever had the `main()`, since that would let you link with all the other c-implemented scripting languages (python, ruby, etc) which would mean that Go extension support could be added to the main cpython trunk as a compile-time option.
I really hope to see Go and Rust move in the direction of pluggable runtimes that can be embedded into a C host program. Currently, both runtimes require being seated as the main entry-point of the program to function. Both projects also have issues in their bug trackers to rectify this:
Also, a way to compile without kernel (system) calls, so that it can be use for kernel development. It used to be possible with go, but no more with go1. Then these languages would truly be able to replace C.
Clay[1] doesn't get as much attention as it deserves. The authors seem well-versed in programming language theory, but they also understand the practical needs of a systems language. What you end up with is a language that's sufficiently low-level to be a C replacement, but it offers a plethora of mechanisms for abstraction.
Of all the new languages billed as "systems languages", Clay is the only one that excites me.
It can also "quack like a duck" (to put it in TFA terms).
I don't think this stands up: "It must be possible for such contender to be called from C code, because that is what it takes to be called from a modern scripting engine too."
Later the author says, "If programmers wanted a statically-typed, compiled system language for their scripting efforts, they would be using the existing system languages for that already."
I think the reason that scripting languages exist is because of limitations in the established languages. What we think of as "a scripting language" is defined by the gap. If someone else could be a system programming language without the gap - maybe it doesn't need to be accessed.
The traditional unix platform is (1) C and a compiler for it, with access to the unix standard library and (2) some sh or csh derivative, and access to the likes of sed and awk.
Shell gave people a glimpse of what was possible. But shell has poor access to the unix API, and hits the wall quickly once you do anything other than very basic tool development.
So these things called scripting languages started to pop up. People wanted easy access to the unix API like C. They want easy string manipulation like shell. And they want access to functions, maps and lists without having to think about memory allocation.
If you start from the shell and awk and work into that gap you get perl - the canonical scripting languge.
TCL inhabits a similar space to perl. It's also "a scripting language". But it started at the opposite end. TCL starts as an extension language for C.
So I can understand the author's emphsais on close interaction with C. But I don't think they're important if you can take away the shortfalls they're compensating for.
I hope Go will mean I don't have to think about these things. That I'll be able to get a standard build and a go compiler/runtime and have immediate access to lots of power. No more headaches from C build tools or reaching around in the murky water underneath my scripting language.
You are leaving out the importance of libraries and coupling of systems that were initially written independently. This is the same mistake that many Lisps have made, assuming that just because the technology is "superior", people will flock to it and eagerly rewrite all their code (starting with their dependencies).
The C ABI is not great for interacting with scripting languages though, even if it is the best we have right now. For a start there is no type information obtained with dlsym and you basically have to parse headers the interpretation of which is not clearly defined as the C language really defines an API and the ABI is defined by the compiler. Many libraries for this reason do not define stable ABIs and are a pain to use from a scripting language. Hence Python has two ffi modes an ABI interface and a compiler mode.
Now if someone produced a better solution to that it would be useful...
In practice it works quite well, if you restrict the ABI to function calls and word-sized values (long and void). It especially means not to use structs. Instead you need void and getter/setter functions.
there's also ats [http://www.ats-lang.org/]. somewhere on my long-term todo list is to write a library of efficient data strucures in ats, to take advantage of some of its safety guarantees.
It's a pity ATS doesn't get more attention in the "Systems programming language" field. It can call into C, be called from C, provides a powerful type system, and can be used to write very low level code. I've written a few posts about how ATS can interact with C on my weblog here: http://www.bluishcoder.co.nz/tags/ats/
Vala does not seem to get much attention. It borrows C# syntax (and maybe Java), compiles down to C and offers nearly symmetrical interoperability as a result. I believe Ubuntu's Unity is written in Vala. Knowing Java, it was easy to acquire the syntax from the tutorial [https://live.gnome.org/Vala/Tutorial] in very little time.
For transparent access to compiled C libraries, external C functions need to be declared in a 'vapi' file and although these were already provided for a large part of the GNOME ecosystem, documentation on vapi syntax was nearly non-existent. Object-orientation and standard libraries are integrated with Gnome's GObject system and Glib libraries which I found unfamiliar and usually better documented in C at the time.
If it has GObject as a dependency, then it's not designed for "low-level systems programming". Hell, just having a GC more-or-less discounts it due to the associated problems with runtime size and performance[0]. As a general rule of thumb: if you can't write a kernel in it, it's not a systems language.
[0] It is possible to write a systems language with these features, but low-level systems stuff is not what Vala was designed for.
And yes, a language with which you cannot dload and call any shared library you wish is, how to say, handicapped language.
NodeJS, Java to name a few. Look what they do due to inability to call, say, libpq.so.)
But of course, re-monkey-patching everything from scratch by amateurs coders is much better idea. And, you know, you could create an entire "standard" with hundreds of pages of specification and then push your bloatware - JDBC.
How handicapped is C++ considering it subsumes C, can use C right at the source level, has an inline FFI, and yet C++ libraries can't be called by C because C++ provides type-safe linkage.
Advanced languages are advanced because they improve on C and this inevitably means they must go beyond C. This is why such languages require setting up an environment, usually by providing main().
Even if you could set up an environment with a C function call, which is trivially achieved by replacing main with mymain(), you would simply be starting a foreign environment, not providing a library of functions which could be individually called by C.
My system Felix (http://felix-lang.org) is specifically designed to live with the compromises of C/C++ compatibility, but you still cannot put arbitrary Felix code in C callable functions because Felix provides an advanced environment supporting, for example, fibres with channels. Fibres are scheduled by returning control to a scheduler so they cannot be nested inside a function which puts its return address on the machine stack.
If you Felix write code without using features requiring advanced support that code can be made into a C callable library (both static and dynamic linking is supported).
Felix is probably better at C/C++ integration than any other language except C++.
I believe his thesis was actually that most languages are handicapped because they can not easily be called via the usual "load a .so file and call a function" ABI. Java, for instance, would be a more useful language to me if I could easily create Python bindings for useful Java libraries like Lucene, without needing to use something like Jython that runs on the JVM.
I'm so happy Java can't load libpq you won't believe.
Because it means that Postgresql API gets reimplemented as a crispy JVM code that never crashes my JVM.
If we would live in the world where a Java program would use libpq and libxml and libldap and whatever, it would be so much less stable, introspective and consistent that it won't be Java anymore. It would be a particularly ugly kind of Python.
And yes, it would crash when you use threads. Since all those C libraries pretend that threads don't matter. Guess what, our Java program has 500 threads and feels fine.
True, and so can C call back into go code with cgo. The problem is that it still requires that the main function be in a Rust/Go binary. A program written in C can't currently call into a library written in one of these two languages since the runtimes don't work unless they control the main entry point.
From the Rust doc linked: "The primary motivation of extern functions is to create callbacks for foreign functions that expect to receive function pointers."
If you read what you linked to it says the main purpose is to provide C functions with callback arguments when called from Rust. It isn't clear from what you have written that this mechanism is available to generally access from C programs and scripting languages.
So what's the catch when it comes to D? If I can use all C libs easily, and get some more palpable syntax, what am I missing exactly and why haven't people transitioned to it?
Compatability with the C ABI I feel isn't a necesssity. On platforms built on C it is, but if you wanted to invent a new machine running a new system language with the ABI messes of C exonerated, you could. You would need to devise a translation layer in the OS executor to be able to interpret classic C ABI .so or .dll files for legacy applications and so you could compile C on these platforms, but that isn't a prohibitively hard challenge.
The need for C isn't so much restricted by the operating system, but by the hardware manufacturers. The simple fact is that C is the defacto choice of language for hardware vendors to ship a compiler for, and that doesn't look to change any time. It's a standard that can't be avoided, unless we can work cooperatively with enough major vendors to change this.
Doesn't almost all systems programming take place on platforms built on C? If creating a new machine and new language, convincing the existing systems programmers to move is going to take something very compelling.
If programmers wanted a statically-typed, compiled system language for their scripting efforts, they would be using the existing system languages for that already.
I think Go has been fairly successful at Google in terms of improved code maintainability, and as an alternative to not just C but C++.
It seems to me that they are, in fact, using "statically-typed, compiled system language[s] for their scripting efforts" - that is in fact what people are doing with server-side Java, Scala and C#, not to mention Java/Dalvik and Objective-C on mobile platforms.
Perhaps this article misses the point a bit. For several decades we have been using scripting languages to glue together sharp pointy C[++] tools (sh/Perl/Python/Ruby approach) or to puppetize monolithic C[++] hulks from within (VB/elisp approach).
More recent practice, on both mobile and server platforms, has been to abstract away from C[++] with a middle layer of memory-managed, compiled C-ish goop - JVM, .Net, Objective-C, whatever. This is the level where people are doing everything from amazing high-availability big data things to amazing little bleep-bloop handheld things. I think the answer to "what's the state of low-level systems programming?" is "not much is going on, but damn, there's a lot of fresh stuff happening in mid-level systems programming."
Go and Rust are late entrants to this party, and probably exist more as a hedge against Oracle litigating Java/JVM into the ground than anything else.
>I think Go has been fairly successful at Google in terms of improved code maintainability, and as an alternative to not just C but C++.
Google at large barely uses the language. All the examples they've given are of peripheral stuff, like a load balancer for MySQl for YouTube and an even simpler replacement for an aged and cranky C++ system for Google Downloads. Nothing to write home about, and no replacing C++/Java for core systems at all.
For example, Java and C# cannot walk like a duck, not even if that is what it would take to save themselves from drowning. Therefore, they are absolutely unsuitable as system languages.
Hadoop held the world record for sorting a Terrabyte for a while. If sorting large amounts of data as fast as possible isn't a system's programming task, then I'm a three-headed monkey.
>The Go language and the Rust language. What they have in common, though, is that both Go and Rust are entirely incapable of walking like a duck. Since you cannot call Rust -or Go code from a scripting engine, they are inadvertently positioning themselves as alternatives to the scripting engines themselves.
Before writing all these BS, maybe he should do a little research first?
Here's Rust (which is not even released yet) being capable of being called from C. It's a feature they are actively work on:
> Before writing all these BS, maybe he should do a little research first?
There's a subtle difference between being able to call back into Go/Rust code from C (where the main function is in Go/Rust) which both languages allow now, and allowing a call to originate from a C program, which neither currently do. The author is talking about the later.
I would agree the article could go into more detail about what's currently possible, and why it's not enough. But calling BS, I think, is going a bit too far.
it's probably an interesting read except that being hosted on google's blogspot means it only display blank pages due to content being embedded in some javascript shenanigans.
Even a screenshot would be better than this.
Please move on to some actual html website that just works or rehost the content elsewhere before posting it here. Thanks.
I used to be one of These Guys who never used JavaScript, too, but honestly? It's 2012 and JavaScript is a thing pretty much everywhere worth targeting. Demanding that somebody cater to your microsegment is a little much.
[+] [-] pcwalton|13 years ago|reply
1. M:N scheduling. We wanted high scalability with hundreds of thousands or millions of tasks, which means we need the capability to have our own scheduler. OS threads aren't lightweight enough for this, especially on Windows, where over 80% of our Firefox users are. Having our own scheduler means we need our own ABI. We would like to make the scheduler optional, but getting it working was a higher priority than making it optional.
2. Safety. On many platforms (basically, anything but MSVC on Windows), the C stack is not bounds-checked; either there is no guard page or the guard page is unreliable because you can overshoot it. This is not acceptable for our Rust code, since stack overflows are a frequent source of exploitable security vulnerabilities for us. So we do our own stack management with a custom ABI that performs stack checks at every function entry, freeing us from this source of pain on all operating systems. Again, this is planned to be optional, but it was more important at this early stage for us to demonstrate advantages over C than for us to duplicate its ABI exactly.
Many libraries, however, would prefer direct interoperability with C to M:N scheduling or stack overflow safety. So we plan to provide a mechanism for Rust code to be called from C without having to set up a scheduler or perform stack checks. It's not implemented yet, but it's very much part of the current design, and I would be thrilled to see it happen.
[+] [-] tehwalrus|13 years ago|reply
Now technically you can compile a special python which can handle Go extensions, which is the wrapper mentioned in the second answer, but you can't distribute python modules that ask people to install "this special python" to run (which will reliably work with all their other installed code..)
I would like to see a "go init" function in a C API, that would let you interop the two whichever had the `main()`, since that would let you link with all the other c-implemented scripting languages (python, ruby, etc) which would mean that Go extension support could be added to the main cpython trunk as a compile-time option.
[+] [-] bluehex|13 years ago|reply
Rust issue: https://github.com/mozilla/rust/issues/3608
Go issue: http://code.google.com/p/go/issues/detail?id=2790
I wonder who will get there first.
[+] [-] PuerkitoBio|13 years ago|reply
[+] [-] groovy2shoes|13 years ago|reply
Of all the new languages billed as "systems languages", Clay is the only one that excites me.
It can also "quack like a duck" (to put it in TFA terms).
[1] http://claylabs.com/clay/
[+] [-] ponce|13 years ago|reply
[1] http://nimrod-code.org/
[+] [-] MatthewPhillips|13 years ago|reply
[+] [-] cturner|13 years ago|reply
Later the author says, "If programmers wanted a statically-typed, compiled system language for their scripting efforts, they would be using the existing system languages for that already."
I think the reason that scripting languages exist is because of limitations in the established languages. What we think of as "a scripting language" is defined by the gap. If someone else could be a system programming language without the gap - maybe it doesn't need to be accessed.
The traditional unix platform is (1) C and a compiler for it, with access to the unix standard library and (2) some sh or csh derivative, and access to the likes of sed and awk.
Shell gave people a glimpse of what was possible. But shell has poor access to the unix API, and hits the wall quickly once you do anything other than very basic tool development.
So these things called scripting languages started to pop up. People wanted easy access to the unix API like C. They want easy string manipulation like shell. And they want access to functions, maps and lists without having to think about memory allocation.
If you start from the shell and awk and work into that gap you get perl - the canonical scripting languge.
TCL inhabits a similar space to perl. It's also "a scripting language". But it started at the opposite end. TCL starts as an extension language for C.
So I can understand the author's emphsais on close interaction with C. But I don't think they're important if you can take away the shortfalls they're compensating for.
I hope Go will mean I don't have to think about these things. That I'll be able to get a standard build and a go compiler/runtime and have immediate access to lots of power. No more headaches from C build tools or reaching around in the murky water underneath my scripting language.
[+] [-] jedbrown|13 years ago|reply
[+] [-] justincormack|13 years ago|reply
Now if someone produced a better solution to that it would be useful...
[+] [-] angersock|13 years ago|reply
[+] [-] qznc|13 years ago|reply
[+] [-] zem|13 years ago|reply
[+] [-] doublec|13 years ago|reply
[+] [-] gmt2027|13 years ago|reply
For transparent access to compiled C libraries, external C functions need to be declared in a 'vapi' file and although these were already provided for a large part of the GNOME ecosystem, documentation on vapi syntax was nearly non-existent. Object-orientation and standard libraries are integrated with Gnome's GObject system and Glib libraries which I found unfamiliar and usually better documented in C at the time.
[+] [-] qxcv|13 years ago|reply
[0] It is possible to write a systems language with these features, but low-level systems stuff is not what Vala was designed for.
[+] [-] dschiptsov|13 years ago|reply
And yes, a language with which you cannot dload and call any shared library you wish is, how to say, handicapped language.
NodeJS, Java to name a few. Look what they do due to inability to call, say, libpq.so.)
But of course, re-monkey-patching everything from scratch by amateurs coders is much better idea. And, you know, you could create an entire "standard" with hundreds of pages of specification and then push your bloatware - JDBC.
[+] [-] Yttrill|13 years ago|reply
How handicapped is C++ considering it subsumes C, can use C right at the source level, has an inline FFI, and yet C++ libraries can't be called by C because C++ provides type-safe linkage.
Advanced languages are advanced because they improve on C and this inevitably means they must go beyond C. This is why such languages require setting up an environment, usually by providing main().
Even if you could set up an environment with a C function call, which is trivially achieved by replacing main with mymain(), you would simply be starting a foreign environment, not providing a library of functions which could be individually called by C.
My system Felix (http://felix-lang.org) is specifically designed to live with the compromises of C/C++ compatibility, but you still cannot put arbitrary Felix code in C callable functions because Felix provides an advanced environment supporting, for example, fibres with channels. Fibres are scheduled by returning control to a scheduler so they cannot be nested inside a function which puts its return address on the machine stack.
If you Felix write code without using features requiring advanced support that code can be made into a C callable library (both static and dynamic linking is supported).
Felix is probably better at C/C++ integration than any other language except C++.
[+] [-] pjscott|13 years ago|reply
[+] [-] guard-of-terra|13 years ago|reply
And yes, it would crash when you use threads. Since all those C libraries pretend that threads don't matter. Guess what, our Java program has 500 threads and feels fine.
[+] [-] rpearl|13 years ago|reply
[+] [-] bluehex|13 years ago|reply
From the Rust doc linked: "The primary motivation of extern functions is to create callbacks for foreign functions that expect to receive function pointers."
[+] [-] josephlord|13 years ago|reply
[+] [-] unknown|13 years ago|reply
[deleted]
[+] [-] MatthewPhillips|13 years ago|reply
[+] [-] pretoriusB|13 years ago|reply
That said, the problem with D: not enough marketing plus the two compilers / 2 libraries issue.
And it's close to C++ complexity enough to not let the scripting guys feel welcome (like they do in Go).
[+] [-] zanny|13 years ago|reply
[+] [-] sparkie|13 years ago|reply
[+] [-] keypusher|13 years ago|reply
[+] [-] damian2000|13 years ago|reply
I think Go has been fairly successful at Google in terms of improved code maintainability, and as an alternative to not just C but C++.
[+] [-] slurry|13 years ago|reply
Perhaps this article misses the point a bit. For several decades we have been using scripting languages to glue together sharp pointy C[++] tools (sh/Perl/Python/Ruby approach) or to puppetize monolithic C[++] hulks from within (VB/elisp approach).
More recent practice, on both mobile and server platforms, has been to abstract away from C[++] with a middle layer of memory-managed, compiled C-ish goop - JVM, .Net, Objective-C, whatever. This is the level where people are doing everything from amazing high-availability big data things to amazing little bleep-bloop handheld things. I think the answer to "what's the state of low-level systems programming?" is "not much is going on, but damn, there's a lot of fresh stuff happening in mid-level systems programming."
Go and Rust are late entrants to this party, and probably exist more as a hedge against Oracle litigating Java/JVM into the ground than anything else.
[+] [-] pretoriusB|13 years ago|reply
Google at large barely uses the language. All the examples they've given are of peripheral stuff, like a load balancer for MySQl for YouTube and an even simpler replacement for an aged and cranky C++ system for Google Downloads. Nothing to write home about, and no replacing C++/Java for core systems at all.
[+] [-] lispertoascheme|13 years ago|reply
[+] [-] martincmartin|13 years ago|reply
Hadoop held the world record for sorting a Terrabyte for a while. If sorting large amounts of data as fast as possible isn't a system's programming task, then I'm a three-headed monkey.
[+] [-] eswangren|13 years ago|reply
[deleted]
[+] [-] pretoriusB|13 years ago|reply
Before writing all these BS, maybe he should do a little research first?
Here's Rust (which is not even released yet) being capable of being called from C. It's a feature they are actively work on:
https://github.com/mozilla/rust/issues/1732
[+] [-] bluehex|13 years ago|reply
There's a subtle difference between being able to call back into Go/Rust code from C (where the main function is in Go/Rust) which both languages allow now, and allowing a call to originate from a C program, which neither currently do. The author is talking about the later.
I would agree the article could go into more detail about what's currently possible, and why it's not enough. But calling BS, I think, is going a bit too far.
[+] [-] frujka|13 years ago|reply
Even a screenshot would be better than this.
Please move on to some actual html website that just works or rehost the content elsewhere before posting it here. Thanks.
[+] [-] eropple|13 years ago|reply
[+] [-] nnq|13 years ago|reply
Google's blogspot allows you to do normal html blogs (maybe now they've made this their "default", but still, it's OP's choice to have it this way)
[+] [-] sublimit|13 years ago|reply