My idea of system programming is that, other than the "near to the bare metal" element, which may not be always true, has this quality of creating infrastructures for other layers to use. A game 3D engine and a DNS server may be both written in C++ and may use the same low level programming techniques to achieve speed, but the fundamental difference is that one is just part of an application program of some type, and the other a system that provides a general service to other programs. So clearly it's technically possible to do system programming in higher level languages, even if often this will require anyway dealing with many system calls regardless of the language, both explicitly or implicitly via abstractions.
Computer systems are only useful because they can run application programs. "Systems programming," therefore, is one that aims to improve the utility of the computer, as opposed to "application programming" which solves concrete problems posed by users.
> part of an application program of some type, and the other a system that provides a general service to other programs
I find that distinction to be a bit arbituary. To me, systems programming is one which require adherence to some sort of well defined spec/standard so as to interoperate with some (potentially underlying) "system".
In a 3D engine, the systems programming part is the part that interfaces with the graphics drivers, but not the part that derives the 3D geometry from reading files or running logic to generate the content.
In a DNS, the systems programming part is the socket interface, but not the DNS protocol (tho you could argue that the protocol is also considered "system-ish").
In a web application, the systems programming is also the socket, in a very similar vein to DNS servers.
In a desktop application, the systems programming is the interface into any native resources (such as any socket calls, any disk, or peripherals, as well as any of the graphical hardware interfacing required).
But because many of these tasks are quite common, they get put into libraries, and so many application programmers do not deal with these "systems" part, and just deal with their own application logic, and so it feels like there's no systems programming.
I think systems programming should be split in two, honestly.
You have the kernel-level systems programming where you need to be near the bare metal, where you write kernels or program microcontrollers with less RAM than a x86 CPU has L1 cache, where taking a microsecond longer can mean the overall system crashing (or even costing a human life).
And you have system-level systems programming where you write services and supportive infrastructure to enable user programs, where garbage collection and non-realtime behaviour is acceptable, where you can skip a microsecond and it'll be alright.
Great article. This is an important distinction to make. I've noticed that the more experienced I got, I felt less and less need to micro-optimize every aspect of a program at the expense of code readability. That's where I think the design and implementation of a system intersect; making it easy to understand how it works and giving future maintainers the confidence to make changes is best accomplished with clear and readable code with the right abstractions (even if it hurts performance a little). I think that documentation will always be second to that.
Depends on the kind of software, though. If you're writing an interpreter/compiler, web server, database or operating system you don't have much choice.
Andrei didn't create D; he's been influential in it, especially w.r.t. D's metaprogramming story, but the language was created and is primarily maintained by Walter Bright.
> You should be able to forge a number into a pointer, since that’s how hardware works.
It's a C-centric view of the world and I wonder what old school FORTRAN77 people or lispers would think of this.
Anyway, I think the right to do this should be a privilege of the compiler. As soon as you claim this right, you abandon all possible support she can give you in battling all kinds of silly mistakes.
I'm not an "old-school lisper", but I have done some Lisp development related to embedded systems. The phrasing may be somewhat unfortunate but it ultimately does describe how hardware works: you need to write something at a numerical address that you find in the reference guide or the datasheet. At the end of the line, you do have to forge a number into a pointer. In James Micken's words, "You can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis."
"Higher-level" languages (like Lisp) can help you in other regards. They can include proper primitives (or give you the proper tools to write the proper primitives yourself) to ensure that direct-to-hardware access is safe, can give you better tools to manipulate DMA buffers (e.g. proper support for coroutines when manipulating double-buffered data is cool), can help you write more generic state-applying and state-reading code (i.e. code for "turn these bytes from this circular buffer into this structure" or "take this structure that represents a peripheral's current config and use it to write the proper magic numbers in the right registers"). They can offer you better semantics for translating human-readable(-ish) configuration into the right stream of bytes, or manipulate peripherals in real-time. They can help you write a better VM (or their runtime can outright be the better VM you need).
But at the end of the day, writing to the config space of a memory-mapped device is still going to consist of taking a number (or building it from several numbers) and writing bytes at that address, no matter what language you're using. Better runtime support for safe operation under this scenario, better semantics -- they're all important, but what the author expresses is not C-centric in any way.
Pointers are not even necessary for writing operating systems or memory allocators. The Oberon system writes those low-level components using intrinsic peek/poke functions (like some ancient BASIC!) that are recognised specially by the compiler and turned into direct memory modifications. This is clearly just as unsafe as pointers (probably more so), but it means you don't generally pollute the language to support some very specialised code. The rest of Oberon is memory-safe and garbage-collected (with the garbage collector also written in Oberon).
As someone who likes Fortran90+ for numerics in HPC, I'd put it this way:
* Pointers are required mainly so you can do pointer swaps and be sure there is no unnecessary allocation (in simulations this often means being sure there's no 10-100GB allocation each timestep, which is where it really matters).
* Pointer math is unnecessary and only there because C has no/poor support of multidimensional arrays. Use a reasonable language like Julia or Fortran instead. In fact, the potential for pointer math has a big negative impact on performance since compilers have to assume aliasing. The __restrict solution is clunky in C and completely non-standard in C++, Fortran does not require this at all (no aliasing allowed except where explicitly stated).
* For almost all usecases (except swaps) a semi-managed approach like Fortran's allocatables or Objective-C's ARC or even retain/release seems to me the most straight-forward.
I am not sure what you mean, it's not like we cast integers to pointers by hand is it? The compiler does the casting. And sure, there is usually no support from the compiler when you are programming any device other than the CPU but these devices need to be programmed. For you to read this message somebody had to program the display device and for you to even be able to access this web site somebody had to program a whole bunch of things starting from your NIC, going through various routers and to the different NIC on the HN's server. No compiler had been able to do this so far.
Every operating system I’ve worked on has had a tiny few lines of relatively isolated code for this sort of thing. Even some crappy RTOSes, it seems it is natural to want to abstract it. You need to forge a number into a physical address pointer, not just a pointer. Look at some Linux drivers, they will have some offsets defined but they don’t just create a void* and assign it to a memory location, they use macros and helper functions to fix things up. They can even check if the address is in reserved io spaces and do things like that. There are also helpers to write the bytes to those addresses. It’s a tiny amount of code that is usually already abstracted.
I don’t know. Seems like a lot of kernels do extra things so driver writers can write fairly normal looking C and produce fairly safe drivers without just arbitrarily assigning pointers addresses from numbers. And even if you use C as your comparison, those helpers will often be written with some assembly language (read C isn’t completely good enough for the task all by itself) I don’t know of it existing but a kernel in go with ref counting GC and a handful of cgo primitives seems very possible and maybe even delightful to work on.
The author's point that functional programming principles are important to systems programming and should be taught alongside it is well taken. I think he confuses the issue somewhat with all the discussion of what is a "systems programming language" as these days it's just used to refer to languages substitutable for C.
System programming is definitely a bit overloaded these days though I believe it is now widely understood to be languages like C/C++ that can be used for OS, kernel, and embedded development.
I would agree though that the language ecosystem is shifting a lot in the last few years. IMHO there are now a few languages that are becoming proper full stack languages in the sense that they scale from embedded all the way to browser development. Rust is a great example.
I'm really impressed with what the Rust community has done in just a few short years. All my life, C/C++ was the only game in town for the stuff that is now done with Rust with arguably the same or in some cases slightly better performance.
Stuff like WASM allows the use of system programming languages in places where they would not be used in the past. Using Rust transpiled to WASM makes it a proper fullstack language. And people are using it for that. And people do server development as well in it as well as OS development.
> IMHO there are now a few languages that are becoming proper full stack languages in the sense that they scale from embedded all the way to browser development.
I think you're being a bit optimistic here: traditionally, the only language that has been able to do this is C++, and Rust is displacing it simply because Rust really tries to target C++ developers and their pains. But other than that I don't see much else development here. Swift has promise, but unfortunately nobody's really writing much framework or foundational code with it; at least not yet.
Rust is nice but I wouldn't call it full stack because I don't think you should choose it in situations where you can afford a garbage collector. In those cases there are much more ergonomic options available.
Its more what you do with the language than the language its self.
eg when I extended the GINO-F drivers for the top of the line HP plotter that's systems programming but the GUI system I built to analyse soil samples was not - even though they where both written in the same language
Irrespective of language and framework, if the code needs to be aware of underlying hardware then it is systems programming. ex. kernel code, user-space drivers, databases, portions of cloud or server code which need to be hardware aware, etc. What golang targets, I think, should be more of middleware or service-level programming, whether its containers or servers.
I'm curious where Erlang fits into this story. It was built to be a "systems" language but algorithms are tersely implemented (I forget exactly, but I remember reading about a study that an Erlang program is much shorter than the comparable C program), and has gradual typing of a sort. It also is garbage collected (I don't think you could "write your own memory allocator in it", as Andrei Alexandrescu described).
My view of Systems Programming will forever be colored by my start with IBM mainframes. The Systems Programmer maintained all the utilities that the mainframe provided, and also the frameworks that supported online programming.
I suppose today's equivalent is a combination of a sysadmin and an applications developer, providing the applications are things like container components, build pipelines, etc.
A systems programming language can run on bare hardware by itself, or nearly so. It is acceptable to require a very small amount of assembly code, for example to implement something like memcpy or bcopy, or to provide atomic operations.
A language is disqualified if it requires an OS or if it requires code written in a different non-assembly language. Cheating, by adding that as a huge (impractical) amount of assembly, also disqualifies a language.
Funny story about gcc: The compiler demands a memcpy, which it sometimes uses for struct assignment. If you implement memcpy in the normal way, gcc will helpfully replace your implementation with a call to memcpy! Despite this, C is still a systems programming language.
> It is acceptable to require a very small amount of assembly code, for example to implement something like memcpy or bcopy, or to provide atomic operations.
There is some stuff missing in your list: preparing the stack pointer, address layout, etc. You also need tools to produce text and data sections that can be loaded at a specific address. Even if C is a low level language, there is still quite some stuff between it and the "bare hardware". In that sense, the only language that gives you enough control to produce the binary exactly in the form needed by the hardware (without relying on external tools or libraries) is assembly.
I don't think a single definition makes any sense now: decades ago you may have had a definition of a 'computer system' that describes every big program. Nowadays there are many sets of requirements depending on context, each set of requirements implies its own constraints on memory management and scheduling so that there is no one description of how a complex program would look like.
> Companies like Dropbox were able to build surprisingly large and scalable systems on just Python.
Many of Dropbox’s services are written in Go, and Magic Pocket is written (partly, at least) in Rust. Earlier in their development, Dropbox relied on S3, which is obviously not in Python.
Fundamentally, I think the important part of the “production” aspect of “systems programming” is that it’ll be used a lot. That’s what drives the requirement for efficiency: if you’re 10-100x less efficient, someone else will be more cost effective / solve bigger problems.
As an additional example, GitHub was written in Ruby, but that’s fine because the underlying Git and filesystem manipulation is all in C. The same thing is sort of true of Facebook’s world of PHP: PHP is mostly a wrapper around C libraries. Until it got complex enough that they rewrote the language.
tl;dr: I don’t think systems programming needs to be “low level”, but production systems do need to be efficient. More powerful computing just moves that further to the right!
In old days, computers that came from manufacturers like IBM, DCE was refrred to as system. The operating software for the system was referred to as “system software”. People who created that software were called “systems programmer”. Much of it invariably involved writing direct to metal/assembly code which is why systems programming has been associated with low level programming. People using this term to refer to task of doing some complex application architecture are doing it wrong.
I think that it is not useful to build on Ousterhout's definition as his definition of systems programming language is mostly meant as contrast to Tcl with it's everything is a string memory model (in that view one could very well conclude that Python is a systems programming language).
when I was programming in the 70s and 80s (in the UK), the term systems programmer applied to those people involved in the configuration, tuning, management of software such as CICS, MVS, and their equivalents on other hardware bases. I worked at Sperry Univac for several years and the "systems programmers" were the engineers who tweaked the OS, the TP schedulers, built the product installation scripts, designed the configs that controlled the software. It was a specific tradecraft suited to those who were more technically focused, remember that COBOL was the prevalent language for nearly all programming roles in the 70s and that there was no "web" per se. The term started to fade away roughly when PCs started entering into mainstream computing
I think the author makes fair points. My conclusion: Do the bit twiddling and strongly constrained parts in a low level language and the rest in a language that supports strong static typing and good interfaces.
Maybe something like Android, OS X/iOS, ChromeOS, UWP, Fuchsia, Unikernels, Language runtimes on Hypervisors.
And since I sense the question coming, Android, OS X/iOS, ChromeOS might use an UNIXy kernel, but its exposure surface to applications is so small, that it can be easily exchange for something else, it is just a matter of cost.
Which brings us to the next point, given the commodity of free UNIX-like OSes, maybe only an hardware reboot like Quantum computers will actually do it.
[+] [-] antirez|7 years ago|reply
[+] [-] Koshkin|7 years ago|reply
[+] [-] chii|7 years ago|reply
I find that distinction to be a bit arbituary. To me, systems programming is one which require adherence to some sort of well defined spec/standard so as to interoperate with some (potentially underlying) "system".
In a 3D engine, the systems programming part is the part that interfaces with the graphics drivers, but not the part that derives the 3D geometry from reading files or running logic to generate the content.
In a DNS, the systems programming part is the socket interface, but not the DNS protocol (tho you could argue that the protocol is also considered "system-ish").
In a web application, the systems programming is also the socket, in a very similar vein to DNS servers.
In a desktop application, the systems programming is the interface into any native resources (such as any socket calls, any disk, or peripherals, as well as any of the graphical hardware interfacing required).
But because many of these tasks are quite common, they get put into libraries, and so many application programmers do not deal with these "systems" part, and just deal with their own application logic, and so it feels like there's no systems programming.
[+] [-] zaarn|7 years ago|reply
You have the kernel-level systems programming where you need to be near the bare metal, where you write kernels or program microcontrollers with less RAM than a x86 CPU has L1 cache, where taking a microsecond longer can mean the overall system crashing (or even costing a human life).
And you have system-level systems programming where you write services and supportive infrastructure to enable user programs, where garbage collection and non-realtime behaviour is acceptable, where you can skip a microsecond and it'll be alright.
[+] [-] thegeomaster|7 years ago|reply
[+] [-] codr4|7 years ago|reply
[+] [-] mrec|7 years ago|reply
Andrei didn't create D; he's been influential in it, especially w.r.t. D's metaprogramming story, but the language was created and is primarily maintained by Walter Bright.
[+] [-] wcrichton|7 years ago|reply
[+] [-] mhh__|7 years ago|reply
[+] [-] bausshf|7 years ago|reply
[+] [-] toolslive|7 years ago|reply
It's a C-centric view of the world and I wonder what old school FORTRAN77 people or lispers would think of this.
Anyway, I think the right to do this should be a privilege of the compiler. As soon as you claim this right, you abandon all possible support she can give you in battling all kinds of silly mistakes.
[+] [-] alxlaz|7 years ago|reply
"Higher-level" languages (like Lisp) can help you in other regards. They can include proper primitives (or give you the proper tools to write the proper primitives yourself) to ensure that direct-to-hardware access is safe, can give you better tools to manipulate DMA buffers (e.g. proper support for coroutines when manipulating double-buffered data is cool), can help you write more generic state-applying and state-reading code (i.e. code for "turn these bytes from this circular buffer into this structure" or "take this structure that represents a peripheral's current config and use it to write the proper magic numbers in the right registers"). They can offer you better semantics for translating human-readable(-ish) configuration into the right stream of bytes, or manipulate peripherals in real-time. They can help you write a better VM (or their runtime can outright be the better VM you need).
But at the end of the day, writing to the config space of a memory-mapped device is still going to consist of taking a number (or building it from several numbers) and writing bytes at that address, no matter what language you're using. Better runtime support for safe operation under this scenario, better semantics -- they're all important, but what the author expresses is not C-centric in any way.
[+] [-] Athas|7 years ago|reply
[+] [-] m_mueller|7 years ago|reply
* Pointers are required mainly so you can do pointer swaps and be sure there is no unnecessary allocation (in simulations this often means being sure there's no 10-100GB allocation each timestep, which is where it really matters).
* Pointer math is unnecessary and only there because C has no/poor support of multidimensional arrays. Use a reasonable language like Julia or Fortran instead. In fact, the potential for pointer math has a big negative impact on performance since compilers have to assume aliasing. The __restrict solution is clunky in C and completely non-standard in C++, Fortran does not require this at all (no aliasing allowed except where explicitly stated).
* For almost all usecases (except swaps) a semi-managed approach like Fortran's allocatables or Objective-C's ARC or even retain/release seems to me the most straight-forward.
[+] [-] pandaman|7 years ago|reply
[+] [-] TheCondor|7 years ago|reply
I don’t know. Seems like a lot of kernels do extra things so driver writers can write fairly normal looking C and produce fairly safe drivers without just arbitrarily assigning pointers addresses from numbers. And even if you use C as your comparison, those helpers will often be written with some assembly language (read C isn’t completely good enough for the task all by itself) I don’t know of it existing but a kernel in go with ref counting GC and a handful of cgo primitives seems very possible and maybe even delightful to work on.
[+] [-] dleavitt|7 years ago|reply
[+] [-] jillesvangurp|7 years ago|reply
I would agree though that the language ecosystem is shifting a lot in the last few years. IMHO there are now a few languages that are becoming proper full stack languages in the sense that they scale from embedded all the way to browser development. Rust is a great example.
I'm really impressed with what the Rust community has done in just a few short years. All my life, C/C++ was the only game in town for the stuff that is now done with Rust with arguably the same or in some cases slightly better performance.
Stuff like WASM allows the use of system programming languages in places where they would not be used in the past. Using Rust transpiled to WASM makes it a proper fullstack language. And people are using it for that. And people do server development as well in it as well as OS development.
[+] [-] saagarjha|7 years ago|reply
I think you're being a bit optimistic here: traditionally, the only language that has been able to do this is C++, and Rust is displacing it simply because Rust really tries to target C++ developers and their pains. But other than that I don't see much else development here. Swift has promise, but unfortunately nobody's really writing much framework or foundational code with it; at least not yet.
[+] [-] guelo|7 years ago|reply
[+] [-] walshemj|7 years ago|reply
eg when I extended the GINO-F drivers for the top of the line HP plotter that's systems programming but the GUI system I built to analyse soil samples was not - even though they where both written in the same language
[+] [-] n_t|7 years ago|reply
[+] [-] steveklabnik|7 years ago|reply
At first I thought it was poking fun; now I see it as a comment about what is relevant at a given level of abstraction.
[+] [-] jkingsbery|7 years ago|reply
[+] [-] RickJWagner|7 years ago|reply
My view of Systems Programming will forever be colored by my start with IBM mainframes. The Systems Programmer maintained all the utilities that the mainframe provided, and also the frameworks that supported online programming.
I suppose today's equivalent is a combination of a sysadmin and an applications developer, providing the applications are things like container components, build pipelines, etc.
[+] [-] vram11|7 years ago|reply
[+] [-] burfog|7 years ago|reply
A language is disqualified if it requires an OS or if it requires code written in a different non-assembly language. Cheating, by adding that as a huge (impractical) amount of assembly, also disqualifies a language.
Funny story about gcc: The compiler demands a memcpy, which it sometimes uses for struct assignment. If you implement memcpy in the normal way, gcc will helpfully replace your implementation with a call to memcpy! Despite this, C is still a systems programming language.
[+] [-] tralarpa|7 years ago|reply
There is some stuff missing in your list: preparing the stack pointer, address layout, etc. You also need tools to produce text and data sections that can be loaded at a specific address. Even if C is a low level language, there is still quite some stuff between it and the "bare hardware". In that sense, the only language that gives you enough control to produce the binary exactly in the form needed by the hardware (without relying on external tools or libraries) is assembly.
[+] [-] pjmlp|7 years ago|reply
[+] [-] MichaelMoser123|7 years ago|reply
[+] [-] boulos|7 years ago|reply
But I disagree with this:
> Companies like Dropbox were able to build surprisingly large and scalable systems on just Python.
Many of Dropbox’s services are written in Go, and Magic Pocket is written (partly, at least) in Rust. Earlier in their development, Dropbox relied on S3, which is obviously not in Python.
Fundamentally, I think the important part of the “production” aspect of “systems programming” is that it’ll be used a lot. That’s what drives the requirement for efficiency: if you’re 10-100x less efficient, someone else will be more cost effective / solve bigger problems.
As an additional example, GitHub was written in Ruby, but that’s fine because the underlying Git and filesystem manipulation is all in C. The same thing is sort of true of Facebook’s world of PHP: PHP is mostly a wrapper around C libraries. Until it got complex enough that they rewrote the language.
tl;dr: I don’t think systems programming needs to be “low level”, but production systems do need to be efficient. More powerful computing just moves that further to the right!
[+] [-] sytelus|7 years ago|reply
[+] [-] dfox|7 years ago|reply
[+] [-] theoldgit|7 years ago|reply
[+] [-] kmooney|7 years ago|reply
[+] [-] tempodox|7 years ago|reply
[+] [-] gjvc|7 years ago|reply
[+] [-] pjmlp|7 years ago|reply
And since I sense the question coming, Android, OS X/iOS, ChromeOS might use an UNIXy kernel, but its exposure surface to applications is so small, that it can be easily exchange for something else, it is just a matter of cost.
Which brings us to the next point, given the commodity of free UNIX-like OSes, maybe only an hardware reboot like Quantum computers will actually do it.
[+] [-] tanilama|7 years ago|reply
[+] [-] matachuan|7 years ago|reply