Systems Programming at Twitter

[halayli]

This is more about Scala than systems programming.

[codewright]

For that matter, it's an incredibly anachronistic and inaccurate usage of systems programming. People generally reserve systems programming to problems that in order to be solved require something closer to gluing toothpicks than spraying memory.

[rudasn]

On topic then, this is the first time I see Scala syntax and was somewhat pleasantly surprised (the only languages I really know are JavaScript and Python).

On slide http://monkey.org/~marius/talks/twittersystems/#16:

    case class Stock(ticker: String, price: Double) { 
      def <(other: Stock) = price < other.price
    }

    val goog = Stock("GOOG", 675.15)
    val aapl = Stock("AAPL", 604.00)

    aapl < goog == true
    goog < aapl == false

Correct me if I'm wrong but it seems to me that the class Stock gets a method called "<". So appl.< google (or the other way around) would also work. The previous slide also shows this, but the Stock example is makes it even clearer that "Scala is a pure object oriented language: every value is an object." appl and goog are objects, of type Stock, and < is an object, a method of Stock.

I wonder if this kind of method would work

    def ==(other: Stock) = price != other.price

and what would happen if it was like this

    def ==(other: Stock) = self != other

(where self would be a reference to the Stock instance, ala python, or this in javascript)

and it was invoked like this

    appl == goog

and return true

[duncans]

Off-topic somewhat, but I appreciate the way the slides used `location.replace('#'+n)` so my browser history wasn't littered with slide navigations, hence the back button brought me back here (but slides are still bookmarkable).

[shin_lao]

Scala is interesting but as others have pointed out, I think it gives a disservice to the presentation to call it "Systems programming at Twitter" .

To me it shows two things:

* The author doesn't know what system programming is;

* Future, promises and generally speaking asynchronous operations are I/O management 101. It doesn't paint your company in a good light to exhibit these techniques as "how we do things".

Am I too pedantic?

[frou_dh]

Bikeshedding what constitutes Systems Programming has been done to death in Go-related discussions.

[mariusaeriksen]

Hello from the author. Indeed the vernacular has changed, and “systems programming” refers also to the distributed systems server software.

Composable futures and promises are still fairly new in use; I would wager our systems are some of the largest using the techniques described therein.

While this talk focused on composable IO and services, there is much more to it. Finagle (https://github.com/twitter/finagle) is where the rubber meets the road. Its interfaces are described in these terms and it makes great use of their power.

Also, as with any talk, it is difficult to convey the message using only the slides.

[pretoriusB]

>The author doesn't know what system programming is

No, it's that systems programming is not what it was. The term now also applies to large scale server side systems, not just drivers and lower level work. See also how Go was presented as a "systems language" for these same uses.

>Future, promises and generally speaking asynchronous operations are I/O management 101. It doesn't paint your company in a good light to exhibit these techniques as "how we do things".

I/O 101? You'd be surprised. Take a look around the industry, and even HN posts, and you'll find out this is far far from the truth.

[codewright]

Do the dispatches from engineering-land from Twitter strike anyone else as:

"Please learn Scala, we can't hire people that already know Scala.

Please?

Learn Scala."

The last presentation of any substance I saw re: Twitter Engineering was their cute BitTorrent based deployment stack.

[DanielRibeiro]

If you take slides 52[1] to 63[2], you can see the emphasis put into run-time diagnosis.

Which is really important for largely distributed systems. Google wrote a little about it when they introduced their global run-time performance tracing tool called Dapper[3] in "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure"[4]"

Modern Internet services are often implemented as complex, large-scale distributed systems. These applications are constructed from collections of software modules that may be developed by different teams, perhaps in different programming languages, and could span many thousands of machines across multiple physical facili- ties. Tools that aid in understanding system behavior and reasoning about performance issues are invaluable in such an environment.

[1] http://monkey.org/~marius/talks/twittersystems/#52

[2] http://monkey.org/~marius/talks/twittersystems/#63

[3] http://highscalability.com/blog/2010/4/27/paper-dapper-googl...

[4] http://research.google.com/pubs/pub36356.html

[smegel]

Yawn. Headline should have been "Scala tutorial".

[abhijat]

It appears to have an error on slide 23, where the flatMap should return 1, -1 at the beginning rather than 1, -2?

I don't know enough Scala to say with certainty but that seems to be an error:

Seq(1,2,3,4) flatMap { x => Seq(x, -x) } == Seq(1,-2,2,-2,3,-3,4,-4)

[domlebo70]

Yeah you're right.

[fkaminski]

I agree this is more about scala than system programming..

I was very happy back then when scala shows up.. but with time i think computer languages should just get out of your way.. be simple as possible..

nowadays im more a Dennis Ritchie/Ken Thompson fan.. they had make simple tools with C and Unix, that just worked..

This is also the Go language school of thought.. Languages must get out of our way, get things done in the simplest way possible..

Functional language people , are more passionate about proving some theoretical points like using immutability all way down.. even if this will blow out your memory resources, or make your code longer..

i think language designers need to be more pratical, and i like when the simplicity way of thought like the one seen in Go comes around the table, to make discussion a little bit more interesting..

We nee more simplicity, work less, so we can be more with our children or family.. and spending less time on our computer.

[mattdw]

When all you have is a flatMap, everything looks like a nested list ;).

(The author seems to be shoe-horning a lot of different concepts into flatMap via traits. I count several different Monads in there.)

[sanderjd]

It is possible that you're making the same mistake you accuse the author of making. "You have a hammer and are seeing everything as a nail, but I have a screwdriver and all those things are actually screws."

[jebblue]

I can't get my head around stuff like a function is a value (I can what it's doing, it's the syntax that boggles me) :

val f: Int => Int = { x => x*2 } f(123) == 246

[mrcrassic]

If you do more functional programming, it will become second-hand to you.

[ChrisArchitect]

err http://news.ycombinator.com/item?id=4732958

[pionar]

OT, but it took me way too long to figure out how to make the slides advance. I'm an idiot :)

[duaneb]

Whatever presentation software is used appears to be nigh unusable on the iPad.

[wtracy]

I don't know if this helps you, but the presentation is quite readable on a browser that has JavaScript completely disabled.

[tzury]

Well,

    aapl < goog == true

Should be false, at least by real market values.

;-)