So HN, what are some cool, shiny new technologies that you worked with this past year? Care to tell us what those technologies are and why they are so cool?
2010 was, for me, the year of JS-related technologies. (I'm actually rather disappointed I haven't had more time to check out Clojure and to use Haskell and Scala more--I was doing quite a lot of front-end web stuff.)
It lets you use websockets and automatically fall back to flash sockets, long polling, or several other real-time communication methods if websockets aren't supported by the client. There's a JS client and node-compatible server, as well as in-progress server implementations in a few other languages. Node is nice by itself, but it's with things like Socket.IO that it really shines.
Coffeescript is a nice-looking and nice-to-type syntax on top of JavaScript. It's made JS development a lot friendlier, and I now miss things about it every time I'm programming in Python and Ruby. I now use it whenever I'm doing any significant amount of coding in JS.
Node should, by this point, need no introduction. Server-side JS. Plays very nicely with websockets thanks to Socket.IO, making it very easy to write the server-side part of real-time webapps. I've also found it very useful when trying to quickly prototype simple non-webapp things that have to communicate over a network.
I haven't had a chance to check out Backbone.js (http://documentcloud.github.com/backbone/) yet beyond a very quick look, but I expect to use it (or something like it) next time I'm developing something that uses a significant amount of client-side JS.
I'm also very excited by the continued development on (and Yehuda Katz's participation in) SproutCore (http://www.sproutcore.com/).
I'm looking forward to seeing someone combine Backbone.js with Websocket-based persistence. It's a bit out of scope for DocumentCloud to tackle, but the building blocks are there.
One client makes a change to a model -> changes are synced to the server -> other clients that currently reference that model have its attributes updated -> all the UI that displays the model is automatically re-rendered.
It's not critical for most applications, but is a really nice, nice-to-have for any JS app that displays the same editable data to more than one user at a time. Built-in conflict resolution mechanism for bonus points.
Varnish, the reverse proxy, has been my favorite new-to-me technology of 2010. It sits in front of Apache and caches static content (or anything really) based on rules you define in the Varnish Configuration Language (VCL).
Varnish is cool because it is very fast. It was written by Poul-Henning Kamp, who has a lot of experience in FreeBSD kernel development. He makes effective use of virtual memory, is careful to avoid memory operations that result in expensive bus transactions on mutli CPU systems, and knows how many system calls it takes to serve up a cache hit. All of this work has paid off. Varnish can turn a plodding CMS into a site that screams, and your profiling tools (siege, apache ab) will fall over before the site does.
Of course it helps if your CMS supports cache control headers, and isn't utterly laden with cookies, but that's where the VCL language comes in. You can write code to strip bogus cookies (like google analytics) coming from both the client and the server which vastly improves your cache hit ratios.
I like the way Varnish uses a shared memory pool for statistics and logging -- a wealth of information about the system is available to you but it doesn't generate a ton of I/O logging it to a file unless you ask it to. I love how you can use the telnet admin interface to compile new VCL code into a running system and then switch to it, while keeping old named configurations around in case you need to revert back.
Varnish has really helped me make slow sites fast this year, although it hasn't happened without some VCL coding effort and some understanding of how the sites operate.
But despite all that description, all my deployments have been one-liners with 3-4 cmdline params only. It really adds near-zero deployment cost.
Squid can do what varnish does, but it's 2x slower by my measurements, and it has a gigantic config file. (Squid can do things that varnish can't, so you may have to use it anyway.)
The iPad has changed everything. It's an entirely new type of computer that turned out to be substantially better than anyone imagined. Watch a 5-year-old use an iPad for the first time and you will immediately see and understand why this is a major paradigm shift. It's the first "socially acceptable" computer -- at Christmas I can pull out my iPad, plop it down at the dinner table and share pictures with the family, and it's not at all considered rude.
Redis. Fucking awesome database. Does exactly what it's advertised to do, with no unexpected surprises. Great documentation. Finally we can go beyond the simplistic key-value map/reduce datastores, for when you don't need all the guarantees that traditional SQL forces you to have.
Yes! I first laid hands on Redis this year while working on Notifo and it's raw speed lets me do some great things. Currently hacking on some service analytics stuff that redis is making pretty easy. Though keeping everything in memory can get expensive when we want to minimize server costs so we keep the first month of data in redis and the rest as an archive in mysql (how often is someone going to look up a notification they received more than a month ago?)
These aren't new to the world, but they are new to me this year, and a lot of them sort of hit some sort of "usable by those without active interest in the continuation of said tech".
1. Mongodb -- This year it really hit its stride and have been able to use it without worry for storing test results and experimental data. This is much nicer than the textfile logs -> sql -> processing datapath I was using previously.
2. flask: this little framwork is in my sweetspot. It does all the annoying crap of webby stuff, without all the "use our orm/routing model/way of thinking of http" so common in the space
3. mongrel2: I like it because it uses 0mq as the backend and sanely integrates some components in a way I feel could be better for many use cases than traditional stacks.
4. 0mq: This gets special mention, because it has been around for a while and I was actively using it, but 0mq 2 came out sometime this year, and is different enough from the first round, that it could be considered a separate technology. It isn't revolutionary in the MOM space, but it is a cool lightweight approach, and the core team has the type of dedication I like to see in OSS projects.
5. ABSOLUTELY NOT NEW: Haskell -- this year is the first year I've had time to sit down with Haskell for real, and start understanding the weird FP thing. This has made everything I do feel shiny and new, because even though I never actually use Haskell, I find myself writing very short hsskell programs in python and c and the other languages I use in my day job. When I started coding I remember thinking "This must be what a wizard feels like!", Haskell has brought back that feeling for me.
I continued to work with F#, deploying a couple of small apps.
The really cool part came when I realized that with F# I was programming at the language level -- that I could effectively and easily write my own languages. So I decided I would like rails-like entities, where the entity reads the structure of the table and then conforms itself to whatever is in the table.
Couple hundreds of lines of code later, and presto chango, I could simply say "give me a list of customers" and point it to the table and I had a list of customers. This totally disconnected the database data structure from the code. Add a new field in the database and there was nothing to change in code. Or add a new field in the type and have it percolate out to the database. Change database providers and it was only a few function changes. Very cool. The kind of simple fix Microsoft should have done with data access instead of writing ODBC/ADO/OleDb//EF/etc
Then I had a blast with mailboxes, er monads, agents, and threads. Ended up writing a small app that was purely functional and all ran in the background. It was so automatic, at first I couldn't figure out how to start the dang thing!
This led to a venture into MPI and other technologies which has just begun. I'm also trying to wrap up my language work with a full DSL sometime soon (if I have a project that needs it). Looking forward to parsing and setting up trees and walking them. I also broke out of windows and started working in a linux environment using Mono, Apache, and MySQL.
Incredibly fun stuff. Looking back, I really had a blast this past year. Next year should be even better.
I am a huge fan of Vagrant ( http://vagrantup.com ) which is virtualized development environments, package-able. Works with chef and virtualbox. I don't know if I can state just how game changing this is for me.
DocumentCloud really dropped some bombs this year. backbone.js, underscore.js are really great.
Socket.io saved my ass. I promised some big clients that I could make websockets driven apps for the iPad and then apple pulled websockets support without saying anything. So I was able to get socket.io for the win.
1. MongoDB / Mongoid have blown me away this year. Is now my default database for new rails projects.
2. CoffeeScript language is an amazing replacement for Javascript. I can't see myself going back to pure JS at any point in the future.
3. Rails 3 finally feels like a stable and maintainable web framework. All the web frameworks now all seem to work together using Rack. The ruby web development world is really a nice place to be at the moment.
4. Bundler really nailed the gem dependency management issue (though the journey to 1.0 was very painful)
5. Sass / SCSS / Compass got really good. It feels unimaginable to go back to regular CSS.
Wow, now that I think about it, way too many great technologies to list. 2010 was an insanely good year for ruby web developers.
Great list. With the exception of Coffeescript (I just don't "get it"), that's basically my list, too. I'd like to branch out for this next year, though. Clojure and Erlang are both on my list.
Clojure - it's the gift that keeps on giving. It keeps getting faster and the feature set for writing robust object-oriented software (minus the broken stuff) just keeps getting better. In fact, it's changed how I assess the feature set of other OO languages old and new.
miniKanren - logic programming w/o the Warren Abstract Machine. Has opened my eyes to a ton of incredible literature on this under appreciated programming paradigm.
First touched it two months ago just to tinker with but didn't really do anything with it. Then after numerous frustrations with my current blogging setup, I spent the last 5 days hacking on it over the holidays and I think it's almost ready to launch. Had to do some custom stuff that I'll write about in a post. It's extremely hackable and I love it. The only thing that doesn't work for me is LSI for related posts. Even with a fast computer and gsl/rb-gsl it still takes 10+ hours with my 1,000+ posts. Anyways, having a super fast site is going to be a breath of fresh air. Google was saying 88% of sites loaded faster than mine ( http://paulstamatiou.com ), though likely due to the images in many of my reviews.
Also installed Google mod_pagespeed and all is well so far.
* Though to be fair most of that is just my redesign that is more minimal, less ads, etc, but there's something extremely attractive about simple, flat files. No worrying about if your database will get corrupted. Everything is in git..
I love Jekyll. It fits into my workflow perfectly and made it really easy to translate the entire site into Japanese for that market. The whole thing took 3 days.
I'm kind of ashamed to admit it (at the end of 2010!), but I did some Objective-C/iOS apps for the first time this year, and I was pretty amazed by how good the Apple dev tools and the iOS simulator actually are.
The other thing which impressed me is kvm, in contrast to Xen.
Ditto on the Objective-C stuff, though personally I've been focusing on Mac OS X rather than iOS. Apple really documents things well, and the Cocoa community is extremely nice and helpful. The shiny glare of quality extends deep into the inner depths of Apple's products.
This is the first year I've really used the real released Windows 7 - on my new SSD-loaded work laptop and on my personal netbook. It's amazing to see Windows more or less work and do what I want it to do most of the time.
GPU-accelerated VLC on my netbook has been amazing.
I got a Canon T2i / 550D this year. It shoots some amazing HD video and will only get better as I spend more on lenses and develop better techniques.
The Kindle 3 (brighter display and cheap price) have me reading books I've been putting off for years. It's great to have a device that's great at one thing and not very good at random browsing, Facebook, Twitter, HN, etc.
All thanks to jashkenas: CoffeeScript, Underscore.js, and Backbone.js.
Using those tools has helped me to really enjoy writing JavaScript and to start doing it in a much more organized manner. They have been a huge catalyst for my growth as a JavaScript developer.
I was shocked by how powerful SproutCore was, once I actually started hacking on it. I suspect that it will be a very big deal in 2011.
I am also really excited by socket protocol tech advancements in the browser. I was able to pull off seriously cool stuff using http://pusherapp.com/ and also http://faye.jcoglan.com/ which is a nifty JS implementation of the Bayeux protocol.
1. Groovy - a programming language, it rocks because it less verbose and more powerful than Java and I can fall back onto standard Java syntax when I don't care to figure out how to do something in the 'Groovy' way. (http://groovy.codehaus.org)
2. Gaelyk - a groovy framework that runs on Google App Engine. Google App Engine is great for launching apps. It's free until it gets traction. (http://gaelyk.appspot.com)
3. Objectify - The simplest convenient interface to the Google App Engine datastore. Takes a lot of the pain out of using Bigtable. (http://code.google.com/p/objectify-appengine)
These all pack a mean punch and let me work on my night/weekend projects quite productively after overcoming a small learning curve.
1) MongoDB - I started off using it in place of a few tables that had some varied column requirements, and I'm now in the middle of converting my entire DB to it. So awesome.
2) Kohana - I love working with this framework. I never really worry about the ugly warts in PHP, because honestly? I'm not programming in PHP any more. I'm programming in Kohana, and I only occasionally fall back to PHP for "low-level" stuff.
Not what you would call cool technology, but definitely the best technology I have used this year. Protobufs get out of the way so you can get work done.
Not new, but I've really enjoyed working with Lua (http://lua.org). The C API is really nice and I like how you can start from a known-safe, minimal interpreter and add new procedures carefully.
libev (http://software.schmorp.de/pkg/libev.html) was also a lot of fun to use for multiplexing sockets, plus it has a whole pile of other useful watchers that can use its event loop.
[+] [-] samdk|15 years ago|reply
1. Socket.IO (http://socket.io/)
It lets you use websockets and automatically fall back to flash sockets, long polling, or several other real-time communication methods if websockets aren't supported by the client. There's a JS client and node-compatible server, as well as in-progress server implementations in a few other languages. Node is nice by itself, but it's with things like Socket.IO that it really shines.
2. Coffeescript (http://jashkenas.github.com/coffee-script/)
Coffeescript is a nice-looking and nice-to-type syntax on top of JavaScript. It's made JS development a lot friendlier, and I now miss things about it every time I'm programming in Python and Ruby. I now use it whenever I'm doing any significant amount of coding in JS.
3. Node.js (http://nodejs.org/)
Node should, by this point, need no introduction. Server-side JS. Plays very nicely with websockets thanks to Socket.IO, making it very easy to write the server-side part of real-time webapps. I've also found it very useful when trying to quickly prototype simple non-webapp things that have to communicate over a network.
I haven't had a chance to check out Backbone.js (http://documentcloud.github.com/backbone/) yet beyond a very quick look, but I expect to use it (or something like it) next time I'm developing something that uses a significant amount of client-side JS.
I'm also very excited by the continued development on (and Yehuda Katz's participation in) SproutCore (http://www.sproutcore.com/).
[+] [-] jashkenas|15 years ago|reply
One client makes a change to a model -> changes are synced to the server -> other clients that currently reference that model have its attributes updated -> all the UI that displays the model is automatically re-rendered.
It's not critical for most applications, but is a really nice, nice-to-have for any JS app that displays the same editable data to more than one user at a time. Built-in conflict resolution mechanism for bonus points.
[+] [-] hasenj|15 years ago|reply
[+] [-] spudlyo|15 years ago|reply
Varnish is cool because it is very fast. It was written by Poul-Henning Kamp, who has a lot of experience in FreeBSD kernel development. He makes effective use of virtual memory, is careful to avoid memory operations that result in expensive bus transactions on mutli CPU systems, and knows how many system calls it takes to serve up a cache hit. All of this work has paid off. Varnish can turn a plodding CMS into a site that screams, and your profiling tools (siege, apache ab) will fall over before the site does.
Of course it helps if your CMS supports cache control headers, and isn't utterly laden with cookies, but that's where the VCL language comes in. You can write code to strip bogus cookies (like google analytics) coming from both the client and the server which vastly improves your cache hit ratios.
I like the way Varnish uses a shared memory pool for statistics and logging -- a wealth of information about the system is available to you but it doesn't generate a ton of I/O logging it to a file unless you ask it to. I love how you can use the telnet admin interface to compile new VCL code into a running system and then switch to it, while keeping old named configurations around in case you need to revert back.
Varnish has really helped me make slow sites fast this year, although it hasn't happened without some VCL coding effort and some understanding of how the sites operate.
[+] [-] drewp|15 years ago|reply
Squid can do what varnish does, but it's 2x slower by my measurements, and it has a gigantic config file. (Squid can do things that varnish can't, so you may have to use it anyway.)
[+] [-] dstein|15 years ago|reply
[+] [-] rphlx|15 years ago|reply
[+] [-] jaxtapose|15 years ago|reply
> turned out to be substantially better than anyone imagined
By anyone you mean analysts, right?
[+] [-] apu|15 years ago|reply
http://redis.io/
(I still use postgres and sqlite for other database needs, but I'm strongly considering moving a few of those over to redis if I have time.)
[+] [-] PStamatiou|15 years ago|reply
[+] [-] sage_joch|15 years ago|reply
[+] [-] sophacles|15 years ago|reply
1. Mongodb -- This year it really hit its stride and have been able to use it without worry for storing test results and experimental data. This is much nicer than the textfile logs -> sql -> processing datapath I was using previously.
2. flask: this little framwork is in my sweetspot. It does all the annoying crap of webby stuff, without all the "use our orm/routing model/way of thinking of http" so common in the space
3. mongrel2: I like it because it uses 0mq as the backend and sanely integrates some components in a way I feel could be better for many use cases than traditional stacks.
4. 0mq: This gets special mention, because it has been around for a while and I was actively using it, but 0mq 2 came out sometime this year, and is different enough from the first round, that it could be considered a separate technology. It isn't revolutionary in the MOM space, but it is a cool lightweight approach, and the core team has the type of dedication I like to see in OSS projects.
5. ABSOLUTELY NOT NEW: Haskell -- this year is the first year I've had time to sit down with Haskell for real, and start understanding the weird FP thing. This has made everything I do feel shiny and new, because even though I never actually use Haskell, I find myself writing very short hsskell programs in python and c and the other languages I use in my day job. When I started coding I remember thinking "This must be what a wizard feels like!", Haskell has brought back that feeling for me.
[+] [-] endgame|15 years ago|reply
ETA: `zmq_poll()` is nice, but some way to make it play nicely with other event loops (like libev's) would be even better.
[+] [-] sibsibsib|15 years ago|reply
I'm loving flask + mongo in combination; they make prototyping webapps almost effortless.
[+] [-] DanielBMarkham|15 years ago|reply
The really cool part came when I realized that with F# I was programming at the language level -- that I could effectively and easily write my own languages. So I decided I would like rails-like entities, where the entity reads the structure of the table and then conforms itself to whatever is in the table.
Couple hundreds of lines of code later, and presto chango, I could simply say "give me a list of customers" and point it to the table and I had a list of customers. This totally disconnected the database data structure from the code. Add a new field in the database and there was nothing to change in code. Or add a new field in the type and have it percolate out to the database. Change database providers and it was only a few function changes. Very cool. The kind of simple fix Microsoft should have done with data access instead of writing ODBC/ADO/OleDb//EF/etc
Then I had a blast with mailboxes, er monads, agents, and threads. Ended up writing a small app that was purely functional and all ran in the background. It was so automatic, at first I couldn't figure out how to start the dang thing!
This led to a venture into MPI and other technologies which has just begun. I'm also trying to wrap up my language work with a full DSL sometime soon (if I have a project that needs it). Looking forward to parsing and setting up trees and walking them. I also broke out of windows and started working in a linux environment using Mono, Apache, and MySQL.
Incredibly fun stuff. Looking back, I really had a blast this past year. Next year should be even better.
[+] [-] datapimp|15 years ago|reply
DocumentCloud really dropped some bombs this year. backbone.js, underscore.js are really great.
Socket.io saved my ass. I promised some big clients that I could make websockets driven apps for the iPad and then apple pulled websockets support without saying anything. So I was able to get socket.io for the win.
[+] [-] railsjedi|15 years ago|reply
[+] [-] railsjedi|15 years ago|reply
2. CoffeeScript language is an amazing replacement for Javascript. I can't see myself going back to pure JS at any point in the future.
3. Rails 3 finally feels like a stable and maintainable web framework. All the web frameworks now all seem to work together using Rack. The ruby web development world is really a nice place to be at the moment.
4. Bundler really nailed the gem dependency management issue (though the journey to 1.0 was very painful)
5. Sass / SCSS / Compass got really good. It feels unimaginable to go back to regular CSS.
Wow, now that I think about it, way too many great technologies to list. 2010 was an insanely good year for ruby web developers.
[+] [-] cheald|15 years ago|reply
[+] [-] swannodette|15 years ago|reply
miniKanren - logic programming w/o the Warren Abstract Machine. Has opened my eyes to a ton of incredible literature on this under appreciated programming paradigm.
[+] [-] PStamatiou|15 years ago|reply
First touched it two months ago just to tinker with but didn't really do anything with it. Then after numerous frustrations with my current blogging setup, I spent the last 5 days hacking on it over the holidays and I think it's almost ready to launch. Had to do some custom stuff that I'll write about in a post. It's extremely hackable and I love it. The only thing that doesn't work for me is LSI for related posts. Even with a fast computer and gsl/rb-gsl it still takes 10+ hours with my 1,000+ posts. Anyways, having a super fast site is going to be a breath of fresh air. Google was saying 88% of sites loaded faster than mine ( http://paulstamatiou.com ), though likely due to the images in many of my reviews.
Also installed Google mod_pagespeed and all is well so far.
* Though to be fair most of that is just my redesign that is more minimal, less ads, etc, but there's something extremely attractive about simple, flat files. No worrying about if your database will get corrupted. Everything is in git..
[+] [-] rgrieselhuber|15 years ago|reply
[+] [-] rmc|15 years ago|reply
[+] [-] jackowayed|15 years ago|reply
[+] [-] patio11|15 years ago|reply
[+] [-] cglee|15 years ago|reply
[+] [-] prawn|15 years ago|reply
[+] [-] sarkozy|15 years ago|reply
[+] [-] rdl|15 years ago|reply
The other thing which impressed me is kvm, in contrast to Xen.
[+] [-] dgallagher|15 years ago|reply
[+] [-] ja27|15 years ago|reply
GPU-accelerated VLC on my netbook has been amazing.
I got a Canon T2i / 550D this year. It shoots some amazing HD video and will only get better as I spend more on lenses and develop better techniques.
The Kindle 3 (brighter display and cheap price) have me reading books I've been putting off for years. It's great to have a device that's great at one thing and not very good at random browsing, Facebook, Twitter, HN, etc.
[+] [-] zefhous|15 years ago|reply
Using those tools has helped me to really enjoy writing JavaScript and to start doing it in a much more organized manner. They have been a huge catalyst for my growth as a JavaScript developer.
Also, using MongoDB has been awesome!
[+] [-] SandB0x|15 years ago|reply
[+] [-] ctkrohn|15 years ago|reply
[+] [-] peteforde|15 years ago|reply
I am also really excited by socket protocol tech advancements in the browser. I was able to pull off seriously cool stuff using http://pusherapp.com/ and also http://faye.jcoglan.com/ which is a nifty JS implementation of the Bayeux protocol.
[+] [-] RoyceFullerton|15 years ago|reply
1. Groovy - a programming language, it rocks because it less verbose and more powerful than Java and I can fall back onto standard Java syntax when I don't care to figure out how to do something in the 'Groovy' way. (http://groovy.codehaus.org)
2. Gaelyk - a groovy framework that runs on Google App Engine. Google App Engine is great for launching apps. It's free until it gets traction. (http://gaelyk.appspot.com)
3. Objectify - The simplest convenient interface to the Google App Engine datastore. Takes a lot of the pain out of using Bigtable. (http://code.google.com/p/objectify-appengine)
These all pack a mean punch and let me work on my night/weekend projects quite productively after overcoming a small learning curve.
I built http://icusawme.com and http://chatroulettespy.com with all three.
I'm looking forward to diving deeper into Appcelerator Titanium Mobile in early 2011.
[+] [-] oemera|15 years ago|reply
[+] [-] DanHulton|15 years ago|reply
2) Kohana - I love working with this framework. I never really worry about the ugly warts in PHP, because honestly? I'm not programming in PHP any more. I'm programming in Kohana, and I only occasionally fall back to PHP for "low-level" stuff.
[+] [-] gsmaverick|15 years ago|reply
[+] [-] crawshaw|15 years ago|reply
Not what you would call cool technology, but definitely the best technology I have used this year. Protobufs get out of the way so you can get work done.
[+] [-] endgame|15 years ago|reply
libev (http://software.schmorp.de/pkg/libev.html) was also a lot of fun to use for multiplexing sockets, plus it has a whole pile of other useful watchers that can use its event loop.