I'm 3.5 years into using Go exclusively as well, and this rings very true to me with regard to generics:
> Interfaces are good enough 99% of the time.
Generics would be really nice for generic data structures (heaps, trees, etc), but code generation is ok at that.
Generics could allow for some nicer (and safer!) nil handling and error checking, but I think the benefits-vs-complexity are less clear than with generics for data structures.
I think it's hard to quantify the massive benefit Go's simplicity is to onboarding developers -- especially for a new language where hiring experienced devs is next to impossible. The ease of onboarding is reason enough for businesses to consider Go as over an org's life a lot of time will be spent (wasted?) onboarding.
Any language features which increase cognitive overhead had better offer some extremely compelling benefits to outweigh an increased learning curve.
> And I don’t mean interface{}. We used interface{} rarely in Juju, and almost always it was because some sort of serialization was going on.
This has been my experience as well. Whenever I hear someone complaining about interface{}, I wonder what they're doing. I think it's often people used to having generics or a dynamic language trying to follow similar patterns in Go and not considering alternative patterns.
I've never used a language that didn't lack type information at the edges (where serialization occurs). Even using strongly typed serialization (eg Protobufs) in a language with strong type features and patterns (eg Java) I always see a fair amount of glue code converting from "weaker" serialization types to stronger internal representations.
As long as those edges are architected to be easily testable (even fuzzable!); I don't see it as a problem. You have to convert from bytes-on-the-wire to typed variables somehow.
>Generics would be really nice for generic data structures (heaps, trees, etc), but code generation is ok at that.
Either code generation is an implementation detail of generics, or this is an afterthought that adds incidental complexity and was almost assuredly better off being a core language feature.
>I think it's hard to quantify the massive benefit Go's simplicity is to onboarding developers
This is really only applicable to small applications using new languages where the main cognitive onboarding is learning a new language/paradigm. As soon as you reach a certain size of application, the frameworks, architecture and domain knowledge heavily outweigh the cost of learning even the most esoteric of language features.
>Any language features which increase cognitive overhead had better offer some extremely compelling benefits to outweigh an increased learning curve
I wonder what kind of language feature you're talking about? The whole idea of generics in a more functional oriented language is to reduce cognitive overhead. The beauty of parametricity is it takes things away from the developer to help them reason about the code!
Our (very large) team of engineers use this to great affect throughout our codebases. There is no such thing as a silver bullet, but this comes as close as I've seen in the 12 years I've been an engineer (from startups to a Fortune 10 company).
> Any language features which increase cognitive overhead had better offer some extremely compelling benefits to outweigh an increased learning curve.
This is a common misunderstanding that occurs (I think) because most people only know C++-, Java-, and C#-style generics, which can be conceptually complicated, because of the often tricky interactions with the type system (or, in the case of C++, the use for metaprogramming).
In contrast, module-based genericity (SML, OCaml, Ada, Modula-3) is conceptually pretty simple, as it avoids complicating the type system. Its main downside is (relative) verbosity, but Go has never really eschewed verbosity.
>>> I think it's hard to quantify the massive benefit Go's simplicity is to onboarding developers -- especially for a new language where hiring experienced devs is next to impossible. The ease of onboarding is reason enough for businesses to consider Go as over an org's life a lot of time will be spent (wasted?) onboarding.
Agree. I feel it so much.
I stopped counting the companies I couldn't join because they had exotic languages or just the latest fad of the year.
That's a very effective way for a company to stay away from having any experienced engineer while making it very hard to recruit at all (note that both effects amplify each other!). The hardest and newest the languages, the worst it is.
As an employee, that's a painful way to have a company cancel a decade worth of experience. I'm not interested in starting fresh again. Bye.
> The first was assuming forward slashes for paths in tests. So, for example, if you know that a config file should be in the “juju” subfolder and called “config.yml”, then your test might check that the file’s path is folder + “/juju/config.yml” - except that on Windows it would be folder + “\juju\config.yml”.
Go gives you the tools to handle this, but you have to do it yourself. filepath.Join(), filepath.ToSlash() and filepath.FromSlash() are used for this.
I find myself very productive in Go, I've written a lot of it now, but I do also find myself writing more code than I thought I would, having had some expectations set by Python and Java. Go's philosophy seems to avoid doing something which can be done incorrectly, and instead punting it to developers. It's not always as simple as simply replacing forward and backward slashes.
Even though I've written more ruby than anything else over the last decade, I have to agree with this. Ruby is nice OO language, but it's unabashedly multi-paradigm, and that leads to a lot of mess when you get a team with varying backgrounds the result is potentially a byzantine mix of procedural, object-oriented and functional styles.
Since you can write for loops in anything other than a pure functional language, this is equivalent to saying that, unless your language is pure functional, it shouldn't have any FP features in it. I think this is obviously false: Ruby is not any worse for having the map method.
> You have to fix my code one day I'm on vacation.
> You think that it's ugly, and rewrite it as a map.
Well, if your colleagues do that, their is a social problem in your company, and I'm not sure a langague has anything to do with that. Your colleagues could also rename all your variables at this point…
I'm looking forward to my first project with GO. It appears to offer a lot with minimal complexity.
> Because Go has so little magic, I think this was easier than it would have been in other languages. You don’t have the magic that other languages have that can make seemingly simple lines of code have unexpected functionality. You never have to ask “how does this work?”, because it’s just plain old Go code.
That lack of magic and his comparison to C# sounds like a really good mix.
I have a really hard time understanding what people mean when they say magic. In every language I've ever worked in I spend a fair bit of time saying "how does this work". Go doesn't seem any different in that regard to me.
I think the go language has taken a position along the lines of "re usability and abstraction are overrated". I certainly think there is some truth to that, I am really enjoying working with go on smaller projects and it is this philosophy that has made understanding the code much easier.
But I wonder how it really scales on a large code base like this? Some of the best projects I've worked on leverage usability more effectively to create a sort of vocabulary. They're far more concise, there is rarely more than one source of truth, they're far easier to change and improve. Does this hold true for 540,000 lines of go code?
> But I wonder how it really scales on a large code base like this? Some of the best projects I've worked on leverage usability more effectively to create a sort of vocabulary. They're far more concise, there is rarely more than one source of truth, they're far easier to change and improve. Does this hold true for 540,000 lines of go code?
Doesn't this article speak to this? It mentions juju has over a million lines.
To actually answer the question - yes it holds true, yes it scales (in my experience). There are a lot of abstractions that we built for Juju. And we absolutely tried to ensure there was only a single source of truth for everything. It would have been totally unworkable if we couldn't reuse logic etc. We may not always have chosen the best abstraction, but that's a problem in any language.
Great point about time. At work we've adapted github.com/WatchBeam/clock and it's helped a lot.
Thanks for blogging about your work on juju! Despite Go already being five years old, many of the patterns around building large applications are only emerging now.
This is a minor nit, but it seems to me that it would have been easier to configure the unit with a shorter timeout duration rather than mocking out the time functions. Am I mistaken?
Implement a heap that works for any type with a user-defined ordering relation (in particular, you should be able to implement both max heaps and min heaps over the same type by changing the ordering). The heap should allow for efficient operations to add an element, retrieve the minimum element, and to merge two heaps; merging should be able to make use of specialized bulk operations rather than just adding elements one by one. The data structure should be opaque, so that you can (e.g.) switch out binary heaps for Fibonacci heaps later on. The interface should be typesafe.
Heaps are useful, inter alia, to define efficient priority queues.
Other examples:
* Implement directed graphs using arbitrary types for nodes and edges. Graph algorithms are useful in a number of application areas.
* Implement a parser combinator library that works for various types of tokens, semantic values, and states.
General problems with lack of parametric polymorphism (where subtyping is not enough) are:
* The necessity for the client of a service to cast the result to the desired type.
* Difficulty in implementing binary operations efficiently where both operands are of the same or related types (such as the heap merge above), because you can't know for certain that they are of the same type just because they conform to the same interface.
* Lack of type safety guarantees when you're mixing incompatible instances (such as merging a min heap and a max heap over the same type).
Yes they are. And actually they would play nice with generics! Imagine
type Ord a interface {
Compare(a) int
}
Also it would be nice to do something like
func Max[a <: Ord a](a, a) a
(Syntax stolen from Scala)
Right now, interfaces are too opaque. You can't take a value of interface type X and return the same type. You have to return an opaque X, which gives you no guarantees about its concrete implementation. Parametricity is an extremely well-motivated and solved language feature!
I think if you add parametricity to Go functions (not even parametric types. Just type variables that play nice with all the builtins) you can write this and get guarantees about its implementation (assuming it follows the functor laws)
Been a few years, and it was cool in the day, but very complex. With k8 now, not sure why one would choose Juju first. They are slightly different problem domains, but not exactly.
I think so? Assuming a regular function (a method should do one thing and one thing only) is about ~20 lines, we're looking at ~6 functions (170 lines - 17 comments - about 15 lines of newlines, imports and so on = 130-odd).
The comment ratio is going to vary heavily depending on the problem the project is trying to solve. If it was a library, I would expect the ratio of comments to go up to 30% or more.
The LOC per file average seems slightly high for my taste but okay.
This sounds normal in go. In go a minimal import (package) is a while directory, not a single file, so it's normal to split packages into smaller files, each file implements a smaller set of features.
Using cloc against the Go source code, I found that it has 934025 loc with 3080 files. This averages to about 300 loc per file. There are 149688 lines of comments, which averages to about 50 per file.
Does anyone know where this developer is going after Canonical or why they left? It seems like they got to work on a nice project while there. Maybe that's another blog post though. It piqued my curiosity I guess.
It's written with knocking down a very specific set of straw men in mind, but rather carefully avoids coming anywhere close to addressing the legitimate criticisms of Go as a language. One of the things that's most irritating about Go enthusiasts is the way they try to close ranks on legitimate critique and reframe their language's warts as "simplicity".
Also: 20 bonus blub points for pulling the old 'I don't need generics, therefore NOBODY does' gambit.
Off topic rant: I don't know much about the details of godeps hash file but I do wish that there were a better infrastructure for merging contents of various file formats. Built into git or shipped as a separate repository. I wasted too much time merging vcxproj.filters files just because in its XML representation one item of a sequence of folder assignments occupies 3 lines (opening tag, contents, closing tag) instead of having everything on one line. Similar problems with JSON files when new items added concurrently to the end of the array.
I found a program someone wrote called sortxml that will prettify and sort XML files. It's written C#. Might be possible to run it on Linux also, or to find another similar program for Linux that will. I'd investigate further if it wasn't for the fact that I rarely work with XML files.
You may be able to write a small wrapper script that does the following: For each filename passed as argument to your script you will run sortxml on it and store the result in a temporary file, then once you've done so for all files, you pass the sorted files to an existing merge tool such as for example meld, kdiff3 or vimdiff.
Let's say that you name your script mergexml.bash and that you put it in ~/bin/. Then it should be possible to first git merge as usual but when a conflict arises in an XML file, you will run something like
git mergetool -t ~/bin/mergexml.bash example.xml
And your script will have simplified the process.
For bonus points, you might write an unsortxml program that your script will call at the end to rearrange the nodes and attributes to be in the same order as they were in one of the original XML files -- the user would be prompted to select which of the original XML files to use for ordering the nodes and attributes.
I haven't tested this of course, otherwise I'd probably have actual code to share for it but I think something like what I've outlined above should work.
The godeps file is just a dependency-per-line, tab-separated values, deliberately so it's easily amenable to shell script processing.
Aside: the conflicts mentioned in the article should never be a real problem because you can always resolve the conflict by just recreating the dependences.tsv file (you should never be editing it manually anyway).
[+] [-] schmichael|9 years ago|reply
> Interfaces are good enough 99% of the time.
Generics would be really nice for generic data structures (heaps, trees, etc), but code generation is ok at that.
Generics could allow for some nicer (and safer!) nil handling and error checking, but I think the benefits-vs-complexity are less clear than with generics for data structures.
I think it's hard to quantify the massive benefit Go's simplicity is to onboarding developers -- especially for a new language where hiring experienced devs is next to impossible. The ease of onboarding is reason enough for businesses to consider Go as over an org's life a lot of time will be spent (wasted?) onboarding.
Any language features which increase cognitive overhead had better offer some extremely compelling benefits to outweigh an increased learning curve.
> And I don’t mean interface{}. We used interface{} rarely in Juju, and almost always it was because some sort of serialization was going on.
This has been my experience as well. Whenever I hear someone complaining about interface{}, I wonder what they're doing. I think it's often people used to having generics or a dynamic language trying to follow similar patterns in Go and not considering alternative patterns.
I've never used a language that didn't lack type information at the edges (where serialization occurs). Even using strongly typed serialization (eg Protobufs) in a language with strong type features and patterns (eg Java) I always see a fair amount of glue code converting from "weaker" serialization types to stronger internal representations.
As long as those edges are architected to be easily testable (even fuzzable!); I don't see it as a problem. You have to convert from bytes-on-the-wire to typed variables somehow.
[+] [-] runT1ME|9 years ago|reply
Either code generation is an implementation detail of generics, or this is an afterthought that adds incidental complexity and was almost assuredly better off being a core language feature.
>I think it's hard to quantify the massive benefit Go's simplicity is to onboarding developers
This is really only applicable to small applications using new languages where the main cognitive onboarding is learning a new language/paradigm. As soon as you reach a certain size of application, the frameworks, architecture and domain knowledge heavily outweigh the cost of learning even the most esoteric of language features.
>Any language features which increase cognitive overhead had better offer some extremely compelling benefits to outweigh an increased learning curve
I wonder what kind of language feature you're talking about? The whole idea of generics in a more functional oriented language is to reduce cognitive overhead. The beauty of parametricity is it takes things away from the developer to help them reason about the code!
Our (very large) team of engineers use this to great affect throughout our codebases. There is no such thing as a silver bullet, but this comes as close as I've seen in the 12 years I've been an engineer (from startups to a Fortune 10 company).
[+] [-] rbehrends|9 years ago|reply
This is a common misunderstanding that occurs (I think) because most people only know C++-, Java-, and C#-style generics, which can be conceptually complicated, because of the often tricky interactions with the type system (or, in the case of C++, the use for metaprogramming).
In contrast, module-based genericity (SML, OCaml, Ada, Modula-3) is conceptually pretty simple, as it avoids complicating the type system. Its main downside is (relative) verbosity, but Go has never really eschewed verbosity.
[+] [-] user5994461|9 years ago|reply
Agree. I feel it so much.
I stopped counting the companies I couldn't join because they had exotic languages or just the latest fad of the year.
That's a very effective way for a company to stay away from having any experienced engineer while making it very hard to recruit at all (note that both effects amplify each other!). The hardest and newest the languages, the worst it is.
As an employee, that's a painful way to have a company cancel a decade worth of experience. I'm not interested in starting fresh again. Bye.
[+] [-] VMG|9 years ago|reply
Wait what? I thought this was solved ten years ago https://en.wikipedia.org/wiki/Path_(computing)#MS-DOS.2FMicr...
[+] [-] oppositelock|9 years ago|reply
I find myself very productive in Go, I've written a lot of it now, but I do also find myself writing more code than I thought I would, having had some expectations set by Python and Java. Go's philosophy seems to avoid doing something which can be done incorrectly, and instead punting it to developers. It's not always as simple as simply replacing forward and backward slashes.
[+] [-] NateDad|9 years ago|reply
If you ask what the path of a file is on Windows, it'll give you the backslash version.
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] greenhouse_gas|9 years ago|reply
You want a functional language? Use it.
You want a procedural language? Use it.
You want an OOP language? Use it.
They're all good, but not in one language.
For example, you like FP idioms, and program everything with maps.
I like for loops.
You have to fix my code one day I'm on vacation.
You think that it's ugly, and rewrite it as a map.
I get back, bug comes up, I rewrite it back to for loops.
Repeat.
Go showed how to finally end indentation wars, maybe it can show how to end style wars.
[+] [-] dasil003|9 years ago|reply
[+] [-] pcwalton|9 years ago|reply
[+] [-] stymaar|9 years ago|reply
> You think that it's ugly, and rewrite it as a map.
Well, if your colleagues do that, their is a social problem in your company, and I'm not sure a langague has anything to do with that. Your colleagues could also rename all your variables at this point…
[+] [-] oxalorg|9 years ago|reply
[+] [-] ben_pr|9 years ago|reply
> Because Go has so little magic, I think this was easier than it would have been in other languages. You don’t have the magic that other languages have that can make seemingly simple lines of code have unexpected functionality. You never have to ask “how does this work?”, because it’s just plain old Go code.
That lack of magic and his comparison to C# sounds like a really good mix.
[+] [-] tonyedgecombe|9 years ago|reply
Actually I think it offers little with minimal complexity.
[+] [-] kasey_junk|9 years ago|reply
I have a really hard time understanding what people mean when they say magic. In every language I've ever worked in I spend a fair bit of time saying "how does this work". Go doesn't seem any different in that regard to me.
[+] [-] excepttheweasel|9 years ago|reply
But I wonder how it really scales on a large code base like this? Some of the best projects I've worked on leverage usability more effectively to create a sort of vocabulary. They're far more concise, there is rarely more than one source of truth, they're far easier to change and improve. Does this hold true for 540,000 lines of go code?
[+] [-] weberc2|9 years ago|reply
Doesn't this article speak to this? It mentions juju has over a million lines.
[+] [-] NateDad|9 years ago|reply
I fail to see how that is saying that abstraction and usability are overrated. They're not. They're important.
[+] [-] NateDad|9 years ago|reply
[+] [-] jrs95|9 years ago|reply
[+] [-] zalmoxes|9 years ago|reply
Thanks for blogging about your work on juju! Despite Go already being five years old, many of the patterns around building large applications are only emerging now.
[+] [-] weberc2|9 years ago|reply
[+] [-] akerro|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] al2o3cr|9 years ago|reply
[+] [-] k__|9 years ago|reply
I mean, you don't have to implement all the interfaces explicitly, you just have to get your structure right and be done with it.
Am I missing something?
[+] [-] rbehrends|9 years ago|reply
Heaps are useful, inter alia, to define efficient priority queues.
Other examples:
* Implement directed graphs using arbitrary types for nodes and edges. Graph algorithms are useful in a number of application areas.
* Implement a parser combinator library that works for various types of tokens, semantic values, and states.
General problems with lack of parametric polymorphism (where subtyping is not enough) are:
* The necessity for the client of a service to cast the result to the desired type.
* Difficulty in implementing binary operations efficiently where both operands are of the same or related types (such as the heap merge above), because you can't know for certain that they are of the same type just because they conform to the same interface.
* Lack of type safety guarantees when you're mixing incompatible instances (such as merging a min heap and a max heap over the same type).
[+] [-] whateveracct|9 years ago|reply
Right now, interfaces are too opaque. You can't take a value of interface type X and return the same type. You have to return an opaque X, which gives you no guarantees about its concrete implementation. Parametricity is an extremely well-motivated and solved language feature!
I think if you add parametricity to Go functions (not even parametric types. Just type variables that play nice with all the builtins) you can write this and get guarantees about its implementation (assuming it follows the functor laws)
[+] [-] buckhx|9 years ago|reply
That being said, it's never been a deal breaker for me and don't end up missing generics THAT much.
[+] [-] woah|9 years ago|reply
[+] [-] jasonwatkinspdx|9 years ago|reply
[+] [-] therealmarv|9 years ago|reply
[+] [-] brianwawok|9 years ago|reply
[+] [-] alsadi|9 years ago|reply
For container orchestration it's clear that kubernetes is the winner (if you have k8s deployed you don't need juju)
For managing the whole thing manage iq have much to offer.
[+] [-] Gys|9 years ago|reply
540,000 lines of Go code
65,000 lines of comments
So on average only 170 lines per file, including 17 lines of comments.
Are this normal ratios ?
[+] [-] adtac|9 years ago|reply
Six functions sounds very reasonable to me.
[+] [-] sulam|9 years ago|reply
The LOC per file average seems slightly high for my taste but okay.
[+] [-] fishywang|9 years ago|reply
[+] [-] butabah|9 years ago|reply
I'd say those ratios are very similar.
[+] [-] michaelmcmillan|9 years ago|reply
[+] [-] bogomipz|9 years ago|reply
[+] [-] josteink|9 years ago|reply
Reading about projects with 100+ types of collections can certainly lead you to think so.
[+] [-] rjammala|9 years ago|reply
[+] [-] grabcocque|9 years ago|reply
http://paulgraham.com/avg.html
It's written with knocking down a very specific set of straw men in mind, but rather carefully avoids coming anywhere close to addressing the legitimate criticisms of Go as a language. One of the things that's most irritating about Go enthusiasts is the way they try to close ranks on legitimate critique and reframe their language's warts as "simplicity".
Also: 20 bonus blub points for pulling the old 'I don't need generics, therefore NOBODY does' gambit.
[+] [-] mylons|9 years ago|reply
[+] [-] Nanshan|9 years ago|reply
[+] [-] mynegation|9 years ago|reply
[+] [-] eriknstr|9 years ago|reply
https://github.com/kodybrown/sortxml
You may be able to write a small wrapper script that does the following: For each filename passed as argument to your script you will run sortxml on it and store the result in a temporary file, then once you've done so for all files, you pass the sorted files to an existing merge tool such as for example meld, kdiff3 or vimdiff.
Let's say that you name your script mergexml.bash and that you put it in ~/bin/. Then it should be possible to first git merge as usual but when a conflict arises in an XML file, you will run something like
And your script will have simplified the process.For bonus points, you might write an unsortxml program that your script will call at the end to rearrange the nodes and attributes to be in the same order as they were in one of the original XML files -- the user would be prompted to select which of the original XML files to use for ordering the nodes and attributes.
I haven't tested this of course, otherwise I'd probably have actual code to share for it but I think something like what I've outlined above should work.
[+] [-] rogpeppe1|9 years ago|reply
Aside: the conflicts mentioned in the article should never be a real problem because you can always resolve the conflict by just recreating the dependences.tsv file (you should never be editing it manually anyway).