Python List Comprehensions Explained Visually

[tsumnia]

Maybe its me, but "Explained Visually" sounds more like it would have some 'visual' aspect to it besides a GIF of code (don't get me wrong, I think most code should be made an image to limit copy/pasting).

A website like VisuAlgo (http://visualgo.net/) is a great example of exampling something visually should mean

[th]

I agree. I was using "List Comprehensions: Now In Color" as a working title but I thought that was too nonsensical. It was actually the colorized code and not the gifs in particular that made me think "visual".

Sorry for the misleading title!

[scrollaway]

> I think most code should be made an image to limit copy/pasting

You think what now?

[ttkeil]

Setosa (http://setosa.io/) hosts some impressive interactive visualizations as well.

[x1798DE]

Why do you want to limit copy pasting?

[birchb]

I agree.

[danso]

Coming from Ruby to Python...I could not believe how damn hard it was to comprehend the beautifully explained and simple code (just 21 lines!) in Peter Norvig's writeup of how to create a spelling corrector [1], purely because of the list comprehensions (and a few other visual differences, such as partial list notation)

For example:

     inserts = [a + c + b for a, b in splits for c in alphabet]

It's no different (conceptually, at least) than doing:

     inserts = []
     for a, b in splits:
        for c in alphabet:
            inserts.append(a + c + b)

Or in Ruby (OK, maybe this isn't perfect Ruby):

     inserts = splits.reduce([]) do |x, (a, b)|
         alphabet.each{|c| x << a + c + b }
         x
     end
  

But the list comprehension was so difficult for me to read that it might as well have been a brand new concept. But I found I got quickly over it after rewriting all of his comprehensions as for-loops, and then trying to convert them back into comprehensions. It was a good reminder that sometimes the barrier is just plain muscle/memory reflex (and probably much more so for beginning programmers...)

[1] http://norvig.com/spell-correct.html

[mjevans]

I most frequently use List Comprehensions in a nested manor, whenever I do that I /ignore/ the normal python syntax and do this for readability.

    inserts = [ a + c + b
                for a, b in splits
                for c in alphabet ]

It has the effect of showing the nested loops stacked like they would be in code, but without intending.

[akavi]

I'd do the ruby version as:

    splits.flat_map do |(a, b)|
      alphabet.map{|c| a + b + c}
    end

Which feels pretty close to the list comprehension to my eyes.

[viraptor]

Depends on a programmer I guess. I'm pretty confused by the Ruby version and I can do Ruby in anger. I'd also use maps rather than each if I was writing it.

[chestervonwinch]

Cool. As a math guy, I enjoy the similarity between comprehensions and set-builder notation:

https://en.wikipedia.org/wiki/Set-builder_notation

[Gnewt]

In the opposite direction, having knowledge of Python's list comprehensions (and to some degree, an understanding of lazy evaluation) helped me a ton when set-builder notation was introduced in my first discrete math class.

[th]

Agreed! Haskell's list comprehensions look even more like set-builder notation: https://wiki.haskell.org/List_comprehension

[iamsohungry]

Are Python List Comprehensions really hard for people? I feel like my memory of learning them was I saw one in production code I was modifying, intuited what it did, pulled open a REPL and played with it for a bit, and then moved on.

[brbsix]

I agree. It really surprised me to see this getting so much traffic. I sometimes get hung up with nested comprehensions when I haven't been using them in a while, but other than that, they are quite mundane.

[coherentpony]

Is it just me or were there zero visualisations in this post?

[bgilroy26]

It might not be what you were expecting, but there's an animation if you ctrl-f "Here’s the transformation animated:"

[ebola1717]

That translation is kind of why i don't like python list comprehensions though. It's often clearer as a for-loop or map/filter, cause the different predicates are broken out into separate lines, rather than lumped together in one long line.

[d0mine]

It is not much easier to understand the explicit loop:

  doubled_odds = []
  for n in numbers:
       if n % 2 == 1:
           doubled_odds.append(n * 2)

compared to the list comprehension:

  doubled_odds = [n * 2 for n in numbers if n % 2 == 1]

Here's the explicit loop without newlines for comparison:

  for n in numbers: if n % 2 == 1: doubled_odds.append(n * 2)

Notice that the for-loop and the if-statement are in the same order in the loop and in the list comprehension. Only the value expression (n * 2) is out-of-order. OPs jumping around is often unnecessary in the gif.

The one line list comprehension is as much readable for people familiar with Python as the variant with the explicit loop.

It is not uncommon to use several list comprehensions in a function. If you rewrite them as explicit loops then it may increase the line count 4 times e.g., it can convert a simple 3-lines function that could be understood at a glance into 12 lines function that you have to read line by line to understand -- it is easy to keep track of 3 values on 3 lines but it is not clear whether it is true for 12 lines.

[viraptor]

A good reason to use list comprehension for simple things though is that you can sometimes quickly change [...] to (...) and get an equivalent generator. With a loop, you'd have to extract it into a separate yielding function.

[sanderjd]

I'm a big fan of just using map/filter, but I think it makes sense to wonder whether such a common and important operation is worth its own syntax. I do think the way nesting works is a bit surprising. It didn't really occur to me until reading an earlier comment that in essence the most nested loop is a `map` and all the others are `flatMap`s. (I think?)

[msellout]

Perhaps a for-loop is more similar to your memory of other code, but a list comprehension is more similar to an English sentence.

    "All names longer than 4 characters in the list of cities?"
    all(len(name) > 4 for name in cities)

    "Sum of x-squared for x from 10 to 20."
    sum(x**2 for x in range(10, 20))

[c3534l]

Frankly, comprehensions are one of Python's worst features. Some properly implemented functional syntax would be both more readable and concise. Plus double-nested (I'm not sure what you'd call it) comrpehensions seem to be backwards. Rather than [ingredient for ingredient in recipe for recipe in cookbook] it's [ingredient for recipe in cookbook for ingredient in recipe] which makes it sound as if there's only going len(recipe) number of ingredients in your list. Plus it's so verbose and horizontal, half the time if you want descriptive names you're gonna write something so verbose it doesn't even make it that much more readable.

[has2k1]

When reading code I find comprehensions easier to comprehend than combinations of map, filter and reduce. That is considering that I played around with some Common-Lisp before getting into Python.

The parts in the comprehension

    [*wanted* *iteration(s?)* *condition?*]

compares well to

    *iterations(s?)*
        *condition?*
            *append wanted*

for normal looping behaviour without changing order of the iterations.

One of my favourite comprehensions is of the sort

    [item for item in items in for i in (0, 1)]

of course this can be the same as

    list(items) * 2

but the comprehension is more versatile

    [item for item in items in for i in (0, 1) if items[0] != 'a']

For most once the mental fog clears comprehensions become natural very quickly as they are not that far removed from normal loops.

[oinksoft]

Haskell, Erlang, and Clojure all have sane lambdas (assuming that's what you mean by "properly implemented functional syntax"), and list comprehensions are useful and idiomatic in those languages. Agreed about sequence names feeling out-of-order in nested comprehensions in Python.

[darkerside]

From the Zen of Python: > There should be one-- and preferably only one --obvious way to do it.

If you have something written as a simple for loop, there's a real danger that you're moving in the wrong direction by translating it into a list comprehension.

[iamsohungry]

> Plus double-nested (I'm not sure what you'd call it) comrpehensions seem to be backwards.

Why are you even doing this? This isn't an acceptable way to write code.

> Plus it's so verbose and horizontal, half the time if you want descriptive names you're gonna write something so verbose it doesn't even make it that much more readable.

If your list comprehension is more than 80 characters wide, it's not because list comprehensions are bad, it's because you did something too complicated. Pull it into a function and name it so people know what the hell you're doing.

[cballard]

> Sometimes a programming design pattern becomes common enough to warrant its own special syntax. Python’s list comprehensions are a prime example of such a syntactic sugar.

I disagree with this thesis, although the post is otherwise a very nice guide to list comprehensions. Special syntax for specific types indicates a failure of the language to be generic enough.

List comprehensions anoint lists as a special thing. You don't get to play with list comprehensions, unless you're using the type that the language has decided to let you play with. If you decide to use a different type for some reason, you... can't.

map and filter can be part of a typeclass/interface/protocol (in Python these would just be informal), so you can use them on arbitrary types. If you want to switch your list type, it should just work.

I write a lot of Swift, and I'm constantly frustrated that optional chaining (?.) and exceptions are special things. I can't implement ?. for my type (say, a Result). Only the language creators can use it. Somewhat confusingly, ?? is a thing I can implement.

In Python's case, I think that list comprehensions are necessary because the language's support for first-class closures is poor.

[leni536]

> You don't get to play with list comprehensions, unless you're using the type that the language has decided to let you play with. If you decide to use a different type for some reason, you... can't.

You can use generator expressions and pass it to your custom type's constructor.

https://www.python.org/dev/peps/pep-0289/

[dragonwriter]

> List comprehensions anoint lists as a special thing. You don't get to play with list comprehensions, unless you're using the type that the language has decided to let you play with.

That's one reason why I like Scala's comprehensions; they have the conciseness of list comprehensions, but are more generic.

[sanderjd]

Yeah, it would be neater if they were just "iterable comprehensions" and worked with anything that is iterable.

[sdrothrock]

One basic misstep here for me is a comparison of why list comprehensions are better than loops -- speed, ease of reading, etc. There are a lot of comparisons, but no discussion of WHY list comprehensions.

I work on a project that's written mostly in Python; on occasion, other coworkers have to edit/work with my code and they're not used to Python at all, so I find myself writing for loops instead of list comprehensions for clarity.

[TheCowboy]

I find the utility of list comprehensions is maximized when I'm in a Python shell and dealing with indentation when writing a loop can become obnoxious to edit. I tend to have a Python shell tab open in my terminal throughout the day, and it's nice to bang out a quick loop in a one-liner.

When writing actual code, I tend to stick with regular for loops.

[tsurantino]

For those confused - the visual part is done in a GIF (i.e. http://treyhunner.com/images/list-comprehension-condition.gi...).

I think animating the transformation from a normal loop to a list comprehension is a great way to show how the syntax translates between the two forms. Very awesome and comprehensive post.

[hatmatrix]

Maybe I'm an FP snob but it would have been nice to mention the analogy to set builder notation (though lists aren't sets - maybe multisets or bags) to indicate the existence of higher abstraction; not just syntactic sugar on the for-loop.

However, from a practical point of view I think it is great introduction for procedural programmers to begin using list comprehensions.

[keepitsurreal]

I've been reviewing Python this week and this was a great brush up. Thanks!

[jredwards]

nested comprehensions are kind of a nightmare to read, but basic list comprehensions seem incredibly straightforward to me. So much so that the explanation is more complex.

[skadamat]

Not sure how this is visual, neat post either way!

[y04nn]

When I started to learn Python, I was quite lost with this feature, but now I love using it.

[plusquamperfekt]

Is it really that hard to grasp?