Every Byte of a TLS Connection Explained and Reproduced

[syncsynchalt]

Author here - I was going to publish this today but it leaked out ahead of time. Enjoy!

EDIT: I'm putting a CDN in place.

[jor-el]

Kudos! This a great work. I remember few years back when I was exploring TLS, there were no such resources and it took many months of trial and error to get some reasonable understanding. I tried doing a similar thing, but was only able to do for couple of packets [0]. Surely such an illustration will go a long way to help newbies to understand a complicated protocol.

[0] https://serializethoughts.com/2014/07/27/dissecting-tls-clie...

[randlet]

Thanks! This is really a fantastic resource. Something like a TLS connection is one of those things that seems really intimidating but when you see it laid out like this it makes you feel like 'I can understand this!'. Great work.

[zzzeek]

very nice! first thing I noticed is that at https://github.com/syncsynchalt/illustrated-tls/blob/master/... the content and samples are hardcoded into HTML - might it be nice if this was generated from some kind of JSON file or similar such that the approach you have here could be generalized to support any network protocol someone might want to annotate with descriptions?

[bebop]

Great illustration, I am going to share this with the rest of my team. One thing that I would find interesting is how x509 client certificates fit into the negotiation. I know this is optional so I don't know how it would fit into your flow easily though.

[TimJYoung]

I did a presentation at our local software meetup on this topic a month ago, and wow, I could have really used this. It's way clearer and more organized than my muddled slides.. :-)

Very nice work !

[rustcharm]

Can you do this for DTLS, too? :=-)

[mito88]

A canadian?

Nice.

[toast0]

This is wonderful!

I may make a version where the bytes used for lengths are highlighted, since it feels like so many bytes are lengths; look at the SNI extension, which has three 16-bit lengths, I know why they're there, but SNI probably shouldn't be a list, and even if it was a list, an extension that consists solely of a list has a list of the length of the extension, you shouldn't need two bytes for that, and if we recast sni into just a type and a string, the string is clearly going to take the rest of the extension length, so it doesn't need a two byte length either.

[syncsynchalt]

Agreed, and I think some pain must have gone into those redundant length bytes. The way they've done it makes it very easy and natural to extend any part of any record later, but it gets ridiculous when you have to document them.

[userbinator]

Moreover, from a security perspective since we're talking about TLS, "overspecified" and/or "redundant" lengths are just begging to be made inconsistent and a source of vulnerabilities.

[bungie4]

Beautiful.

I recently had to implement two way auth over 1.2 and this would have saved much hair pulling. (and who does TWO way auth over an MPLS connection. Turns out, us).

[pimlottc]

I feel your pain. Two-way TLS is a funny thing, it's supported by the standards and even most implementations but its actual use is minuscule compared to "normal" one-way TLS, so much so that it's hard to find documentation even acknowledging two-way TLS exists, let alone how to use it.

And don't get me started about the hassles of obtaining signed certificates that are actually usable for client auth...

[gbajson]

Has anyone of you seen such beautiful explanation for other protocols (TCP, 4-way handshake)?

[syncsynchalt]

They're in text form, but I've always loved the clarity of W. Richard Stevens' (RIP) books on TCP/IP.

He did a great job of demonstrating every byte and jitter on the wire and how it related to the underlying BSD TCP/IP stack.

[e12e]

I'm reminded about the STEPS report of using ascii art from the rfcs as a DSL for describing tcp:

Writeup/comment: http://www.moserware.com/2008/04/towards-moores-law-software...

Comment thread: https://news.ycombinator.com/item?id=846028

The vpri texts: http://www.vpri.org/writings.php

Appendix e and section "A Tiny TCP/IP Using Non-deterministic Parsing" of "STEPS Toward The Reinvention of Programming: First Year Progress Report, December 2007.":

http://www.vpri.org/pdf/tr2007008_steps.pdf

[Xeanort]

You can usually use Wireshark to find out the meaning of each byte for internet connections (and images), it works great.

[rohansingh]

Beautiful. A few years ago, were implementing some TLS handshake stuff at Spotify (https://github.com/spotify/ssh-agent-tls) to be able to use your SSH key as a client-side cert for HTTPS connections.

The first couple drafts took forever to figure out, and I got a bunch of stuff wrong. This guide would've saved a ton of time back then.

[kodablah]

There's a lot of practicality I learned from reading the Go source and implementing my own (e.g. always an array of a single value of 0 for compression method).

Now do a version with DTLS (one more message type, couple more fields on existing types, and logic concerning retries). Also, now do a TLS 1.3 one.

[syncsynchalt]

Yep, reading the Go source is what inspired me to do this, it's still very compact and readable.

I originally used an AEAD cipher but found it was impossible to demonstrate on the command line (openssl enc refuses to do AEAD because it can't confirm the authentication in a streaming context).

A friend asked me to demonstrate ALPN in this but as I looked into it I found it was distracting, as there's already so much going on and any new feature required digression. Maybe next time!

As for 1.3 my next project was going to be implementing it rather than documenting it. Just a throwaway implementation, nothing you'd want to use.

[quadyeast]

I have no need to look at that page at length ... but I am. This is very well made and interesting.

[blazespin]

I remember implementing ssl in java when I worked for Netscape back in the day. Nostalgia rush looking at this. Great stuff

[kalimatas]

The irony is: I get "Your connection is not secure" when trying to open the link.

[syncsynchalt]

I've uncapped MaxRequestWorkers while I wait for the Cloudflare queue, it should work now (I believe you were seeing an error related to TLS timeout).

[phaedrus]

I love the way you present the data (where clicking on the hex values of the bytes gives their explanation). At work I work on software that communicates via RS-232 serial packets; I should make something that turns logs of the packet data into HTML files with an interface like this!

[amenghra]

http://www.networksorcery.com/enp/Protocol.htm has a bunch of network diagrams for various protocols.

This website lays things out really nicely, would love to have more protocols :)

[Obi_Juan_Kenobi]

So the record header has 2 bytes for the payload size, and the handshake header has 3 bytes. Am I correct in thinking that the 3 bytes of the handshake header is superfluous? Isn't payload size limited to 2^16 bytes

[max23_]

Good work on the illustration.

Using Diffie-Hellman to generate a shared key with each party's private key and the other party's public key is the part that amazed me most when I was trying to understand the handshake back then.

[userbinator]

The parties had agreed on a cipher suite using ECDHE, meaning the keypairs will be based on a selected Elliptic Curve, Diffie-Hellman will be used, and the keypairs will be Ephemeral rather than using the public/private key from the certificate.

I think it's important to mention that even with ephemeral cipher suites, the server's ephemeral public key is signed using the server's certificate private key and verified by the client, since otherwise one would be able to MITM the key exchange.

[pcunite]

Nice, could have really used this a few years ago when I was making my own HTTP server implementation. Could someone make a PDF of this? Any C++ and C# examples of this?

[syncsynchalt]

I might post a PDF later (as the content becomes final), or make a printable view. In the meantime try this:

    # in your javascript console, paste this:
    [].forEach.call(document.querySelectorAll(".record, .calculation"), function(el){el.classList.add("selected")});
    [].forEach.call(document.querySelectorAll(".record, .calculation"), function(el){el.classList.add("annotate")});
    [].forEach.call(document.querySelectorAll("codesample"), function(el){el.classList.add("show")});

Then you can print the page to PDF.

[umanwizard]

As a crypto ignoramus - Why is the random data from each side necessary? Why can’t things just be encrypted with the PreMasterSecret directly ?

[tialaramex]

Involving random data gives everybody who gets to pick the random data (so in TLS that's both client and server) a freshness guarantee.

Because the other party needed to know you'd picked this particular random data to make the keys, the messages from them encrypted with those keys couldn't possibly have been pre-recorded / replayed.

In the ephemeral Diffie Hellman modes both parties contribute to the key anyway so this isn't as important, but with old school RSA the random values are the only thing preventing Replay attacks.

TLS 1.3 capable servers also scribble "DOWNGRD" in part of the random field if a client message says it can't do TLS 1.3. If a TLS 1.3 client sees that unusual "random" choice it knows bad guys tampered with the connection (attempted a downgrade attack). If bad guys just change the values, they won't match between client and server and the connection aborts. Older clients think nothing of the unusual random value and carry on as before.

[erseveR]

It's about the nonce / initialization vector. Basically it is used inside the cryptos primitives to add some entropy to the encryption itself and prevent a various ranges of attacks. A good introduction to crypto is "Introduction to modern cryptography" imho

[kapad]

There's a typo in the server keys calculation. It should read "server" and not "client" IMO.

[DontSueMeBro]

This is very nice. I think the green on green and blue on blue highlight are a little hard to see.

[based2]

https://www.reddit.com/r/netsec/comments/9nk764/the_illustra...

[ct520]

As someone that frequently deals with large scale integration failures I greatly appreciate this site. Thanks syncsyncchalt!