Routers treat HTTP and HTTPS traffic differently

[aminuit]

This article is based on a false premise.

Because HTTPS clients and servers set the DF or Do Not Fragment IP flag on packets and regular HTTP clients and servers do not.

There is nothing anywhere in any standard that requires DF to be set for HTTPS traffic. There are certain links on the Internet that will set DF on your traffic in transit because it is more expensive to forward and even more expensive to do deep inspection on fragmented IP packets, but there is no reason this would be done for HTTPS and not for HTTP.

EDIT: For completeness: links to RFC 2246 for TLS 1.0 and one of the draft specs for SSL 3.0 (it was never an official standard). There is nothing about setting the DF bit in the IP header in either document.

TLS 1.0: http://www.ietf.org/rfc/rfc2246.txt

SSL 3.0: http://tools.ietf.org/html/draft-ietf-tls-ssl-version3-00

[mmaunder]

Here's how you set dont-fragment on a socket:

http://manpages.ubuntu.com/manpages/jaunty/man7/ip.7.html See: IP_MTU_DISCOVER

This premise isn't taken from an RFC. It's based on my own real-world observations of what is crossing the wire. Grab wireshark, fire up tcpdump and form your own opinion - and please do share what you find with us.

Mark.

[wmf]

This is setting off alarms in my head. He's talking about path MTU discovery, but for some reason he doesn't actually use that term. http://en.wikipedia.org/wiki/Path_MTU_discovery

Also, I've never heard of a system that sometimes uses PMTUD and sometimes doesn't. I can imagine that those SSL offload boxes may be doing weird stuff; maybe that's where the author saw this problem.

[mmaunder]

Thanks wmf I should have included the reference and have added it. There's a lot more that I don't cover e.g. PMTUD black hole routers, etc. But the article is targeted at an intermediate level audience, so leaving those out for now.

[axod]

I just tried tcpdump of http vs https, and both had the flag unset :/ (As I'd expect)

Would have been nice if the OP had included some example logs showing the issue.

[mmaunder]

Hi axod, on request from a commenter on my blog I've described how I captured packets and analyzed. I'm interested that you saw the DF flag for non-HTTP traffic. Can you tell me which OS and client or browser you used? I used Ubuntu 9.04 and curl, but have seen the same df/non-df behavior with a variety of other clients on the same OS.

Mark.

[noonespecial]

Tinkering with your network cards MTU just to try to access a web site is probably a bad idea in most cases!