Akamai blocks unordered HTTP request headers

[idbehold]

In the second example the author does the following:

  $ ACCEPT="text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
  $ curl -v -H "$UA" -H "$ACCEPT" $URL |& grep '< HTTP'

The author fails to prefix $ACCEPT with the actual header key. It should be:

  $ curl -v -H "$UA" -H "Accept: $ACCEPT" $URL |& grep '< HTTP'

[jstanley]

From personal experience I'd be willing to give him the benefit of the doubt (i.e. he did it right, but wrote it up wrong). Good spot though.

[gwillem]

Thanks! Indeed a copy paste error, I updated the article.

[jnfurst]

This is just the configuration for a single site. The author did not even try it against www.akamai.com:

$ URL=http://www.akamai.com

$ UA="User-Agent: Mozilla/5.0 My API Client"

$ ACCEPT="Accept: /"

$ curl -v -H "$UA" -H "$ACCEPT" $URL

< HTTP/1.1 301 Moved Permanently

< Content-Length: 0

< Location: https://www.akamai.com

< Date: Tue, 02 May 2017 14:46:59 GMT

< Connection: keep-alive

[squeed]

Is it Akamai? Or is it a single site on Akamai? CDN customers can configure their sites in a million ways.

My guess is a single site that was getting DDoS'd added this as an attack signature and forgot about it.

[tyingq]

My money, in this case, is something like Akamai Kona or Shape Security, that does bot blocking. Comparing user-agent against known header order for that specific user-agent sounds like something they would do.

[jnfurst]

This is just a single sites configuration.

[michaelmior]

> most libraries use random order

Most libraries use an undefined order. This is not the same as random.

[AznHisoka]

Did Akamai recently just make this change?

I'm asking because I've been running a web crawler for years now, and in the past week, I have noticed that the crawler is being rejected in more websites then usual.

[gumby]

I disagree with the author's title (and I see it was submitted with a different title).

This is actually a report of two bugs:

1- the standard doesn't require an order

2 - the IETF's admonition that you be liberal in what you accept.