The Heartbleed Challenge

[eastdakota]

You can read more about the CloudFlare Challenge and our own tests on obtaining SSL Private Key material here:

http://blog.cloudflare.com/answering-the-critical-question-c...

Matthew Prince Co-founder & CEO, CloudFlare

[ominous_prime]

Thanks. This makes a lot of sense, and matches what I saw myself. I never got anything out of nginx, but I found apache quite easy. I never built more than a POC, and left it at that.

I didn't think about it at the time, but it was only on a newly started apache instance.

Coupled with the fact the apache as a frontend tls server is pretty rare on big sites nowadays, I'm feeling pretty good about what did happen, vs what could have happened.

[apaprocki]

It seems like a "feature" that you'd want your process to store the private key material as low as possible in the address space so that arbitrary read overruns don't run the risk of hitting it. It seems to just accidentally be this way in nginx, but I wonder if it should just be another (tiny) layer in the overall security design.

[conformal]

nice write up. we saw similar results in that the keying material never made it into the memory leaked.

i've never felt so thankful for a memory allocation pattern.

[danielweber]

What is the format of the private key?

I.e., is it a 8192-bit AES256 key?

[d0ne]

We will add a $10,000 bounty for the first published and confirmed successful completion of this challenge.

Conditions:

1) CloudFlare confirms success.

2) The winner publishes their solution, including source, publicly.

3) Promptly send the link to the publication to [adam | ionicsecurity | com] (for tracking the order of submissions)

Good luck!

[indutny]

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Hello Adam!

My email is public: [email protected] , and here is my proof of identity: https://keybase.io/indutny .

If you wish - you could contact me via it. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1

iQIcBAEBAgAGBQJTSJ7IAAoJEPsOEJWxeXmZYnkP/1r9qVARb89x2bAteh9RPcRI VGUmRZVz/1wLqy/LmvB+XwgkEGRyjQBCa2vHPi8PpwenLEl8IXMMYyzQSqx94tkV y54ZTwABtXXcPIaFOu4O8sG8RM6rDVsF9FpJICVxuYzrkyQPVDMEFa3faBNEgTHo zpgOf5keNq3nCnhTwhkMryBzYVVMLUdQy6JUzhzOXTarmgNH5CtRW0CzFN+9sxM9 6/EH1W4VcJt0lpcvCQK75Kv9syrbavB6qXP85b0gSvKcMHvkX0z5dPphUJcyL/9Q QyXE2vloNj6qwjLQRPoCymSjePeQsodhec47iQxVgil72U5X4YFYJpDHurE5KVOb VIGjmiXhAcL7M8MgywNNtP9iIsi45WiOBmNQVYrBr3/37TSL6FFMfpFVuOxxVrNV fRKRx7VFihTyYxqacwBLAkNPQ6V4QiEdEt74DQFZsokgk1dcchP4GrSypNbrM4SX SJW0RwqF1t44mvuAHmt0U6otgzKy4XyjmDGvki6FNE8ww+OIEQX6tgRPSTD0bURn PyNtZ1EKYZguQt3b4pveVK8JMgWxuCcO9LgKFbPTZJ8YBYOTCU6WtTm4OfCdnTG7 1EtOv6c2k5nlOiK11K8M9ZPEkjq//6C0MZFn1CB7/43+tkWDTr/vSayW/6Yi8pF5 /C1vMZXz3MmpY/gj9z+W =mH1/ -----END PGP SIGNATURE-----

[suyash]

That is awesome Adam, please have it posted on the Challenge page so that more people can see it. Hopefully someone solves it soon :)

[tptacek]

Didn't Juliano Rizzo already post that he'd been able to extract keys from a server? I'm not clear on the circumstances; for instance, it might have been right after boot, and it might have been Apache, and it might have been FreeBSD.

[wglb]

Errata security modified their original post to say that yes they are extractable. And a moderator properly fixed the title after the article was changed to reflect the retraction:

https://news.ycombinator.com/item?id=7561399

[conformal]

i suspect the memory accessible by this bug depends a lot on the software, OS and possibly hardware, e.g. on openbsd and bitrig amd64, the amount of memory leaked per exploit is less than 64 KB, closer to 32 KB. if you go much past the 32 KB mark on these OSes, it segfaults.

running an exploit script against one of our own services showed only 1-2 KB of information, most if it being the (public) cert, and the rest zeroed out.

[quasque]

I wonder if Cloudflare is logging all data sent to and from the challenge server, and then searching it for private key fragments. If not, they should be! As it's not guaranteed that a lucky attacker will be aware of receiving key material, if it is exposed.

[danielweber]

It's cool seeing other people's attempts to extract the key in my return buffer.

It's like multiplayer microcorruption.com

[tptacek]

Which, FWIW, is in the works.

[kaivi]

This is fun, indeed.

Has anyone already made a patch for this bug, where the lib returns random data instead of actual heap chunks?

[aus_]

Someone is submitting false keys in their POST data :D

  -----BEGIN RSA PRIVATE KEY-----
  MIIEowIBAAKCAQEAwYUYN1fR2/z+Net8DdDhNR7poeX608gwFNDVTQZNjldwi503
  cLCaFbPpGkkh84sxpsL0PqrMx9OprdNaMPe0ewuTjjZyb1MaqzfXz/XCiOaT7kWZ
  P9bRae2cI5CETZHD/CvWjsxZ+0exFRSGtph+3esDuvwGQprfYY8d2PIpPOFDgIW7
  gCX6/xTpB+3PCt8lQYbrYiOIogrI+1xpqp5WBCHRzL6La/Zhav1Q6rrSHk1woGdk
  gJBNr3k8NsOjFfNbJgLTgsOz7Yu7Wh6UiV7Y5BMxzuK0eMlJey9ODkyFwhG/pnMg
  qeMf4j1bfaA9AdI8sffSd+1hY9tbt4vfZhBphQIDAQABAoIBACGh/glwS9rN52h7
  mr3T9bADxVsL20H/YsvbV5bhZ407iSXalw4Qw1bOQ877stje8iwnaceSa7C3CqBZ
  QS2Nf2J7KpPmxpCugs4LEIjmD383g7I/iljb/t1IHIDZ1wScNtGW1N1QirrvRMmD
  Hb6SVko/VBClSHk+7V/JPci8nYok1vWytXyi/QJ8EaQ5FPglm3n+QHBnFdl3A8qn
  cWpK3t6BYijkJAK5PNIc1g0YxkSerG5hwCqWleRt29V5QPZqgEc0raSFolN0XOo6
  8us1z37IDRGvm0sHbGgKfi3/Mj+b3Npa+0cQgsGVmymLjyMSf/cLiF0c3rDpHamY
  NqsFj8ECgYEA4JHeIM0M/MVCbUaqCFOtGYmRK9CL8eoAGVJgXa96Z+gP7W/bAjw9
  IEe9NeGIwr2KidZljyv23CwK1Fn7nDuloG5ZXvzpP3jNRJ62Z0Y6RkICpWgpN1HL
  uUG6R4LlB3mXyYE4UGoNAxfgWRcPAn0vsKIuY5/N69FBXiE9h6cjHVkCgYEA3Jq8
  v/qx7T73p9iUFAFDUNF/poQShqimGdfubMSbhUBXGkx5hPyKYUrzUwwZTlLwo4SB
  +nP1cuJUdnEY+ZlzSxvh/EADvxiq20aUm7R+o+GXc68JZBmUKg+tQVSTw+Us8uds
  w1jhFa+b4UpSR+ZHOhR5Hhiw1mdRPu5TAwdtzA0CgYBJpQxnTVdY/YePNoFEQcye
  d1K4P6flKYvmwfEASC+oK54ti2UTEpYpI92U8eVbl7T7Bug/T3ehNIBIq0msLpNe
  TVHZqK8FNaR7QGEo9QrfR35+p+38SkK4+ikQkohxLxrr5giFtoGV5hqYnMs7Ubp1
  /nmgCiQJlZIOQHBhMBp/4QKBgEog3OgNz5gPHp0SE8AdJIeVvjUwECGbriTlAEMg
  VboTArHdnL/pNH649sajPCWrjR0FO7/zEzlFgGhEXMYSoEFO30MqJ5ghK9h/ARTJ
  M/zRo1jGKnuudLElj2xyhSzAZ7g/t7Z0uT3WQqUTmk56vNhxZLmORm3lTdG9t7s+
  1dBJAoGBAJWKz/T/0PQLOOggof0KzEciehWuLcV0eLid16DaNWwiVkm88zt54MdB
  FkbsXWJpQxYpKCxt4XaZGBVP5yLIkrWjElWMtrZPh3pYR8WJQTwosv74n7roBt80
  9ejTR8GQ7401BS7foMjCnVevrl2UwN5SPvtYsLLEMxYGGgSwfz9D
  -----END RSA PRIVATE KEY-----

[tomvangoethem]

Note that by visiting, your IP and referer become accessible by anyone running an heartbleed exploit:

  93.142.x.x - - [11/Apr/2014:10:44:36 -0400] "GET /heartbleed HTTP/1.1" 200 1148 "https://news.ycombinator.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36"

[webXL]

Whoops, too late. Were you able to get that info from actually running the exploit?

[conformal]

what i am expecting ppl to see is that you can't actually get to the tls private key itself. we have done some testing with our backup service, cyphertite, and have yet to attack and actually compromise any keying material.

EDIT: forgot to cite neel mehta https://twitter.com/neelmehta/statuses/453625474879471616

[jnbiche]

You all are involved in so many great open source projects -- just wanted to give a shout-out to Conformal. People do notice and appreciate your contributions.

How many full-time people are working there?

[wglb]

This post first agreed, then upon further review retracted that finding: https://news.ycombinator.com/item?id=7561399

[simias]

It's a good idea but even if nobody successfully exploits this particular website with the heartbleed bug doesn't mean much for the rest of the vulnerable sites.

Since the bug exposes a few kilobytes of uninitialized malloc() memory the kind of data the attacker will retrieve is heavily dependent on the software the server is running.

[ygra]

I read elsewhere that the bug exposes first and foremost memory that OpenSSL itself used before (because OpenSSL has its own allocator running on top of malloc).

[51Cards]

Is this really an accurate challenge? I have wondered if the true exposure risk from Heartbleed may be over-stated* due to memory separation between processes, etc. This however is probably a clean server with a fairly static install. There isn't a risk of things like session leakage which I think is the true risk of Heartbleed. Nor would there be the memory fragmentation that would occur in a production system. While intriguing I'm not sure how much it proves?

(* this is still good though as any risk is unacceptable and shouting Fire makes everyone move... and they need to on this issue)

[jerf]

It may be the case that Heartbleed did not practically expose private keys. The risk was not overstated in general, though; a virtually undetectable method that has been proved to do things like leak plaintext username/password credentials is still catastrophic.

[jgrahamc]

There's no memory separation between a web server that incorporates OpenSSL and... itself.

[thedufer]

Anything OpenSSL decrypts surely has to hit the OpenSSL process's memory, right? That would include requests, which could easily contain sessions, passwords, etc.

[51Cards]

Much kudos to the guy who cracked it later in the day. I'm glad this proved a useful test. I gladly stand corrected.

[mikeash]

I wonder if one could use this server (or, of course, any other vulnerable server, but the hostname on this one is nice) for phishing people. For example, e-mail them the link, then try to snarf the referer header out of the server's memory in the hopes that their webmail URL contains something juicy, or give them a fake form that POSTs interesting data to it. Probably far-fetched, but it's amusing to consider.

[apaprocki]

I seriously don't expect anyone will find anything (reasons mentioned in the blog post), but that doesn't mean I haven't pointed my exploit that searches for private key material in the return buffer at it on a one second interval. If anything I suppose CloudFlare will be able to produce some pretty pictures of the amount of incoming heartbeats they received!

[forgotAgain]

A different CloudFlare post on HN https://news.ycombinator.com/item?id=7572666 claimed that they had a fix for the HeartBleed bug 12 days ago.

At CloudFlare, we received early warning of the Heartbleed vulnerability and patched our systems 12 days ago.

I commented on that post that the date of discovery was 7 days ago http://www.vocativ.com/tech/hacking/behind-scenes-crazy-72-h...

For whatever reason that HN post was deleted and resubmitted so it now has no comments on it. https://news.ycombinator.com/item?id=7572796

Was the bug discovered 12 days ago or 7?

[eastdakota]

The bug was discovered by whitehat researchers approximately 12 days ago. It was publicly announced 5 days ago. We got early word of it from the researchers who initially discovered it, allowing us to patch our systems and ensure all sites behind CloudFlare were not vulnerable. However, we have no way of knowing how long blackhats may have had it. It had been present in the OpenSSL software for the last 2+ years. Therefore we're trying to get an informed sense of the security risks. Hence our work attempting to use the vulnerability to recover SSL private keys and, as announced today, the CloudFlare Challenge.

[dcu]

I'm sure this bug was discovered at least 1 month ago. I would not trust my keys are safe even if using cloudflare.

[umrashrf]

Look what I found...

  0390: 00 00 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C  ..OLOLOLOLOLOLOL
  03a0: 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C  OLOLOLOLOLOLOLOL
  03b0: 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C  OLOLOLOLOLOLOLOL
  03c0: 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C  OLOLOLOLOLOLOLOL
  03d0: 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C  OLOLOLOLOLOLOLOL
  03e0: 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C  OLOLOLOLOLOLOLOL
  03f0: 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C  OLOLOLOLOLOLOLOL
  0400: 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C  OLOLOLOLOLOLOLOL
  0410: 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C  OLOLOLOLOLOLOLOL
  0420: 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C 4F 4C  OLOLOLOLOLOLOLOL

[dredge]

For the command to match the description, shouldn't that be "echo -n"? Otherwise the signed string would include a trailing newline.

I do not expect this will make a material difference to the challenge - presumably you used the quoted commands to generate the answer.

[pornel]

There is no portable way to echo without a newline. Use printf instead.

[jkrems]

Maybe they changed the description in the meantime, it now includes the newline, saying: "Proof I have your key\n"

[koliber]

So out of curiosity, I ran the python heartbleed tester against their server. Surprisingly, the first (and every response thereafter) returned a memory dump which contained at least part of what looked like a private key. I immediately became skeptical when each of these apparent private keys ended in LOLJK.

  0690: 79 37 70 71 31 76 63 2F 74 70 49 67 68 4C 4F 4C  y7pq1vc/tpIghLOL
  06a0: 4A 4B 3D 3D 2D 2D 2D 2D 2D 45 4E 44 5F 52 53 41  JK==-----END_RSA
  06b0: 5F 50 52 49 56 41 54 45 5F 4B 45 59 2D 2D 2D 2D  _PRIVATE_KEY----

Seems like a clever setup to return hand-crafted responses.

[zenojevski]

Got that too:

  1ef0: 4E 63 46 39 33 51 75 34 64 2F 48 6E 47 6D 4B 50  NcF93Qu4d/HnGmKP
  1f00: 51 42 72 4A 43 50 53 69 54 67 70 2F 63 79 37 70  QBrJCPSiTgp/cy7p
  1f10: 71 31 76 63 2F 74 70 49 67 68 4C 4F 4C 4A 4B 3D  q1vc/tpIghLOLJK=
  1f20: 3D 2D 2D 2D 2D 2D 45 4E 44 25 32 30 52 53 41 25  =-----END%20RSA%
  1f30: 32 30 50 52 49 56 41 54 45 25 32 30 4B 45 59 2D  20PRIVATE%20KEY-
  1f40: 2D 2D 2D 2D 20 48 54 54 50 2F 31 2E 31 0D 0A 48  ---- HTTP/1.1..H

Curiously enough, one of the first responses, after which they became somewhat predictable (with the occasional "herring").

Seems like some experimenting (based on differing offset, and `%20` vs `_`).

[haldean]

I got a private key without the LOLJK, but I'm 99% certain this is someone trolling me.

Still going to double-check though.

[AshleysBrain]

Would a multi-process server engine help protect against this? Think what Chrome does with tabs. If the network request is received by a dedicated IO process which then uses IPC to communicate with other parts of the server, then perhaps sensitive information like keys would not be in the same address space so could not be leaked? I guess if the bug was in a sensitive process then it would still happen. Disclaimer: I have no idea how modern servers are architected, perhaps they already do this :) Would be interested to hear from anyone more knowledgeable.

[X-Istence]

The keys would be in the same process as the one doing the SSL in the first place, and that is the part that is vulnerable.

Even communicating over IPC you would still be vulnerable.

[rubiquity]

If what CloudFlare is saying is true (and I think it is), that the only possibility of Nginx/Apache leaking the private key is on start up due to how low of a memory address the private key is given, is there anything in Nginx/Apache to fire a bunch of dummy requests at itself before accepting public connections? This should effectively bury the memory address that the private key is stored in. Would this be useful if it doesn't exist or would it only be useful in hindsight?

[kurokikaze]

Someone POSTs the "BEGIN SECRET KEY" strings and such to server. Funny thing is, key is probably stored as binary and without file header/footer.