Discovering private APIs with Charles.app

[bri3d]

I really wish Charles generated a self-signed cert on installation, rather than using a root that you download from them.

I suspect that a lot of people in the iOS dev community (where Charles seems ubiquitous) walk around with the Charles root on their phone, ripe for an easy malicious MITM against them.

I'm also surprised more iOS and Android apps don't bundle and pin their certificates - it's still an obscurity measure since, worst comes to worst, the user can root / jailbreak the device and attach a debugger or watch the network stack, but it keeps any random user with Charles (or random malicious attacker with a stolen root cert) from reversing private APIs.

[beamatronic]

I believe Fiddler does this

edit: Details on the process: http://blogs.telerik.com/fiddler/posts/13-08-19/faq---certif...

Relevant Quote: "Every Fiddler root certificate is uniquely generated, per user, per machine. No two Fiddler installations have the same root certificate. The only way for a Fiddler user to be “spoofed” by a bad guy is if that bad guy already is running code inside the user’s account (which means you’d already be pwned anyway)."

[sprite]

Certificate pin bypass for Android: https://github.com/iSECPartners/Android-SSL-TrustKiller

iOS: https://github.com/iSECPartners/ios-ssl-kill-switch

Sure it's a little bit extra work but not much. If someone is reversing your app chances are they are using a jailbroken device anyways to extract the unencrypted IPA or to attach gdb to your app.

[timrogers]

I agree. I'm glad people don't though, as I like de-privatising "private APIs" ;)

[passfree]

You can use Proxy.app (http://www.websecurify.com/desktop/proxy.html) which does that. As for the certificate pinning - it makes proxying a very difficult.

[fmela]

mitmproxy does this.

[sjtgraham]

I've used MITMproxy for this. It's great fun to MITM every app on your phone, you'll find some really interesting stuff. I happened across a WSDL in the British Airways app that appeared to enable booking flights as if done by a member of staff back of house. It might have been possible to book flights and not pay for them, obviously I didn't go as far to test this. But like I say, interesting stuff.

Tip: Some apps do SSL pinning so the handshake will fail with the cert that your proxy provides. You can disable any kind of SSL cert checks on a jailbroken iOS device with SSL kill switch (https://github.com/iSECPartners/ios-ssl-kill-switch)

[timrogers]

Never seen MITMproxy - how does it compare to Charles? I've found super interesting stuff - pretty much invariably find that (a) there's cool stuff you can do and (b) the app's developers haven't gone to significant lengths to stop you doing it.

[Mandatum]

Interesting, if you hadn't already reported it, I'd go ahead and do that shortly.

[dozy]

Yep, Charles is a great proxy. Fiddler for Windows is a great free option as well.

Warning: the author of this blog very nonchalantly instructs readers to install the Charles certificate. If readers don't know what this does, it can be quite dangerous. Next time your device connects to a Wifi network that you don't control, you could very well be going through somebody else's proxy and have all of your https traffic sniffed!

Better to sign your own certificate and use that instead! Or, at least uninstall the Charles cert from your device after you've had your fun sniffing traffic on your own network.

[timrogers]

Fair point on the Charles certificate. Not something to keep installed! I'll add that now.

[voltagex_]

Fiddler will generate its certificate on the fly for you, reducing the risk a fair amount.

[lkrubner]

I would add, sometimes the "private API" that you would like to debug is the one that you are creating. I have found Charles to be essential whenever I'm asked to enable CORS so that browsers can make cross-domain Ajax calls to the server where I'm creating software. For complex requests (and any PUT or DELETE) the browsers will do a "preflight" OPTIONS request to see if the request is allowed. Strangely enough, the CORS spec actually encourages browser-makers to hide (from the Javascript client) the OPTIONS request. FireFox hides the OPTIONS request completely -- and so this is one of the few times that FireBug failed me. FireBug never sees the request, so it is unable to tell me about it. Getting CORS right usually means seeing what is in the OPTIONS request, and what your own response is (I mean, the response of the server software that you are writing), and I found Charles extremely useful for that bit of debugging. (cURL will also fail you in this case, as cURL is not limited the way Ajax requests are limited -- the cURL request will always work, so it won't tell you why your Ajax call is failing.)

[leoh]

If you are building an app and want to protect against this kind of thing, you can use SSL pinning. This means that you hard-code your API server's public SSL key into your app and explicitly reject and other public keys, even if they are installed of trusted by the OS. Incidentally, the twitter iOS app does this — you can't observe its traffic via Charles or other snoopers.

[martinald]

It's not really a protection though. With a jailbroken phone you could probably change the key with a hex editor or similar. There's also this: https://github.com/iSECPartners/ios-ssl-kill-switch

Also, you're totally screwed if you need to reissue your SSL cert for a security problem (think heartbleed). You'd have to reissue the cert, wait for apple to approve the update, then hope that a significant proportion of your users actually update your app.

[coffeecheque]

Ah - that might explain why I had trouble trying to sniff the data for QuizUp too.

I didn't know you could do that, so thanks!

[thoughtpalette]

I typically use Fiddler (Windows) for this. Dig the article. Very clear steps on how to get setup and start playing around.

I remember a similar article about how the author intercepted the API requests for CandyCrush and was able to give himself lives and whatnot. Pretty neat.

[dozy]

I sent a friend of mine about 2000 Yo's (Charles will show you that the Yo app uses Parse...very simple API requests) with Charles's request-replay feature - quite easy to do and oh so funny...or at least I thought so.

[timrogers]

Thanks! I think this is a super interesting thing to do - there's tonnes of great stuff to discover - and something that's not very well documented online. I wanted to make it easy for anyone to do this kind of thing.

[Chris911]

I like Charles and have been using it for this exact purpose for a while now. I'm still wondering what's the best approach from a developer perspective to get around that and keep a private API private. Even if you build a different key for each request on Android you can work your way through obfuscated code and rebuild the logic. If the app requires the user to login running your own OAuth server can be a solution but are there any easier solutions?

[timrogers]

The short answer is "no" - you're always playing a cat and mouse game, so you're wiser not to put things behind an API that you're really not happy for people to play with.

I suspect this is why we've traditionally seen banks (in the UK, at least) use web-pages-embedded-in-apps rather than true native apps.

[kabdib]

You should expect APIs to be public. For a sufficiently popular service, you should expect people to utterly replace (emulate or rewrite) your fine client and do things that you don't expect, which means implementing things like rate limiting and blacklisting on your server, where you can control things.

[click170]

I get that some developers want to protect what they consider a private API, but consider it from the other side.

As a user, I think I have the right to know that you're not secretly uploading my contact list to your servers. Anything you do to block that prohibits me from trusting your app.

There must be some kind of happy medium where I can protect myself from malicious apps, and developers can protect themselves from malicious users too.

[yellowapple]

I love how the article's closing remark is about releasing source-code, yet the entire rest of the article talks about a closed-source and proprietary program. ;)

Pretty cool idea, though. I reckon this is doable with something like mitmproxy[0], which is open-source, and it would certainly be interesting to poke around in some of these hidden APIs.

[warrenmiller]

mitmproxy is the free way to do this.

[pmhpereira]

Bitdefender blocked me this page, saying it has malware.

[nnd]

I've reversed a private API once and built a an app on top of it. Needless to say, the company behind it wasn't happy and I was looking at a lawsuit.

[maaaats]

It's great for private use, though. Life-hacking by enabling you to instantly book flight-seats when available, for instance.

[drdaeman]

Lawsuit on basis of what? IANAL, but I've heard many times that ToSes/EULAs which forbid you from querying their servers and reverse engineering their apps aren't legally binding in most jurisdictions.

[dalek2point3]

does this method work if the APIs are passing data that is encrypted? I believe Waze does this ...

[narzero]

Great post, thanks.

[freeslugs]

very creative idea. +1

[orf]

Other than providing a nice interface what does Charles.app do that something like Burp can't?

[kybernetikos]

There's often more than one piece of software to do something and that's OK. I've hacked up simple MITM proxies myself more than once in the last 15 years.

Charles is not new, it's been around more than 10 years.

I think that generally saying 'what does X do that Y doesn't' comes across as fairly negative, and unless there's some obvious reason why Y should be the default adds little.

I'd never heard of burp, but I had heard of Charles. Is there some obvious reason why burp should be the default?

[timrogers]

Nothing really - I suspect that Charles is more approachable for the average person though. And besides, I hadn't spotted any alternatives when I wrote the article ;)

[passfree]

Proxy.app from Websecurify is native Mac proxy and it is pretty good too.

[mateuszf]

Or excellent mitmproxy.