Mitmproxy is so awesome, I use it all the time! Can recommend it for anyone who needs to do some fiddling with HTTP(S) traffic for debugging / reverse engineering purposes.
I'm slightly too much of a GUI person to use it in a "explorative" phase, then I'm more a fan of Charles/Proxyman, but whenever I wanna modify the traffic or do some scripting to i.e log certain traffic, I reach for Mitmproxy.
I'd highly recommend https://httptoolkit.tech/ for that explorative GUI phase. I found it recently and the rule configuration, UI and interception setup is significantly better than Charles/Fiddler/Proxyman.
Thanks for the kind words! We have a Google Summer of Code student working with us on mitmweb, our web interface, at the moment, so we'll hopefully have a better GUI story soon as well :)
I just used this app the other week do dig into what I think is a bug with rclone. Seeing the exact http traffic to / from the AWS s3 endpoint made it much simpler to understand what was going on. The CLI is super intuitive, it's a great application all around.
Mitmproxy was an interesting solution to a problem I had a few weeks ago. One of my IT support gigs was trying to use "Mevo"-branded cameras to stream some live events to Facebook using an iPad. We had to use the venue's Wi-Fi network. We found the Mevo app would not detect that we were "connected to the Internet" and wouldn't allow us to stream.
The venue's network admin said they weren't filtering anything outbound to the Internet. We could access websites from the iPad just fine. Same w/ Facebook, Youtube, etc.
I put up Mitmproxy, an adhoc Wi-Fi network on a second Wi-Fi NIC, a DHCP server, and iptables NAT on my laptop. I set the iPad to use my adhoc SSID and my machine as an explicit HTTP proxy. My intention was to snoop on the traffic to determine what Mevo was using for criteria to consider being "connected".
For whatever reason the Mevo app just started working in this configuration. I ended up sticking a spare Wi-Fi router and a PC w/ Mitmproxy installed in the venue for the duration of the event. I never did figure out what the Mevo app didn't like about the venue's network.
If I had to guess why you had issues, the app probably had certificate pinning. Apple makes it pretty easy to pin domains, but it's also not too difficult to circumvent it.
If I had to guess, there may have been some sort of captive portal in place. I've seen even the "invisible" kind that don't require user intervention cause interference with connectivity checks for some devices.
Does anyone know how the story currently is for MITM'ing Android?
I had fun reverse engineering app APIs a few years back, but I looked into it again more recently and found that Android has started pinning certificates by default even on apps that didn't pin anything themselves. I also had trouble getting my custom certificate to be used at all, when it used to be pretty easy.
Can Frida still bypass pinning, even this new default one, or is it done at OS level?
* By default, all modern Android apps only trust CA system certificates, and on a normal device you cannot change those.
* You can change system certificates on rooted devices and most (but not all) emulators.
* Apps can opt-in to trusting user-installed CA certificates within their manifest config, but almost all don't.
If you're on a non-rooted device, the only way to MITM traffic is by modifying the application itself, so that it opts into trusting your cert. You can either inject Frida-Gadget to do that, do it manually, or use https://github.com/shroudedcode/apk-mitm which tries to do it automatically (but it's a bit hit & miss whether it works).
That's separate to explicit certificate pinning. That's also possible on Android (with some standard config settings, or manually in code) although it's got much less popular I think since the defaults were tightened up. In that case you do usually need Frida, and it's generally easiest to just use a rooted device. I wrote a blog post about exactly how to do that last week: https://httptoolkit.tech/blog/frida-certificate-pinning/#how...
How practical is it (or not) to intercept TLS traffic for IoT devices on my network where I don't have the option of setting a proxy?
I've been on a crusade the past few years to try and get my self-signed CA onto "smart" devices in my home, but with some it's difficult or I risk damaging expensive hardware.
I've tried various ARP poisoning techniques and such in the past with limited success.
mitmproxy is a great tool, but it lacks upstream certificate lookup and attributes transfer to a generated host certificate. For example, Charles does that. Some applications perform attribute inspection to detect MITM proxies, namely Spotify. Are you planning to implement this feature? Or maybe you already did it in version 7?
Thanks for Mitmproxy! It would be amazing to be able to see non-HTTPS (but still TLS) traffic in the Flows view, and even to intercept it live instead of having to drop to raw TCP mode and write Python. Any idea whether that's on the roadmap?
If I wanted to use mitmproxy to mitm and observe traffic during automated testing, what mode/settings should I be using on Windows?
The setup is: A client (dev machine or pipeline) running tests that communicate to a Windows VM that hosts the app under test. The client RPCs to call automation APIs on the host/server.
However I couldn't get both communicating and using mitmproxy to observe traffic to work at the same time.
I know that's not the purpose of mitmproxy, but wondering if it's possible to create automation scripts, like for puppeteer or export requests to curl/httpie/vscode rest client. I tried through web-interface but couldn't find anything like.
I am still using version 2.0 after the big changes in shortcuts. I used mitmproxy weekly, now I have switched to mostly using Charles, but would love to switch back.
Any plans for a recommended shortcut setup again?
Not writing 30 characters for copy body of a POST request, but fast key bindings.
Is anyone using this for page rewriting? Not just for the all-important ad blocking but removing the whitespace where the removed ad was, rewriting autoplay videos and gifs into click-to-play, etc. So many apps have web views that the doing it in the browser is a step backwards.
I used to do all my rewrite via a proxy but it stopped being particularly useful when https became almost universal.
It's getting harder and harder to do even with HTTPS! Look at the other commenter on here that described the absolute wasteland with apps and OS-vendors (Android,IOS) being actively hostile to device-owners.
I’ve used it to diagnose (and work around) webhook problems between third-party applications that are supposedly compatible.
One that sticks out in my mind is trying to receive Slack webhooks in Mattermost. I was unaware of Slack’s new BlockKit message format until I saw what was being sent. Then I wrote a simple python script to transform the payload, all within mitmproxy. It’s not the most efficient long term solution, but the fact I could diagnose and resolve the issue with a single tool, all within a few hours of first discovering it, was very satisfying.
Here’s the result for anyone interested (warning: it’sa quick hack, not a polished/maintained project)
Edit: this doesn’t use TLS at all. I understand that’s probably the primary use case for the tool, but I thought this was a cool example of other ways it can be useful too.
Scripting flow manipulations in mitmproxy is easy. There are a lot of great examples to learn from which help. The filter syntax is expressive and simplifies writing code to mangle flows. I've done body conversion [0], weird types of authorization testing, manipulated system update packages on the fly, all sorts of bizarre things. Auto-reloading scripts on change is such a time saver. If you want a python-based programmable HTTPS proxy for testing I have not found anything better.
We have a product that depends on data received from external web services. We often want to test certain conditions in the data (specific results, error scenarios, etc), and it's difficult or impossible to guarantee these conditions when we call out to our external services. We use mitmproxy to intercept and manipulate the responses we get.
Even if we don't need to manipulate the data, it's a very useful tool for watching requests go back and forth.
I wrote a krunkerio aimbot that unpacks the msgpack-over-websocket packets and injects the calculated mouse movements into the stream.
The biggest problem was the server only syncing the player position every second or so which means my middleware has can only provide exact calculations when I stand still. Solving this would require reading the process memory but that is exactly what I wanted to avoid :/
I've introduced mitmproxy in my team and we use it quite often for flows' inspection and manipulation while testing a desktop app. Can't count cases where a quick look into a proxy made debugging a lot easier, because we were able to point at unexpected requests / responses.
Thank you for your work, @mhils & the team. Enjoy your moments of glory on HN :)
I use it to download music from my music streaming subscription to my son's iPod. None of the files have any tags or metadata so it's was an interesting challenge to get everything tagged.
I’ve used this tool to identify and fix security issues and have also taught software testers how to use it just to spread the love and knowledge. Kudos!
[+] [-] filleokus|4 years ago|reply
I'm slightly too much of a GUI person to use it in a "explorative" phase, then I'm more a fan of Charles/Proxyman, but whenever I wanna modify the traffic or do some scripting to i.e log certain traffic, I reach for Mitmproxy.
[+] [-] MostKaleido|4 years ago|reply
[+] [-] mhils|4 years ago|reply
[+] [-] GekkePrutser|4 years ago|reply
[+] [-] res0nat0r|4 years ago|reply
[+] [-] EvanAnderson|4 years ago|reply
The venue's network admin said they weren't filtering anything outbound to the Internet. We could access websites from the iPad just fine. Same w/ Facebook, Youtube, etc.
I put up Mitmproxy, an adhoc Wi-Fi network on a second Wi-Fi NIC, a DHCP server, and iptables NAT on my laptop. I set the iPad to use my adhoc SSID and my machine as an explicit HTTP proxy. My intention was to snoop on the traffic to determine what Mevo was using for criteria to consider being "connected".
For whatever reason the Mevo app just started working in this configuration. I ended up sticking a spare Wi-Fi router and a PC w/ Mitmproxy installed in the venue for the duration of the event. I never did figure out what the Mevo app didn't like about the venue's network.
[+] [-] derimagia|4 years ago|reply
This looks to cover it pretty well: https://www.guardsquare.com/blog/leveraging-infoplist-based-...
[+] [-] Forbo|4 years ago|reply
[+] [-] pedrovhb|4 years ago|reply
I had fun reverse engineering app APIs a few years back, but I looked into it again more recently and found that Android has started pinning certificates by default even on apps that didn't pin anything themselves. I also had trouble getting my custom certificate to be used at all, when it used to be pretty easy.
Can Frida still bypass pinning, even this new default one, or is it done at OS level?
[+] [-] pimterry|4 years ago|reply
* By default, all modern Android apps only trust CA system certificates, and on a normal device you cannot change those.
* You can change system certificates on rooted devices and most (but not all) emulators.
* Apps can opt-in to trusting user-installed CA certificates within their manifest config, but almost all don't.
If you're on a non-rooted device, the only way to MITM traffic is by modifying the application itself, so that it opts into trusting your cert. You can either inject Frida-Gadget to do that, do it manually, or use https://github.com/shroudedcode/apk-mitm which tries to do it automatically (but it's a bit hit & miss whether it works).
If you're on an emulator/rooted device, it's totally possible, see https://httptoolkit.tech/blog/intercepting-android-https/#in... for how that works.
That's separate to explicit certificate pinning. That's also possible on Android (with some standard config settings, or manually in code) although it's got much less popular I think since the defaults were tightened up. In that case you do usually need Frida, and it's generally easiest to just use a rooted device. I wrote a blog post about exactly how to do that last week: https://httptoolkit.tech/blog/frida-certificate-pinning/#how...
[+] [-] mhils|4 years ago|reply
[+] [-] alias_neo|4 years ago|reply
I've been on a crusade the past few years to try and get my self-signed CA onto "smart" devices in my home, but with some it's difficult or I risk damaging expensive hardware.
I've tried various ARP poisoning techniques and such in the past with limited success.
[+] [-] monai|4 years ago|reply
[+] [-] cjbprime|4 years ago|reply
[+] [-] rsgrn|4 years ago|reply
The setup is: A client (dev machine or pipeline) running tests that communicate to a Windows VM that hosts the app under test. The client RPCs to call automation APIs on the host/server.
However I couldn't get both communicating and using mitmproxy to observe traffic to work at the same time.
[+] [-] c6401|4 years ago|reply
[+] [-] chmaynard|4 years ago|reply
[+] [-] punnerud|4 years ago|reply
Any plans for a recommended shortcut setup again?
Not writing 30 characters for copy body of a POST request, but fast key bindings.
[+] [-] aritmo|4 years ago|reply
Suppose I have set an interception filter and I am viewing a live packet. I am about to edit it, etc.
But when a new packet arrives, the text UI moves to the new packet! I have lost the packet that I was looking a few seconds ago.
Is this an issue in mitmproxy or am doing things the wrong way?
[+] [-] gumby|4 years ago|reply
I used to do all my rewrite via a proxy but it stopped being particularly useful when https became almost universal.
[+] [-] zo1|4 years ago|reply
Edit. Typo.
[+] [-] ericb|4 years ago|reply
Anyone interested in sharing use-cases where it has helped you?
[+] [-] thinkmassive|4 years ago|reply
One that sticks out in my mind is trying to receive Slack webhooks in Mattermost. I was unaware of Slack’s new BlockKit message format until I saw what was being sent. Then I wrote a simple python script to transform the payload, all within mitmproxy. It’s not the most efficient long term solution, but the fact I could diagnose and resolve the issue with a single tool, all within a few hours of first discovering it, was very satisfying.
Here’s the result for anyone interested (warning: it’sa quick hack, not a polished/maintained project)
https://github.com/thinkmassive/mattermost-webhook-slack-pro...
Edit: this doesn’t use TLS at all. I understand that’s probably the primary use case for the tool, but I thought this was a cool example of other ways it can be useful too.
[+] [-] rbdixon|4 years ago|reply
[0]: https://carvesystems.com/news/proxying-wbxml-services/
[+] [-] mpicker0|4 years ago|reply
Even if we don't need to manipulate the data, it's a very useful tool for watching requests go back and forth.
[+] [-] mhils|4 years ago|reply
Development:
- Debug your own apps/devices (which do not come with Chrome Devtools).
- Simple Python scripts to replace/inject website content. (https://docs.mitmproxy.org/stable/addons-overview/)
- Capture a web page/API and then replay the responses offline. (https://docs.mitmproxy.org/stable/overview-features/#server-...).
- Replay requests for testing. (https://docs.mitmproxy.org/stable/overview-features/#client-...)
Privacy:
- Figure out what your Covid app is sending out (https://seancoates.com/blogs/how-i-helped-fix-canadas-covid-...)
- Look at how evil IoT devices for pets are (https://www.nytimes.com/2018/12/11/realestate/spying-on-the-...)
- Generally find more evil stuff (https://mitmproxy.org/publications/)
Security:
- Reverse-engineering custom binary protocols is never fun, but with mitmproxy you can at least write custom contentviews to dissect stuff. (https://github.com/mitmproxy/mitmproxy/blob/main/examples/ad...)
- Fiddle with generic TLS-protected TCP streams. (https://github.com/mitmproxy/mitmproxy/blob/main/examples/ad...)
- Dump TLS master secrets for WireShark. (https://docs.mitmproxy.org/stable/howto-wireshark-tls/)
[+] [-] no_time|4 years ago|reply
The biggest problem was the server only syncing the player position every second or so which means my middleware has can only provide exact calculations when I stand still. Solving this would require reading the process memory but that is exactly what I wanted to avoid :/
[+] [-] KORraN|4 years ago|reply
Thank you for your work, @mhils & the team. Enjoy your moments of glory on HN :)
[+] [-] turtlebits|4 years ago|reply
[+] [-] Proven|4 years ago|reply
[deleted]
[+] [-] beermonster|4 years ago|reply
[+] [-] eklavya|4 years ago|reply