I've been using httpx 0.9.3 in production now for a couple months. I switched from requests when I realized I needed async support and it has been a dream to use.
The only issue I've run into has been with my attempt to reuse the same AsyncClient to make multiple concurrent requests to the same remote host. It looks like this issue may have been fixed in 0.10 or 0.11 so I'll be upgrading soon to check.
Also, be sure to check out the other fantastic projects by Encode. https://github.com/encode I stumbled upon httpx after using Starlette and Uvicorn for one of our microservices and and been pleasantly surprised by how easy to set up and use they are.
I'm currently using aiohttp. There is features I don't find in butterfly docs like : limit connection, limit connection per host or per time frame.
Do you know if those exist in butterfly?
I'm already deep in aiohttp and it's not an easy task to learn an async client (at least in my case, but I'm no dev) so if I do switch it would be for more features (retry option on exceptions, limit requests per host and per time frame...). But that's only my opinion.
This is a huge win compare to requests. AFAIKT requests is too flexible (read: easier to misuse) and difficult to add type annotation now. The author of requests gave a horrible type annotation example here [0].
IMO at this time when you evaluate a new Python library before adopting, "having type annotation" should be as important as "having decent unit test coverage".
The huge win against requests is that Https is fully async.. you can download 20 files in parallel without not too much effort.
Throughout in python asyncio is amazing, something similar to node or perhaps better... That's the main point.
> "having type annotation" should be as important as "having decent unit test coverage".
I mean this is pretty spot on IMO. I've worked with many languages, and have concluded that having a powerful type system catches soooo many bugs before you even try to run the code.
And they're usually "stupid" bugs too, forgetting to sanitize inputs etc. Even worse is when a language tries to be "smart", so you end up with "1" + 2 = "12" and no errors at all.
That's from 2015, his second complaint about interfaces vs inheritance is solved in recent versions with Protocols. Compatibility with 2.7 shouldn't be argument today either.
The complex union is as you say more a sign of a too flexible api rather than problem with type hints, the type hints just brings the root issue to the surface. I mean, why would you accept either a mapping or a list of tuples as input? Just let the user call dict(..) on the tuple-list first if they have it in such format? The documentation doesn't even mention that lists are ok for headers, only dicts: https://2.python-requests.org/en/master/api/#main-interface.
The file-tuple api with various length tuples is perhaps valid and the most convenient way to implement such options, but it's still an exceptionally unique api which requires exceptional type hints, it can be made slightly simpler which chc demonstrated above.
This framework looks incredibly simple and powerful. I’ve coded a bunch of these features manually in the past. I think FastAPI is going to earn a spot in my next project!
In some high-level languages even BSD sockets (and many other POSIX functions) aren't in the standard library, and there are various wrappers to provide "ease of use" and integration with a language's runtime system; plenty of complexity (and alternatives) even at that point.
RFC 2616 (HTTP/1.1, 1999) may seem manageable, but it's much more than just posting data and getting a response, and IME many programmers working with HTTP aren't familiar with all of its functionality. Then add TLS with SNI, cookies, CORS, WebSocket protocol, various authentication methods, try to wrap it into a nice API in a given language and not introduce too many bugs, and it's rather far from trivial. But that's just HTTP/1.1 with common[ly expected] extensions.
Edit: Though I think it'd also be controversial to add support for particular higher-level protocols into standard libraries of general-purpose languages, even if it was easy to implement and to come up with a nice API.
Plenty of reason why it's hard to ship in standard library. Here's some off top of my mind:
- Should the library includes its own CA store, or use the system's CA store? These kind of library often include their own CA store (since they changes often), and httpx seem to use 3rd party lib to handle that (certifi). This is hard to do in a standard library for variety of reasons (users rarely update their python installation, system CA store is not always available/up to date, etc).
- While the http protocol itself is pretty stable, some part of it are still changing overtime. Things like compression types (brotli is gaining traction these days, and we might get a new compression types in the future), new http headers added, etc. Security issue also show up all the time. The user will want tighter release schedule than python's so they can get these stuff sooner. The situation is even worse for users that stuck in a particular version of python for some reason since they now won't have access to these new update ever.
It's simple. Python was created before HTTP even existed. Since then a lot of things had changed, and once you create an API, it is hard to rewrite it when new use patterns emerge.
It's easier for a 3rd party package to come up with a better api, because they can start brand new. Also when there's a radical change it is easier for a new 3rd package to take over. Httpx is example of such evolution, although not due to changes in http but this time changes in Python, it makes use of new functionality that's harder to implement in requests, mainly adding async support and type annotations.
httpx makes a pretty good first impression on me. The homepage provides examples, key selling points, install instructions, and links to any further reading one could hope for. However, I am missing one thing: The features this offers over the popular requests library do not seem to require httpx to be a competitor, but an extension or fork. There surely must be some major incompatibilities that allowed this library to do something fundamentally different, right?
> The features this offers over the popular requests library do not seem to require httpx to be a competitor
Yes it has to be. Requests is not as great as everyone thinks it is. It's API is simple, sure, but when you need more advanced features (doesn't even support HTTP/2 AFAIK) like timeouts, proper exception handling (which you cannot do in requests)) it actually sucks.
I know this is unrelated, but probably a lot of Pythonistas here and I've been wondering: what is the async web framework of choice for you guys today? As for DBs, still SQLAlchemy? What about a prettier (js) alternative?
I've been using FastAPI https://github.com/tiangolo/fastapi which is built on top of Starlette as my main async framework and Uvicorn as my primary web server.
Starlette and Uvicorn are both made by Encode https://github.com/encode and in my experience they consistently put out quality stuff.
I used aiohttp and it is quite good and solid (I also like that it comes with type annotations, which enables autocomplete in PyCharm), I did not have chance to compare it to FastAPI.
Regarding database access, my recommendation might not be popular, but I really like asyncpg[1]. They basically dropped DBAPI and instead created their own API that functionally matches PostgreSQL, giving you full control and increasing performance.
As a result of that you probably won't be able to use ORMs (there's SQLAlchemy driver, but as I understand it is hacky and you are losing the performance benefits).
Personally I compared my code with and without ORM and there's no change in number of lines. If you use PyCharm and configure it to connect to your database (I think that might be a pro feature) it will then automatically detect SQL in your strings provide autocomplete (including table and column names) and provide highlighting, removing all the advantages that ORM provided.
I've moved to Trio over asyncio (I did plain old async for a couple years and Trio makes a ton of sense)
Quart-trio over Flask (just to get a Trio-friendly flask-a-like server) - plain old aiohttp worked really well too. It takes a bit more roll-your-own work, but you get exactly what you want.
peewee over SQLAlchemy (less committed to this change, but peewee has been fine so far and is much more streamlined) I'm mostly just using SQLite. the async version of the ORM looks pretty new, i'm not using it yet.
Looks like this does not include certifi [0] and loads system certificates by default. This is a breath of fresh air to see, because so many packages want to use their own certs and have a custom system to override it to use system certs.
Edit: Well, looks like it does use certifi. But my grumble still stands, I don't understand why does everyone want to mess with your certs.
I want to mess with certs in Python so that my web crawler can actually access the whole web. If you don't talk to a wide variety of hosts, you probably haven't noticed that it's broken for 1%.
One thing requests still has going for it is that it’s used under the covers by a lot of client libraries with a dynamic import, allowing you to pass in a session object to control things like connection pooling and custom headers.
I do look forward to httpx becoming the new “standard” though. Tom is a great developer, and his ecosystem of tools are going to have a really big impact in python web dev over the coming years.
Too bad they carried over the biggest nuisance from requests, having to call raise_for_status() after every single request. One extra line everywhere and another thing that can be forgotten and cause strange errors when you least expect it.
I like async support, but always, in async examples, doesn't the await keyword just block anyways?
I like to see the usage of async inside of a main loop, where other things can be processed while waiting for the async response to come in.
Further, not all applications have the idea of an event loop, so async may not be needed, but it's useful to have the option.
I use async operations to multiplex operations in a queue. The linux scheduler can handle the execution time slices for me, I'm not going to build a scheduler, but the queue controller's role is to accept jobs and handle timeouts and results of each async operation.
The await keyword only blocks the execution of the coroutine in which it is used. It releases the event loop so that other coroutines can continue processing while the result of an awaitable is being fetched.
Comparing the code samples, it looks like it's easier to spin up testing with Httpx than Flask, but that's a superficial conclusion. I don't have experience with Httpx so take this with a grain of salt. Based on the comparison, I'm going to play with Httpx for testing the next time I use Flask because the simplicity looks rad.
Based on similar experience with other tools, that possibly means that Httpx is great for simple testing but if you need to go deep it's better to use the framework provided. That's an assumption, though, so I'd love to hear more from others.
My project has a dozen or so libraries, many of which require different and conflicting requests, urllib3, etc., libraries. I have to setup the packages very carefully to ensure the right module gets the right version of its HTTP library and it has generally turned me off to Python HTTP libraries altogether. For my own code, I stick to the basic built-in libraries, regardless of how difficult to use they are.
I use intensively requests-cache, that add local caching (in sqlite) for python requests and is configurable (should it cache non 200 response, should it cache POST ...) and it's amazing when scrapping. I didn"t find a similar companion package for httpx.
I immediately wondered the same, and I think not. The upper wings are not at a 90 degree angle to the body, so alive and well. Apart from being a black and white drawing of course.
daze42|6 years ago
The only issue I've run into has been with my attempt to reuse the same AsyncClient to make multiple concurrent requests to the same remote host. It looks like this issue may have been fixed in 0.10 or 0.11 so I'll be upgrading soon to check.
Also, be sure to check out the other fantastic projects by Encode. https://github.com/encode I stumbled upon httpx after using Starlette and Uvicorn for one of our microservices and and been pleasantly surprised by how easy to set up and use they are.
vajenetehais|6 years ago
Do you know if those exist in butterfly?
I'm already deep in aiohttp and it's not an easy task to learn an async client (at least in my case, but I'm no dev) so if I do switch it would be for more features (retry option on exceptions, limit requests per host and per time frame...). But that's only my opinion.
yegle|6 years ago
This is a huge win compare to requests. AFAIKT requests is too flexible (read: easier to misuse) and difficult to add type annotation now. The author of requests gave a horrible type annotation example here [0].
IMO at this time when you evaluate a new Python library before adopting, "having type annotation" should be as important as "having decent unit test coverage".
[0]: https://lwn.net/Articles/643399/
jordic|6 years ago
penagwin|6 years ago
I mean this is pretty spot on IMO. I've worked with many languages, and have concluded that having a powerful type system catches soooo many bugs before you even try to run the code.
And they're usually "stupid" bugs too, forgetting to sanitize inputs etc. Even worse is when a language tries to be "smart", so you end up with "1" + 2 = "12" and no errors at all.
Too|6 years ago
The complex union is as you say more a sign of a too flexible api rather than problem with type hints, the type hints just brings the root issue to the surface. I mean, why would you accept either a mapping or a list of tuples as input? Just let the user call dict(..) on the tuple-list first if they have it in such format? The documentation doesn't even mention that lists are ok for headers, only dicts: https://2.python-requests.org/en/master/api/#main-interface.
The file-tuple api with various length tuples is perhaps valid and the most convenient way to implement such options, but it's still an exceptionally unique api which requires exceptional type hints, it can be made slightly simpler which chc demonstrated above.
tiangolo|6 years ago
bransonf|6 years ago
This is a really awesome library, thanks!
carapace|6 years ago
pensatoio|6 years ago
Svenstaro|6 years ago
keshab|6 years ago
EamonnMR|6 years ago
Spiritus|6 years ago
quasarj|6 years ago
thesehands|6 years ago
hannse|6 years ago
A friend wrote respx https://github.com/lundberg/respx which is a utility to mock HTTP requests made by httpx. It works similar to the requests mocking library responses https://github.com/getsentry/responses
teruakohatu|6 years ago
https://vorpus.org/blog/why-im-not-collaborating-with-kennet...
unknown|6 years ago
[deleted]
pbreit|6 years ago
On every single project I do, it's just a bunch of posting JSON and getting a response synchronously. Over and over.
defanor|6 years ago
In some high-level languages even BSD sockets (and many other POSIX functions) aren't in the standard library, and there are various wrappers to provide "ease of use" and integration with a language's runtime system; plenty of complexity (and alternatives) even at that point.
RFC 2616 (HTTP/1.1, 1999) may seem manageable, but it's much more than just posting data and getting a response, and IME many programmers working with HTTP aren't familiar with all of its functionality. Then add TLS with SNI, cookies, CORS, WebSocket protocol, various authentication methods, try to wrap it into a nice API in a given language and not introduce too many bugs, and it's rather far from trivial. But that's just HTTP/1.1 with common[ly expected] extensions.
Edit: Though I think it'd also be controversial to add support for particular higher-level protocols into standard libraries of general-purpose languages, even if it was easy to implement and to come up with a nice API.
neurostimulant|6 years ago
- Should the library includes its own CA store, or use the system's CA store? These kind of library often include their own CA store (since they changes often), and httpx seem to use 3rd party lib to handle that (certifi). This is hard to do in a standard library for variety of reasons (users rarely update their python installation, system CA store is not always available/up to date, etc).
- While the http protocol itself is pretty stable, some part of it are still changing overtime. Things like compression types (brotli is gaining traction these days, and we might get a new compression types in the future), new http headers added, etc. Security issue also show up all the time. The user will want tighter release schedule than python's so they can get these stuff sooner. The situation is even worse for users that stuck in a particular version of python for some reason since they now won't have access to these new update ever.
takeda|6 years ago
It's easier for a 3rd party package to come up with a better api, because they can start brand new. Also when there's a radical change it is easier for a new 3rd package to take over. Httpx is example of such evolution, although not due to changes in http but this time changes in Python, it makes use of new functionality that's harder to implement in requests, mainly adding async support and type annotations.
guggle|6 years ago
Felk|6 years ago
takeda|6 years ago
Will need to check how it compares with aiohttp which is quite good and also has these.
kissgyorgy|6 years ago
Yes it has to be. Requests is not as great as everyone thinks it is. It's API is simple, sure, but when you need more advanced features (doesn't even support HTTP/2 AFAIK) like timeouts, proper exception handling (which you cannot do in requests)) it actually sucks.
httpx is a far superior library already!
bbmario|6 years ago
daze42|6 years ago
Starlette and Uvicorn are both made by Encode https://github.com/encode and in my experience they consistently put out quality stuff.
takeda|6 years ago
Regarding database access, my recommendation might not be popular, but I really like asyncpg[1]. They basically dropped DBAPI and instead created their own API that functionally matches PostgreSQL, giving you full control and increasing performance.
As a result of that you probably won't be able to use ORMs (there's SQLAlchemy driver, but as I understand it is hacky and you are losing the performance benefits).
Personally I compared my code with and without ORM and there's no change in number of lines. If you use PyCharm and configure it to connect to your database (I think that might be a pro feature) it will then automatically detect SQL in your strings provide autocomplete (including table and column names) and provide highlighting, removing all the advantages that ORM provided.
[1] https://github.com/MagicStack/asyncpg
carlosf|6 years ago
https://github.com/tiangolo/fastapi
rtny4821|6 years ago
j88439h84|6 years ago
https://www.starlette.io/
oefrha|6 years ago
Not enough experience with async to comment.
> DB
peewee is good enough and more ergonomic compared to SQLA for a lot of use cases.
> formatter
black. To be clear it often produces truly horrendous code, but at least there’s no arguing and no fussing over options or details.
vpribish|6 years ago
I've moved to Trio over asyncio (I did plain old async for a couple years and Trio makes a ton of sense)
Quart-trio over Flask (just to get a Trio-friendly flask-a-like server) - plain old aiohttp worked really well too. It takes a bit more roll-your-own work, but you get exactly what you want.
peewee over SQLAlchemy (less committed to this change, but peewee has been fine so far and is much more streamlined) I'm mostly just using SQLite. the async version of the ORM looks pretty new, i'm not using it yet.
holler|6 years ago
bwooster|6 years ago
workthrowaway|6 years ago
also, sqlalchemy is an over-engineered system imo. i only go for it when i have no other choices. otherwise i use a database client directly.
thijsvandien|6 years ago
fjp|6 years ago
hyzyla|6 years ago
rajasimon|6 years ago
neuland|6 years ago
Edit: Well, looks like it does use certifi. But my grumble still stands, I don't understand why does everyone want to mess with your certs.
[0] https://pypi.org/project/certifi/
Twirrim|6 years ago
greglindahl|6 years ago
ra5|6 years ago
Looking forward to trying it out
Edit: maybe the homepage can include a very simple async example as well?
takeda|6 years ago
Rapzid|6 years ago
lilydjwg|6 years ago
Also I ran into hard-to-debug issues when there were lots of requests in the past. I'll check again with httpx soon.
jsmeaton|6 years ago
I do look forward to httpx becoming the new “standard” though. Tom is a great developer, and his ecosystem of tools are going to have a really big impact in python web dev over the coming years.
Too|6 years ago
unknown|6 years ago
[deleted]
florimondmanca|6 years ago
[0]: https://github.com/encode/httpx/issues/752
ryanmccullagh|6 years ago
I like to see the usage of async inside of a main loop, where other things can be processed while waiting for the async response to come in.
Further, not all applications have the idea of an event loop, so async may not be needed, but it's useful to have the option.
I use async operations to multiplex operations in a queue. The linux scheduler can handle the execution time slices for me, I'm not going to build a scheduler, but the queue controller's role is to accept jobs and handle timeouts and results of each async operation.
_bohm|6 years ago
The await keyword only blocks the execution of the coroutine in which it is used. It releases the event loop so that other coroutines can continue processing while the result of an awaitable is being fetched.
oefrha|6 years ago
Question: is there any advantage of this over flask’s builtin werkzeug test client?
[1] https://www.python-httpx.org/advanced/#calling-into-python-w...
[2] https://flask.palletsprojects.com/en/1.1.x/testing/
grammarxcore|6 years ago
Based on similar experience with other tools, that possibly means that Httpx is great for simple testing but if you need to go deep it's better to use the framework provided. That's an assumption, though, so I'd love to hear more from others.
sheerun|6 years ago
theon144|6 years ago
Will definitely be checking out and potentially replacing requests and aiohttp.
speedplane|6 years ago
hypewatch|6 years ago
julienfr112|6 years ago
mproud|6 years ago
https://emilydamstra.com/news/please-enough-dead-butterflies...
korijn|6 years ago
Great package though. Love the dual support for async and sync requests.
bouke|6 years ago
Congeec|6 years ago
daze42|6 years ago
bproven|6 years ago
toyg|6 years ago
dhruvkar|6 years ago
daze42|6 years ago
mariocesar|6 years ago
tus88|6 years ago
Well that's going to work wonders for async now isn't it?