Advanced usage of Python requests: timeouts, retries, hooks

[inetknght]

I just recently dealt with a timeout issue in requests: I was setting the timeout to 1 but it took 10 seconds for the request to fail. It turns out that the timeout does not apply to the name resolver and the name lookup was timing out. So the default system-level timeout was 10 seconds.

[hmhrex]

Thank you for this brief reference. I'm always going back to my old projects to remind myself how I did things with requests. I didn't realize some of things existed.

[dhxt]

You're welcome. I thought I'd aggregate all the useful tricks I'd found while working on my last project. It's a collection of snippets I had found in the docs and in Github issues.

The post contains the most important snippets, but you might find the advanced section of the requests docs [1] useful and the requests_toolbelt [2].

[1] https://requests.readthedocs.io/en/master/user/advanced/

[2] https://toolbelt.readthedocs.io/en/latest/

[sciurus]

I wish that requests had a default timeout instead of defaulting to waiting forever.

It's very easy for devs to forget to think about timeouts, but inevitably that ends in a cascading production outage.

[dhxt]

I'm not sure if you noticed, but the post contains a section on how to set default timeouts.

https://findwork.dev/blog/advanced-usage-python-requests-tim...

[jakeogh]

If I enable logging of retries: logging.basicConfig(level=logging.DEBUG)

I dont see any retries on response 200: ConnectionError(ReadTimeoutError("HTTPSConnectionPool(host='domain.com', port=443): Read timed out."))

Is there a way to also retry on ChunkedEncodingError?

[ddeokbokki]

That sentry responses decorator is absolutely beautiful

[da12]

Fantastic reference. Thanks for sharing!