Asynchronous HTTP Requests in Python

f'{player_name.replace(" ", "_")}.json'

Finally.

As a rule, async makes sense when you have tasks that require you to wait (e.g. network bound IO) and your program could be doing something useful in the meantime.

In this case, the task is an API request that mostly consists of waiting for a response. "something useful in the meantime" is initiating other API requests.

One thing to note is that API requests are often rate limited, meaning there is a hard limit on the amount of work you can get done with a single API key even if you're program is capable of processing more.

You may be able to get a little more (micro) optimization by sharing the client session, since it uses connection pooling. Recreating it for each call is a bit unnecessary.

magic. do you know a nice guide or explanation for python async stuff? i find it hard to wrap my mind around it

Fuck that. Use gevent:

from gevent import monkey
monkey.patch_all()
requests.get(URL)

I wonder what the runtime would be with multiple threads. In my opinion, that's really what you should be comparing against when it comes to convincing people to use async for io-bound operations.

Is it really the same outcome? How did you test this? Are you able to access players near the end of the list right after you issue the asynchronous call?

Maybe a silly question but how does this differ from Eventlet / Gevent / Twisted (maybe I can't even compare these with each other?)

My knowledge on these libraries is still rather superficial so I can't truly appreciate the depths and uniqueness of each library but if someone would like to explain to me why this library described in the post is worthwhile than please do.

EDIT: Also came across this SO post explaining some differencces between the libraries and other related ones. http://stackoverflow.com/questions/1824418/a-clean-lightweight-alternative-to-pythons-twisted

Good simple example.

Can't help but notice though the complete lack of mention why this is faster and what the implications are.

In both functions you're sending all the requests at once. In get_players that's only one request so it barely matters (in fact you don't really need to async that one at all). In get_player that means you're opening an unknown number of connections simultaneously, which may cause all sorts of issues. Most importantly the API may decide you're violating its policy agains DoS/Spam and ban you.

The most ugly thing there is

asyncio.gather(*
    (get_player(*args) for args in player_args))

I am using the requests library through locust.io and in this configuration the requests library is partially async at least.

It even mentions it in the requests library docs under advanced usage, it won't wait for a response unless you try to look at reponse.content for example.

Personally, I just use libcurl and let it deal with the details. I don't trust anyone to implement HTTP correctly.