slopfront
preferences

Multiprocessing in async function?

17 points · submitted May 2025 by u/RationalDialog · Question

My goal is to build a webservice for a calculation. while each individual row can be calculated fairly quickly, the use-case is tens of thousands or more rows per call to be calculated. So it must happen in an async function.

the actual calculation happens externally via cli calling a 3rd party tool. So the idea is to split the work over multiple subproccess calls to split the calculation over multiple cpu cores.

My question is how the async function doing this processing must look like. How can I submit multiple subprocesses in a correct async fasion (not blocking main loop)?

17 comments · 100% upvoted · other discussions

comments

Blakex123 9 points · May 2025

Remember that python is inherintly single threaded due to the GIL. You can mitigate this by running fastapi with multiple workers. The requests will then be spread over those different workers.

mrbubs3 5 points · May 2025

You can turn GIL off in 3.13

Asleep-Budget-9932 9 points · May 2025

That feature is experimental and should not be used in production environments.

bbrother92 1 points · May 2025

api requiest are dipenced to dif workers? not the treads?

Blakex123 3 points · May 2025

If u are using uvicorn there is an extra process made that essentially "load balances" the 4 workers. I assume it works the same way with any other server.

RationalDialog [OP] 1 points · May 2025

The requests will then be spread over those different workers. my use case is few requests but each one very heavy. I want each request to run faster, eg do the calculation using multiple cpu cores.

Blakex123 1 points · May 2025

Then u will need to spawn subprocesses from the api to handle the cpu intensive stuff.

adiberk 7 points · May 2025

You can use asyncio tasks.

You can also use a more standard product like celery.

RationalDialog [OP] 2 points · May 2025

You can also use a more standard product like celery.

Yeah I wonder if I should forget about async completely (never used it really so far as no need) and build more kind of a job system. If someone submit say 100k rows, the job could take approx 5 min to complete.

adiberk 1 points · May 2025

Yep that works to. If you are doing a lot of other IO operations, it might be worth making the app async based anyways (ie. Keyword async)

AstronautDifferent19 1 points · May 2025

asyncio to_thread is better for CPU bound tasks than asyncio.create_task, especially if you disable GIL.
asyncio tasks will always block if you do CPU heavy work, which will not work for OP.

adiberk 1 points · May 2025

Good point

KainMassadin 4 points · May 2025

don’t sweat it, just call asyncio.create_subprocess_exec and you’re good

AstronautDifferent19 1 points · May 2025

This is the way.

KainMassadin 1 points · May 2025

that one can be risky, gotta sanitize properly

jimtoberfest 1 points · May 2025

Find a vectorized solution across all rows if you can.

Take in a json array then load that data into a dataframe or numpy array and figure out your calculation using inherently vectorized operations.

Or you could “stream” it: fast api -> duckDB-> do the calc in duckDB over the chunks as you get them from the API.

Also make sure you set some limits so users can’t bomb the API with billions of rows of data.

RationalDialog [OP] 1 points · May 2025

The calculation happens in a 3rd party executable. This is the core limitation. Hence why I need sub process calls, to call multiple instances of this 3rd party executable which is 32-bit hence no way to integrate it more tightly.

jimtoberfest 1 points · May 2025

Oof yeah that’s rough. As long as the .exe runs in diff instances then use multiprocessing and processPoolExecutor library.

Just split it up by how many cores you have // 2.

I find that roughly works the best.