Add a page on parallel and async tools.

[samcunliffe]

Solves #178.

No async yet, but there are a couple of projects using async so I'll ask around for opinions.

[paddyroddy]

I understand the philosophy behind the traffic light system, but one can easily use i.e. numba and multiprocess in conjunction - so not sure if it makes too much sense.

[matt-graham]

Thanks @samcunliffe this looks good. Have made some suggestions about adding a bit more detail around different parallelism models in Python and briefly mentioning global interpreter lock to explain why we can't just use threading module. I don't if worth specifically adding a separate section on libraries with support for running code on accelerator devices like GPUs? Of the tools already mentioned JAX and Numba I think have built-in support for running code on GPUs. Other options similar to JAX would be CuPy, TensorFlow and PyTorch, and there also options like pycuda, cuda-python and pyopencl.

[dstansby]

Maybe this could be split into thread/process based parallelism (multiprocess, multiprocessing, dask, mpi4py I think?) and (JIT) compiler based parallelism (numba, jax, Cython)?

[paddyroddy]

Worth mentioning https://docs.python.org/3/library/concurrent.futures.html too

[samcunliffe]

@paddyroddy

Worth mentioning https://docs.python.org/3/library/concurrent.futures.html too

I'm gonna need a traffic light colour for that, old bean.

[paddyroddy]

I'm gonna need a traffic light colour for that, old bean.

I'm a fan! 🎐

[paddyroddy]

@matt-graham GIL may go in the future https://peps.python.org/pep-0703

[matt-graham]

@matt-graham GIL may go in the future https://peps.python.org/pep-0703

Yep I linked to that PEP in my suggested change above 😉

EDIT: Potentially misunderstood your comment, was the point that is not certain and so not worth linking to?

[paddyroddy]

Yep I linked to that PEP in my suggested change above 😉

EDIT: Potentially misunderstood your comment, was the point that is not certain and so not worth linking to?

To be honest, it is so many words that my brain skipped some of it... I feel like it is going into a lot of detail that won't be relevant for most users. However, perhaps it could be useful in a <details> block?

But yes it is also may fall out of date with the Python ecosystem, so may just be worth linking to the PEP?

[samcunliffe]

I guess we might then have to recommend a specific tool for compiler based parallelism which might require some thought / discussion though.

Left them as all-amber for now. I'd be inclined to factor out the discussion of our favorites to either a new ticket, or to an in-person meetup.

[paddyroddy]

Worth mentioning docs.python.org/3/library/concurrent.futures.html too

Part of standard library too

[matt-graham]

To be honest, it is so many words that my brain skipped some of it...

I wouldn't say ~150 words is all that many, and irrespective I feel it's not unreasonable to expect you to read things you comment on before commenting on them, even if you find them overly wordy 🙁. Not trying to make a big deal of this, but comments like the above do make it less appealing to contribute and I would say we should keep comments kind and constructive.

I feel like it is going into a lot of detail that won't be relevant for most users. However, perhaps it could be useful in a <details> block?

But yes it is also may fall out of date with the Python ecosystem, so may just be worth linking to the PEP?

Even if/when PEP0703 gets fully implemented the GIL will be optional but not removed so a lot of the mentioned details will still be relevant. I think having some explanatory comments is useful as while we want to keep thing simple for users, wanting to exploit parallelism / asynchronicity is a relatively advanced use case, and is inherently a complex topic and I think trying to hide away the complexity doesn't necessarily help. Putting some extra detail in a <details> block I think would be a good approach.

[paddyroddy]

I wouldn't say ~150 words is all that many, and irrespective I feel it's not unreasonable to expect you to read things you comment on before commenting on them, even if you find them overly wordy 🙁. Not trying to make a big deal of this, but comments like the above do make it less appealing to contribute and I would say we should keep comments kind and constructive.

I did read them... What I am saying is that is why I missed you'd already linked to the issue. We all have different reading and interpreting styles.

[samcunliffe]

To stop this PR snowballing, I'm factoring out the asynchronous stuff to

• Add asynchronous processing libraries. #331