Python programmers often need to rely on multiprocessing to speed up their code. In multiprocessing new processes are spawned and executed simultaneously, effectively side-stepping the infamous global interpreter lock. There are several Python libraries that offer multiprocessing capabilities and remove the need to write all the boilerplate code to work with processes and inter-process communication. Yet, even with all these libraries around, none of them were working to my satisfaction. Notably, I was missing a simple Pythonic syntax, stateful computation, user friendly exception handling and intuitive progress bar information. So, four years ago, I’ve created a new multiprocessing library for Python, called MPIRE (MultiProcessing Is Really Easy). After several iterations of feedback and exposure to production environments, it has become the go-to multiprocessing library at Slimmer AI. Recently, we’ve made it publicly available on GitHub and PyPI. This talk provides an introduction to MPIRE. I will cover the basic and more advanced features of MPIRE, how easy it is to use, and will benchmark it against some of the existing libraries. Spoiler: it consistently beats all other multiprocessing libraries in terms of speed.
About Sybren Jansen
As Head of AI Innovation at Slimmer AI, I lead the AI fellowship program – advancing our AI techniques and capabilities. I work closely with our 10+ machine learning engineers to stay up to date on the latest AI innovations and to develop new AI solutions for our ventures.
I have 7 years of experience in rapid prototyping applied-AI solutions. During this time I’ve worked on a range of different problems including computer vision, time series, and NLP. In the last few years, I have focused specifically on NLP. During my working life Python has been, and still is, my programming language of choice. But I’ve had my fair share of exposure to Cython and C++ as well.
Even though machine learning plays a big part in my day-to-day work, it’s not the only thing I’m interested in. Writing optimized and fast code is something I always strive for when programming and I’ve spent many years trying to master the art.
Find out more about the co-building approaches and portfolio of Slimmer.ai