Existing implementations of thread pools have a relatively high overhead in certain situations.
ThreadPoolExecutor at all (see the benchmarks in the docs directory).
In case of
ThreadPoolExecutor don’t use
It can be extremely slow! If you’ve only a small number of jobs and the jobs have a relatively
long processing time, then these overheads don’t count. But in case of high number of jobs with
short processing time the overhead of the above implementations will noticeably slow down the
processing speed. The
fastthreadpool module solves this issue, because it has a very small
overhead in all situations.
fastthreadpool is lightweight it has some additional cool features like methods for
later scheduling, repeating events and generator functions for worker callback functions.
In addition to get the best performance I’ve also written a fast and lightweight semaphore which is more than 20 times faster than the one which comes with the Python installation.
Some reasons why
fastthreadpool is so fast:
- Avoid locks as much as possible
- Use deque instead of Queue
- Do not create a class instance for every work item
The first test in benchmarks.py has a minimum worker callback function which just returns the given parameter. The main thread calculates the sum of the returned values. This is the most extreme case where the overhead of the thread pool implementation counts as much as possible. This is not the typical use case but it shows the overhead of the different thread pool implementations very well.
The results show that
ThreadPool performs good, but
fastthreadpool is a bit
apply_async has a very bad performance, even worse
fastthreadpool performs very well.
A more typical case shows the last example where the worker threads serialize and compress data.
As you can see the worker threads are still 2 times faster with
fastthreadpool when submitting
single jobs to the pool than the other 2 thread pool implementations.
Again this example shows clearly if speed matter then avoid
concurrent.futures. Although it has a
nice interface it is really slow.
For examples how to use
fastthreadpool please have a look at the examples directory.
Check out the fastthreadpool module on github, licensed under the MIT license.