scandir-rs

Because speed matters…

With the increased speed of SSDs single threaded file access does not always fully utilize the disk. The bottleneck more and more is the CPU itself.

When I started learning the great new language Rust I came across a crate called jwalk. This crate does directory scanning in parallel with a thread pool. The benchmarks are amazing. So I thought I write a Python module as a faster alternative to os.walk which uses jwalk. The result is the new module scandir-rs which can be found on pypi.org.

The API is a bit different and provides more features. But it should be easy to replace os.walk and os.scandir with scandir-rs.

Usage examples

Get statistics of a directory:

import scandir_rs as scandir
print(scandir.count.count("~/workspace", metadata_ext=True))

The same, but asynchronously in background using a class instance and a context manager:

import scandir_rs as scandir
C = scandir.count.Count("~/workspace", metadata_ext=True))
with C:
    while C.busy():
        statistics = C.statistics
        # Do something

os.walk() example:

import scandir_rs as scandir
for root, dirs, files in scandir.walk.Walk("~/workspace", iter_type=scandir.ITER_TYPE_WALK):
    # Do something

os.scandir() example:

import scandir_rs as scandir
for entry in scandir.scandir.Scandir("~/workspace", metadata_ext=True):
    # Do something

Benchmarks

Now let’s have a look at some benchmarks. In the below table the line scandir_rs.walk.Walk returns comparable results to os.walk.

Linux with Ryzen 5 2400G and SSD

Directory ~/workspace with

• 22845 directories
• 321354 files
• 130 symlinks
• 22849 hardlinks
• 4 devices
• 1 pipes
• 4.6GB size and 5.4GB usage on disk

Up to 2 times faster on Linux.

Windows 10 with Laptop Core i7-4810MQ @ 2.8GHz Laptop, MTF SSD

Directory C:\Windows with

• 84248 directories
• 293108 files
• 44.4GB size and 45.2GB usage on disk

Up to 6.7 times faster on Windows 10.

Directory C:\testdir with

• 185563 directories
• 1641277 files
• 2696 symlinks
• 97GB size and 100.5GB usage on disk

Up to 17.4 times faster on Windows 10.

Check out the scandir-rs module on github, licensed under the MIT license.