ⓘ

There are many libraries for traversing directories. You can also do this using the standard library. What makes this library different:

• 💎 Beautiful, laconic syntax.
• ⚗️ Filtering by file extensions, text patterns in .gitignore format, and using custom callables.
• 🐍 Natively works with both Path objects from the standard library and strings.
• ❌ Support for cancellation tokens.
• 👯‍♂️ Combining multiple crawling methods in one object.

Table of contents

• Installation
• Basic usage
• Applying a function to each path
• Filtering
• Working with Cancellation Tokens
• Combination
• Transactionality

Installation

You can install dirstree with pip:

pip install dirstree

You can also use instld to quickly try out this package and others without installing them.

Basic usage

The library is easy to use:

• Create a crawler object, passing the path to the base directory and, if necessary, additional arguments.
• Iterate through it.

The simplest example would look like this:

from dirstree import Crawler

crawler = Crawler('.')

for file in crawler:
    print(file)

↑ This recursively prints all files in the current directory, including files in nested directories. At each iteration, we get a new Path object.

Applying a function to each path

If you just want to run a function for each file the crawler finds, you don't have to write the loop yourself — every crawler has an apply() method:

Crawler('src', exclude=['tests/**']).apply(print)

↑ This will print the entire contents of the directory, except for the excluded locations.

ⓘ All of the crawler's settings are respected, exactly as they would be during normal iteration.

Filtering

By default, crawlers iterate over files only. If you need every filesystem entity found under the base directory, pass only_files=False:

crawler = Crawler('.', only_files=False)

Iterating through the files in the directory, you may not want to view all files, but only files of a certain type. To do this, ignore all other files. How to do it? There are three ways:

• Bypass only files with the specified extensions, such as .txt, .doc, or .py.
• Bypass files whose paths follow a specific text pattern.
• Use an arbitrary function to determine whether you need each specific path or not.

To select a specific method, you need to pass a specific parameter when creating the crawler object. Of course, all the methods can be combined with each other.

To set the file extensions you are interested in, use the extensions parameter:

crawler = Crawler('.', extensions=['.txt'])  # Iterate only on .txt files.

ⓘ The extensions parameter is available only in the default file-only mode, so it cannot be combined with only_files=False.

Also, if you only need Python files, you can use a special class to bypass them only, without specifying extensions:

from dirstree import PythonCrawler

crawler = PythonCrawler('.')  # Iterate only on .py files.

ⓘ PythonCrawler is always file-only.

To specify which files and directories you do NOT want to iterate over, use the exclude parameter:

crawler = Crawler('.', exclude=['.git', 'venv'])  # Exclude ".git" and "venv" directories.

↑ Please note that we use the .gitignore format here.

If you need a universal way to filter out unnecessary paths, pass your function as the filter parameter:

crawler = Crawler('.', filter=lambda path: len(str(path)) == 7)  # Iterate only on paths that are 7 characters long.

Working with Cancellation Tokens

You can set an arbitrary condition under which file traversal will stop using cancellation tokens from the cantok library.

There are two ways to do this ↓

1. If you use the crawler as a one-time object for a single iteration, set the token when creating it:

for path in Crawler('.', token=TimeoutToken(0.0001)): # Limit the iteration time to 0.0001 seconds.
    print(path)

2. If you plan to use the crawler object several times, use the go() method for iteration and pass a new token to it every time:

crawler = Crawler('.')

for path in crawler.go(token=TimeoutToken(0.0001)): # Limit the iteration time to 0.0001 seconds.
    print(path)

↑ Follow these rules to avoid accidentally "baking" an expired token inside a crawler object.

By default, cancellation stops iteration silently — the caller cannot tell it apart from natural exhaustion. Pass raise_on_cancel=... to make the crawler raise an exception on cancellation instead:

for path in Crawler('.', token=TimeoutToken(0.0001), raise_on_cancel=True):
    print(path)

↑ raise_on_cancel=True re-raises the native cantok exception; raise_on_cancel=MyError("...") raises that exact instance; raise_on_cancel=MyError instantiates the class with the cantok message and raises that. Default is False (silent).

Combination

You can combine multiple crawler objects into one using the usual addition operator, like this:

for path in Crawler('../dirstree') + Crawler('../cantok'):
    print(path)

↑ The paths that you will iterate over will be automatically deduplicated.

↑ You can also impose arbitrary restrictions on each of the summed objects, all of them will be taken into account.

You can also pass multiple paths to a single crawler object:

for path in Crawler('../dirstree', '../cantok'):
    print(path)

↑ In this case, there is no deduplication of paths.

Transactionality

If you plan to modify the directory while iterating over it — for example, deleting or moving files inside an apply() callback — pass freeze=True to take a snapshot of every matching path up front, then iterate that snapshot instead of the live filesystem:

Crawler('path/to/directory', freeze=True).apply(lambda p: p.unlink())

↑ The snapshot is built on the first step of iteration, with every filter and cancellation token already applied. After that, any creation, renaming or deletion happening in the directory does not affect what is yielded — each call to go() or iter() produces its own fresh snapshot.

↑ Without freeze=True the order of yielded paths depends on the live state of the filesystem, so mid-iteration mutation may silently skip or duplicate entries.