|
| 1 | +--- |
| 2 | +draft: false |
| 3 | +date: 2024-02-11 |
| 4 | +categories: |
| 5 | + - DevLog |
| 6 | +authors: |
| 7 | + - willmcgugan |
| 8 | +--- |
| 9 | + |
| 10 | +# File magic with the Python standard library |
| 11 | + |
| 12 | +I recently published [Toolong](https://github.com/textualize/toolong), an app for viewing log files. |
| 13 | +There were some interesting technical challenges in building Toolong that I'd like to cover in this post. |
| 14 | + |
| 15 | +<!-- more --> |
| 16 | + |
| 17 | +!!! note "Python is awesome" |
| 18 | + |
| 19 | + This isn't specifically [Textual](https://github.com/textualize/textual/) related. These techniques could be employed in any Python project. |
| 20 | + |
| 21 | +These techniques aren't difficult, and shouldn't be beyond anyone with an intermediate understanding of Python. |
| 22 | +They are the kind of "if you know it you know it" knowledge that you may not need often, but can make a massive difference when you do! |
| 23 | + |
| 24 | +## Opening large files |
| 25 | + |
| 26 | +If you were to open a very large text file (multiple gigabyte in size) in an editor, you will almost certainly find that it takes a while. You may also find that it doesn't load at all because you don't have enough memory, or it disables features like syntax highlighting. |
| 27 | + |
| 28 | +This is because most app will do something analogous to this: |
| 29 | + |
| 30 | +```python |
| 31 | +with open("access.log", "rb") as log_file: |
| 32 | + log_data = log_file.read() |
| 33 | +``` |
| 34 | + |
| 35 | +All the data is read in to memory, where it can be easily processed. |
| 36 | +This is fine for most files of a reasonable size, but when you get in to the gigabyte territory the read and any additional processing will start to use a significant amount of time and memory. |
| 37 | + |
| 38 | +Yet Toolong can open a file of *any* size in a second or so, with syntax highlighting. |
| 39 | +It can do this because it doesn't need to read the entire log file in to memory. |
| 40 | +Toolong opens a file and reads only the portion of it required to display whatever is on screen at that moment. |
| 41 | +When you scroll around the log file, Toolong reads the data off disk as required -- fast enough that you may never even notice it. |
| 42 | + |
| 43 | +### Scanning lines |
| 44 | + |
| 45 | +There is an additional bit of work that Toolong has to do up front in order to show the file. |
| 46 | +If you open a large file you may see a progress bar and a message about "scanning". |
| 47 | + |
| 48 | +Toolong needs to know where every line starts and ends in a log file, so it can display a scrollbar bar and allow the user to navigate lines in the file. |
| 49 | +In other words it needs to know the offset of every new line (`\n`) character within the file. |
| 50 | + |
| 51 | +This isn't a hard problem in itself. |
| 52 | +You might have imagined a loop that reads a chunk at a time and searches for new lines characters. |
| 53 | +And that would likely have worked just fine, but there is a bit of magic in the Python standard library that can speed that up. |
| 54 | + |
| 55 | +The [mmap](https://docs.python.org/3/library/mmap.html) module is a real gem for this kind of thing. |
| 56 | +A *memory mapped file* is an OS-level construct that *appears* to load a file instantaneously. |
| 57 | +In Python you get an object which behaves like a `bytearray`, but loads data from disk when it is accessed. |
| 58 | +The beauty of this module is that you can work with files in much the same way as if you had read the entire file in to memory, while leaving the actual reading of the file to the OS. |
| 59 | + |
| 60 | +Here's the method that Toolong uses to scan for line breaks. |
| 61 | +Forgive the micro-optimizations, I was going for raw execution speed here. |
| 62 | + |
| 63 | +```python |
| 64 | + def scan_line_breaks( |
| 65 | + self, batch_time: float = 0.25 |
| 66 | + ) -> Iterable[tuple[int, list[int]]]: |
| 67 | + """Scan the file for line breaks. |
| 68 | +
|
| 69 | + Args: |
| 70 | + batch_time: Time to group the batches. |
| 71 | +
|
| 72 | + Returns: |
| 73 | + An iterable of tuples, containing the scan position and a list of offsets of new lines. |
| 74 | + """ |
| 75 | + fileno = self.fileno |
| 76 | + size = self.size |
| 77 | + if not size: |
| 78 | + return |
| 79 | + log_mmap = mmap.mmap(fileno, size, prot=mmap.PROT_READ) |
| 80 | + rfind = log_mmap.rfind |
| 81 | + position = size |
| 82 | + batch: list[int] = [] |
| 83 | + append = batch.append |
| 84 | + get_length = batch.__len__ |
| 85 | + monotonic = time.monotonic |
| 86 | + break_time = monotonic() |
| 87 | + |
| 88 | + while (position := rfind(b"\n", 0, position)) != -1: |
| 89 | + append(position) |
| 90 | + if get_length() % 1000 == 0 and monotonic() - break_time > batch_time: |
| 91 | + break_time = monotonic() |
| 92 | + yield (position, batch) |
| 93 | + batch = [] |
| 94 | + append = batch.append |
| 95 | + yield (0, batch) |
| 96 | + log_mmap.close() |
| 97 | +``` |
| 98 | + |
| 99 | +This code runs in a thread (actually a [worker](https://textual.textualize.io/guide/workers/)), and will generate line breaks in batches. Without batching, it risks slowing down the UI with millions of rapid events. |
| 100 | + |
| 101 | +It's fast because most of the work is done in `rfind`, which runs at C speed, while the OS reads from the disk. |
| 102 | + |
| 103 | +## Watching a file for changes |
| 104 | + |
| 105 | +Toolong can tail files in realtime. |
| 106 | +When something appends to the file, it will be read and displayed virtually instantly. |
| 107 | +How is this done? |
| 108 | + |
| 109 | +You can easily *poll* a file for changes, by periodically querying the size or timestamp of a file until it changes. |
| 110 | +The downside of this is that you don't get notified immediately if a file changes between polls. |
| 111 | +You could poll at a very fast rate, but if you were to do that you would end up burning a lot of CPU for no good reason. |
| 112 | + |
| 113 | +There is a very good solution for this in the standard library. |
| 114 | +The [selectors](https://docs.python.org/3/library/selectors.html) module is typically used for working with sockets (network data), but can also work with files (at least on macOS and Linux). |
| 115 | + |
| 116 | +!!! info "Software developers are an unimaginative bunch when it comes to naming things" |
| 117 | + |
| 118 | + Not to be confused with CSS [selectors](https://textual.textualize.io/guide/CSS/#selectors)! |
| 119 | + |
| 120 | +The selectors module can tell you precisely when a file can be read. |
| 121 | +It can do this very efficiently, because it relies on the OS to tell us when a file can be read, and doesn't need to poll. |
| 122 | + |
| 123 | +You register a file with a `Selector` object, then call `select()` which returns as soon as there is new data available for reading. |
| 124 | + |
| 125 | +See [watcher.py](https://github.com/Textualize/toolong/blob/main/src/toolong/watcher.py) in Toolong, which runs a thread to monitors files for changes with a selector. |
| 126 | + |
| 127 | +## Textual learnings |
| 128 | + |
| 129 | +This project was a chance for me to "dogfood" Textual. |
| 130 | +Other Textual devs have build some cool projects ([Trogon](https://github.com/Textualize/trogon) and [Frogmouth](https://github.com/Textualize/frogmouth)), but before Toolong I had only ever written example apps for docs. |
| 131 | + |
| 132 | +I paid particular attention to Textual error messages when working on Toolong, and improved many of them in Textual. |
| 133 | +Much of what I improved were general programming errors, and not Textual errors per se. |
| 134 | +For instance, if you forget to call `super()` on a widget constructor, Textual used to give a fairly cryptic error. |
| 135 | +It's a fairly common gotcha, even for experience devs, but now Textual will detect that and tell you how to fix it. |
| 136 | + |
| 137 | +There's a lot of other improvements which I thought about when working on this app. |
| 138 | +Mostly quality of life features that will make implementing some features more intuitive. |
| 139 | +Keep an eye out for those in the next few weeks. |
| 140 | + |
| 141 | +## Found this interesting? |
| 142 | + |
| 143 | +If you would like to talk about this post or anything Textual related, join us on the [Discord server](https://discord.gg/Enf6Z3qhVr). |
0 commit comments