v4.0: New Cache System, Resume Timecodes and Performance

Podbit v4.0

Podbit v4.0 contains a major revamp to the cache system, the addition of resume timecodes and a complete rework to the internal bookkeeping system which should improve lookup performance significantly.

There have been a couple of fairly major changes (technically breaking, although the program will automatically upgrade your files for you) which were sitting for a while on Git without a new release being drafted. I thought it was about time that they get pushed, despite a couple of work-in-progress features originally scheduled for the v4.0 tag not yet being ready.

Changelog

1. Podbit's downloaded media cache now has its internal bookkeeping done in a separate file to the newsboat queue file. This change restores full inter-operability with Podboat and allows their use in tandem. This is technically a breaking change but Podbit will perform a migration on first startup after the upgrade.
2. Resume timecodes are now stored in the cache database (detailed in [1]). This allows Podbit to resume playing of a podcast which was stopped part way where it was left off!
3. When re-opening a podcast marked as "played", Podbit will automatically seek to the saved resume timestamp
4. Podbit no longer guesses nor walks a specific download directory. Instead, files passed as part of the queue file are used exclusively.
5. Major improvement to the performance of podcast matching. A map (hash table) structure is now used internally to avoid having to re-search every single episode each time. This also means that episodes are now strongly tied to their podcast.
6. Podcast matching regex is now pre-compiled at startup time giving significant performance improvements on matching - although (due to [5]), this is now only done once per episode!
7. Podbit will now exit if mpv crashes, rather than hanging
8. Episodes which are not matched to any podcast will now be placed under a dummy podcast called "Unorganised" - rather than simply shown under their own URL.
9. The DB file (containing podcast regex patterns) will no longer be overwritten by Podbit upon exit, allowing live edits. It should be noted that Podbit still does not reload this file dynamically.
10. Added a "contrib" directory for user-submitted (or my own) patches. The first few things are in there already!
11. Fixed a couple of race conditions
12. Removed a couple of deprecated function calls (ioutil mostly)

v3.1: Bug Fixes

Podbit v3.1

Podbit v3.1 is a maintenance release which fixes several irritating bugs, including a couple of crashes and lockups. Major user-facing changes include downloads now showing their titles in the download menu, and major improvements made to download performance. Additionally, the queue file is now periodically saved (after reloading any changes), as well as when SIGINT or SIGTERM are received.

Changelog

1. Fixed a rare deadlock when cancelling a slow download
2. Fixed UI freezing while waiting for slow downloads to cancel
3. Improved performance of player UI (reduced number of redraws)
4. Fixed status messages lingering until user input is processed
5. Episode titles are now shown in the download menu, if available
6. Download percentage completion will now only update periodically, rather than on every single data chunk processed, reducing thread contention and boosting download performance significantly
7. The queue file will now be saved when SIGTERM or SIGINT are received. You will now not lose data by killing podbit from the terminal
8. The queue file will also be periodically saved, or manually when "R" is pressed in the interface
9. Fix "all your base"-level grammar in directory creation error message
10. Fix a crash when multiple downloads fail in series or when attempting to cancel downloads which failed due to IO errors

At some point I will get working toward fixing my own issue with how episodes are loaded, stored and traversed, which should make the program a lot more deterministic and fast when working with loads of episodes.

Enjoy!

v3.0: UTF-8 and Performance

Podbit v3.0

Podbit v3 comes shipped with full Unicode support (UTF-8 encoded), with no hacky escape requirements or random failures. Some UIs (tables in particular) have been redesigned to fit the new UTF-8 model a little better.

This release also features a completely redesigned events system, which has caused a 600% reduction in CPU load on my machine (load avg from 12.5 to merely 2.1 - 3 at worst). This new system has allowed for the fixing of a few irritating bugs that were almost impossible to patch with the old system. Users can expect more stable playback handling and lower battery usage on this new version. And, finally, some nice visual improvements have been sprinkled throughout the program.

Changelog

1. Swap out "libncurses" for "libncursesw", which supports UTF-8 encoding (or, really, whatever encoding you specified via your LC_* environment variables)
2. Change all our UI code to use Unicode characters (runes) for length limits, splits and previews etc. This means that text will no longer distort at the edge of the terminal, especially when resized
3. Fixed tables overspilling onto adjacent rows when UTF-8 characters are used
4. Fixed a rare crash in rendering menus when UTF-8 is involved
5. YoutubeDL downloads will no longer behave strangely on failure, instead correctly reporting that they failed
6. Reduced redundant channels and channel copying, which should cause slightly lower memory footprint and CPU load
7. Reduced sound system update timer from 200ms to 1s
8. Reduced queue reload frequency from every 25s to every 60s
9. Tables will now show text in column colors, rather than black and white with background highlights
10. Prevent redundant screen redraws when idle and paused
11. Ensure that the screen redraws when a download completes or changes status
12. Ensure that tray messages do not linger longer than max message time when last in the queue

It should be noted that this release technically has a new dependency, as plain ncurses is no longer sufficient to run the program; it must now be built with wide character support (-lncursesw).

Enjoy!

v2.1: Unicode and performance

Because podbit uses a portable version of ncurses, supporting unicode is, despite all the help from Go, very difficult. To "fix" most issues, podbit will now escape all displayed titles, replacing common unneeded unicode with safe, ASCII equivalents. You will not even likely notice the difference, but will still see the visual bugs fixed. This will not affect users of languages which use extended ASCII (european accents, etc).

From further research, I have determined that BSD support for ncursesw is good enough to justify swapping over to using UTF-8 natively, along with some more work on proper string slicing. In a future update, I may remove this band-aid fix and actually support unicode properly. For now, it will simply be escape from any text shown.

Changelog

1. Fixed a rare crash/deadlock at startup when welcome message races with UI startup
2. Performance improvements to player starting
3. Cursor will now move down one row when playing/starting a download in most menus
4. If an episode queued to play fails to download, it will no longer infinitely re-try to download itself
5. Tables will no longer reset cursor to 0th position on resize
6. Fixed a crash in very small terminal windows
7. Fixed cursor becoming stuck at last entry when scrolling menus
8. Performance improvement when playing an ongoing download
9. Added "escaper", which will escape unwanted unicode text
10. Removed support for patching with unicode support

v2.0.2: Bug fixes

Numerous small stability, security and quality-of-life fixes.

Changelog

1. Fix several deadlocks as a result of erroneous download lock copying
2. Greatly improve download performance using r/w mutexes
3. Fix screen redraw bind. Previously had no effect. Now orders complete clear and redraw
4. Show custom message when waiting for YouTube to re-encode video, rather than just "In progress" at 100%
5. Remove several redundant usage of goroutines from legacy code. Improves performance
6. Bunp Go version go 1.18 (mainly to use generics)
7. Refactor out list calls to central embedded struct
8. As a result of the above, fixed several small visual bugs involving scroll handling
9. Fix incorrect underscroll calculation, causing visual mismatch and occasional crashes
10. Fix crash when dequeueing waited download when queue has more than one download ongoing
11. Outright prevent most cases of crashes with dequeueing (for some reason these bugs keep cropping up - hopefully not any more)
12. Fix screen flickering when scrolling through table component

v2.0.1: Fix crash on download

Fix a crash/deadlock on attempting to crash downloads using the new system.

Version 2.0: YouTube and generic podcasts support

I have drafted a major release of podbit, as this release introduces both new major features, a complete feature rewrite and improvements of existing capabilities

Changelog

1. Added support for downloading YouTube podcasts using youtube-dl or yt-dlp. Queue entries prefaced with a '+' will be downloaded using a youtube downloader tool. This means that these tools are now a soft dependency, not checked at compile time but required at runtime to download YouTube streams. Failure to install will simply result in YouTube downloads failing
2. Completely refactored download system to utilise threads more effectively, decrease CPU usage and allow for more parallel downloads
3. Downloads are now completely independent of each other and do not preempt each other when updating statistics in the central database of download entries

Final notes

This update may seem small to warrant a bump from 1.0 to 2.0, but an enormous amount of work has gone in under the hood to get this working. For me, this means that a lot of streams from many websites now have titles and metadata embedded correctly. This is a major advantage of using yt-dlp/youtube-dl.

It must be noted that these downloads will intentionally be ignored by podboat, as it cannot handle special characters at the beginning of a line. It will, however, throw out these entries, which may be mildly inconvenient.

Expect many more patch releases in the near future. This is likely to have introduced quite a few regressions.

Closes #3

Version 1.4: Visual spruce up and fix up

Fix a couple of crashes and some weird UI behaviour. Some known issues are still present, such as scrolling on small-ish terminal surfaces (smaller than half the number of podcasts in cells). I will work on patches for these for v1.4.1

Changelog

1. Properly format version number on startup
2. Changed tray progress character to '-'
3. Player will now unpause when a track is added and played to prevent unintuitive behaviour when track stops paused and a new one is added
4. Made tray messages contrast with other text better
5. Added a welcome message showing basic information (version, number of episodes, number of podcasts, time to startup)
6. Added a message shown when attempting to pause a completed/failed download
7. Fixed a regression in list scrolling calculations which assumes that scrolling only happens one line at a time (as used to be the case)
8. Menu lists will no longer reset back to zero when resized. Instead, selection will scroll upward with width and height
9. Fixed a crash when dequeueing a podcast which is waiting to be downloaded. Now, podcast will still be downloaded but download completion is ignored
10. Increase robustness of queue system in a few small ways, fixing a couple of crashes/deadlocks in the process

Version 1.3.1: Directories bug fix

Fix a small yet breaking issue which prevents loading if certain environment variables are unset or empty.

Changelog

1. Fix detection of XDG_DATA_HOME
2. Fix database location detection
3. If XDG_DATA_HOME is empty or unset, fallback to the XDG suggested defaults

Fixes #1

Version 1.3: Automatic cache maintenance

This release adds automatic management and deleting of old episodes from the cache to avoid a build up in disk usage for your podcasts. Episodes which have not been listened to for over three days (by default) will be deleted.

Changelog

1. Add (optional) field to queue denoting the last play time of a podcast episode
2. Add cleanup routine for cache to delete episodes older than the configured timeout (default: 3 days)
3. Added a summary message in the console to show which episodes have been deleted

Despite the small changelog, this release does have quite a bit going on behind the scenes, so expect patch releases soon enough.