use jsonargparse by ThomasWaldmann · Pull Request #9413 · borgbackup/borg

ThomasWaldmann · 2026-02-25T11:00:24Z

Description

Adopt jsonargparse, fixes #6551.

Checklist

PR is against master (or maintenance branch if only applicable there)
New code has tests and docs where appropriate
Tests pass (run tox or the relevant test subset)
Commit messages are clean and reference related issues

src/borg/archiver/_common.py

codecov · 2026-02-25T15:17:00Z

Codecov Report

❌ Patch coverage is 87.08010% with 50 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.45%. Comparing base (0b05b44) to head (e4e484f).
⚠️ Report is 3 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/helpers/parseformat.py	67.21%	30 Missing and 10 partials ⚠️
src/borg/patterns.py	66.66%	3 Missing ⚠️
src/borg/archiver/__init__.py	93.75%	1 Missing and 1 partial ⚠️
src/borg/archiver/diff_cmd.py	66.66%	2 Missing ⚠️
src/borg/archiver/transfer_cmd.py	66.66%	2 Missing ⚠️
src/borg/archiver/completion_cmd.py	95.23%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9413      +/-   ##
==========================================
- Coverage   76.47%   76.45%   -0.03%     
==========================================
  Files          85       86       +1     
  Lines       14803    14905     +102     
  Branches     2213     2248      +35     
==========================================
+ Hits        11321    11396      +75     
- Misses       2803     2820      +17     
- Partials      679      689      +10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ThomasWaldmann · 2026-02-26T10:58:13Z

@mauvilsa can you have a look, please?

maybe look at the individual commits.

is that flatten_namespace code in borg.helpers.argparsing interesting for other jsonargparse users?

had to import some internal stuff to implement borg completion command, guess there is currently no better way?

guess this can soon be merged - it does not use most features of jsonargparse, but that should be in future PRs.

ThomasWaldmann · 2026-02-27T10:41:48Z

Note: I am currently trying to get rid of these _maincommand, _midcommand, _subcommand suffixes.

Guess they are not needed because jsonargparse supports nested namespaces to separate the different command levels.

Also, the suffixes showed up in generated configs, making them ugly...

Update: done!

mauvilsa

I haven't tried to run the code yet. But might do that in case I see weird behaviors.

scripts/make.py

mauvilsa · 2026-02-27T10:59:40Z

src/borg/archiver/debug_cmd.py

-            "info",
-            parents=[common_parser],
+        subparser = ArgumentParser(
+            parents=[mid_common_parser],


I think there aren't tests in jsonargparse related to parent parsers. Could be that it works just fine. But I should take note of this to have a closer look.

it seems to work, at least neither our automated tests nor me practically trying it noticed an issue yet.

src/borg/helpers/argparsing.py

mauvilsa · 2026-02-27T11:17:24Z

src/borg/helpers/argparsing.py

+from jsonargparse._actions import _ActionSubCommands  # noqa: F401
+from jsonargparse._completions import prepare_actions_context, shtab_prepare_actions  # noqa: F401
+from jsonargparse._completions import bash_compgen_typehint  # noqa: F401


I will look at this in more detail. Eventually there shouldn't be private imports. But I need to see if there is already an alternative, or a modification in jsonargparse is required.

Previously, ArgparsePatternAction and ArgparsePatternFileAction appended recursion roots directly to args.paths. This mixed CLI positional paths with paths derived from patterns (e.g., using the `R` root path command in a pattern file), complicating downstream argument parsing and future jsonargparse integration. This commit introduces `args.pattern_roots` as a dedicated list for these accumulated root paths: - All argparse definition sites now initialize `pattern_roots=[]` alongside `paths=[]` - ArgparsePatternAction and ArgparsePatternFileAction write directly to `args.pattern_roots` - The build_matcher utility accepts both `include_paths` and `pattern_roots` and concatenates them internally - `create_cmd` iterations explicitly concatenate both lists before processing This ensures `args.paths` strictly reflects exactly what the user provided positionally, paving the way for a clean jsonargparse implementation without regressions in pattern behavior.

Guess it was triggered due to naming the variable "token", maybe "sort_key" is less problematic.

other validators / specs are also there and it is easier to maintain as Python code. the compress module is Cython code.

…approach The old code worked around argparse's flat namespace by appending _maincommand / _midcommand / _subcommand suffixes to every common option's dest (e.g. log_level_subcommand), then resolving them with CommonOptions.resolve() after parsing. This polluted config key names and env var names (BORG_LOG_LEVEL_SUBCOMMAND instead of BORG_LOG_LEVEL). jsonargparse nests subcommand arguments automatically, so the workaround is no longer needed. Each parser level now registers common options with their clean dest name. flatten_namespace() is updated to a two-pass depth-first walk so the most-specific (innermost) value wins naturally: borg --info create --debug → log_level = "debug" (subcommand wins) borg --info create → log_level = "info" (top-level fills gap) For append-action options (--debug-topic) values from all levels are merged (outer + inner) to preserve the accumulation behaviour.

Borg's ArgumentParser (in borg.helpers.argparsing) now subclasses jsonargparse's ArgumentParser and pre-sets two defaults that every borg parser uses: formatter_class = RawDescriptionHelpFormatter add_help = False

ThomasWaldmann · 2026-02-27T18:36:39Z

(rebased on current master, added auto env var support)

ThomasWaldmann · 2026-02-27T20:14:03Z

Hmm, I can't get "store_true" stuff working correctly:

subparser.add_argument("-s", "--stats", dest="stats", action="store_true", help="print stats")

With that and without the value being set in the default config, overriding via BORG_CREATE__STATS=true crashes the code, because it puts a str into args.stat instead of the expected bool.

If I set the value in the default config, it works, but I can not overwrite it with the env var.

A hack with ActionYesNo also didn't work, because it does not like short options like -s. omni-us/jsonargparse#355 (comment)

Another weird hack: omni-us/jsonargparse#497 - is that the intended way?

mauvilsa · 2026-02-28T07:38:46Z

Hmm, I can't get "store_true" stuff working correctly:

This I would consider this a bug. I will fix it.

mauvilsa · 2026-02-28T07:42:10Z

src/borg/archiver/completion_cmd.py

    parse_file_size,
 )
+from ..helpers.argparsing import ArgumentParser
+from ..helpers.argparsing import _ActionSubCommands


Some time ago I noticed that add_subcommand() does not show up in the API reference. I will make this class public to fix this and so that it can be used like here.

mauvilsa · 2026-02-28T07:59:53Z

src/borg/archiver/completion_cmd.py

+        prog = "borg"
+        preambles = []
+        if args.shell == "bash":
+            preambles.append(bash_compgen_typehint.strip().replace("%s", prog))
+            preambles.append(bash_preamble)
+        elif args.shell == "zsh":
+            preambles.append(zsh_preamble)
+
+        with prepare_actions_context(args.shell, prog, preambles):
+            shtab_prepare_actions(parser)
+
+        script = shtab.complete(parser, shell=args.shell, preamble="\n".join(preambles))  # nosec B604


For this I am thinking that a method is needed to programmatically get a completion script. The code would change as follows:

Suggested change

prog = "borg"

preambles = []

if args.shell == "bash":

preambles.append(bash_compgen_typehint.strip().replace("%s", prog))

preambles.append(bash_preamble)

elif args.shell == "zsh":

preambles.append(zsh_preamble)

with prepare_actions_context(args.shell, prog, preambles):

shtab_prepare_actions(parser)

script = shtab.complete(parser, shell=args.shell, preamble="\n".join(preambles)) # nosec B604

preambles = []

if args.shell == "bash":

preambles.append(bash_preamble)

elif args.shell == "zsh":

preambles.append(zsh_preamble)

script = parser.get_completions_script(f"shtab-{args.shell}", preambles=preambles)

The method would not be shtab specific so that it can be used in potential future extensions, e.g. omni-us/jsonargparse#618 (comment)

…oolean flags Root causes ----------- Two independent bugs conspired to make `BORG_CREATE__STATS=true` a no-op when `--stats` was not also given on the CLI: 1. **`ActionYes` default was `False` instead of `None`.** jsonargparse's config-merging pipeline calls `merge_config()` to layer CLI values on top of env-var values. Because every absent `--stats` flag produced a hard `False` default, that `False` was written into the merged namespace and silently overwrote the `True` that had been read from the environment variable. 2. **`flatten_namespace()` converted every `None` to `False` unconditionally.** After the namespace was flattened, any field that was still `None` (meaning "not set anywhere") was turned into `False`. This was correct for boolean flags, but the blanket conversion also masked the symptom of bug #1 and made it impossible to distinguish "flag absent" from "flag explicitly set to False". Fixes ----- * **`ActionYes` default changed from `False` to `None`** so that an absent CLI flag contributes nothing to the merge rather than a hard `False`. `_boolean_type()` is updated to pass `None` through unchanged. * **`ArgumentParser.merge_config()` override** added with two rules: - Skip any `None` value from `cfg_from` so it never overwrites a real value (e.g. `True` from an env var) in `cfg_to`. - For `ActionYes` fields specifically, a `False` in `cfg_from` must not overwrite a `True` already present in `cfg_to` (env-var `True` wins). * **`flatten_namespace()` now accepts `action_yes_dests`** and converts `None → False` only for those fields, leaving non-boolean `None` values untouched. * **`parse_args()`** collects all `ActionYes` dests from the parser and every subparser and passes them to `flatten_namespace()`. * **Regression test** `test_env_var_bool_flag` added: sets `BORG_CREATE__STATS=true`, parses `["create", "test", "/some/path"]` without `--stats`, and asserts `args.stats is True`.

The previous commit added a merge_config override that used an action_yes_dests set-comprehension to decide which fields to protect. This is unnecessary — the two generic rules are sufficient: 1. Never let a None value from cfg_from overwrite anything in cfg_to. (None means "not set" — it comes from ActionYes's default=None when the flag was absent from the CLI.) 2. Never let a False from cfg_from overwrite a True in cfg_to. (This protects an env-var True from being clobbered by a baked-in False default that jsonargparse's get_defaults/handle_subcommands injects into the top-level namespace before the subparser reads its own env vars.) These rules are safe for all field types, not just ActionYes booleans, so the action_yes_dests lookup in merge_config is removed. The action_yes_dests property and its use in flatten_namespace (to convert remaining None → False only for boolean flags) are kept — that part is still needed.

ThomasWaldmann force-pushed the jsonargparse2 branch 2 times, most recently from 89f6898 to 5aa0956 Compare February 25, 2026 14:08

ThomasWaldmann mentioned this pull request Feb 25, 2026

Use jsonargparse #9336

Closed

ThomasWaldmann force-pushed the jsonargparse2 branch from 5aa0956 to 7db38a1 Compare February 25, 2026 14:50

ThomasWaldmann commented Feb 25, 2026

View reviewed changes

src/borg/archiver/_common.py Outdated Show resolved Hide resolved

ThomasWaldmann changed the title ~~use jsonargparse, try 2~~ use jsonargparse Feb 26, 2026

ThomasWaldmann force-pushed the jsonargparse2 branch 2 times, most recently from 14adfd2 to c6452f7 Compare February 26, 2026 10:44

ThomasWaldmann mentioned this pull request Feb 26, 2026

argparse++ #6551

Open

ThomasWaldmann force-pushed the jsonargparse2 branch from c6452f7 to 5b9b450 Compare February 26, 2026 13:17

mauvilsa suggested changes Feb 27, 2026

View reviewed changes

ThomasWaldmann force-pushed the jsonargparse2 branch 2 times, most recently from d39fa19 to 2675b31 Compare February 27, 2026 14:09

This was referenced Feb 27, 2026

Feature request - default list and per repo default options #8916

Open

Per-repo default parameters (chunker, compression) #346

Open

ThomasWaldmann force-pushed the jsonargparse2 branch 3 times, most recently from 6665f8f to 21f7548 Compare February 27, 2026 14:58

ThomasWaldmann added 9 commits February 27, 2026 19:33

make validators idempotent

39636dc

streamline function names for borg key subcommands

eabc473

migrate to jsonargparse

c149a35

fix make build_man / build_usage

ad21207

SortBySpec: avoid triggering bandit security checker

68ebd68

Guess it was triggered due to naming the variable "token", maybe "sort_key" is less problematic.

blacken src

ec129ac

import from jsonargparse: SUPPRESS, REMAINDER, NameSpace

4e6c16e

adapt borg completion for jsonargparse

dd9ee3d

ThomasWaldmann added 12 commits February 27, 2026 19:34

fix argparsing test

758c7b0

reorg imports

361ba01

helpers.jap_helper -> helpers.argparsing

15d5910

move compress.CompressionSpec to helpers.parseformat.CompressionSpec

c342c24

other validators / specs are also there and it is easier to maintain as Python code. the compress module is Cython code.

make build_usage / build_man: do not show options with SUPPRESS

ad77c48

reorg imports

ceba422

add octal_int validator

f951a12

simplify flatten_namespace, add docstring for argparsing

405d8ff

argparsing: add ArgumentParser subclass with borg's usual defaults

8f51646

Borg's ArgumentParser (in borg.helpers.argparsing) now subclasses jsonargparse's ArgumentParser and pre-sets two defaults that every borg parser uses: formatter_class = RawDescriptionHelpFormatter add_help = False

add support for yaml config files, default config

fb6ce2d

add support for auto-generated environment variables (jsonargparse)

e4e484f

ThomasWaldmann force-pushed the jsonargparse2 branch from 21f7548 to e4e484f Compare February 27, 2026 18:35

use jsonargparse's PositiveInt

18b531d

ThomasWaldmann added this to the 2.0.0b21 milestone Feb 28, 2026

mauvilsa reviewed Feb 28, 2026

View reviewed changes

ThomasWaldmann added 5 commits February 28, 2026 19:46

improve configuration precedence docs

f3488ad

argparsing: replace store_true with ActionYes

da6e3b0

argparsing: set env_prefix

ffc176d

Uh oh!

Conversation

ThomasWaldmann commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

Uh oh!

codecov bot commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ThomasWaldmann commented Feb 26, 2026

Uh oh!

ThomasWaldmann commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mauvilsa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mauvilsa Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

ThomasWaldmann Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mauvilsa Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

ThomasWaldmann commented Feb 27, 2026

Uh oh!

ThomasWaldmann commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mauvilsa commented Feb 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mauvilsa Feb 28, 2026

Choose a reason for hiding this comment

Uh oh!

mauvilsa Feb 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ThomasWaldmann commented Feb 25, 2026 •

edited

Loading

codecov bot commented Feb 25, 2026 •

edited

Loading

ThomasWaldmann commented Feb 27, 2026 •

edited

Loading

ThomasWaldmann commented Feb 27, 2026 •

edited

Loading

mauvilsa commented Feb 28, 2026 •

edited

Loading