Hiccup with pushtag

[aisamu]

Hi!

I'm running into a hiccup trying to import transactions into a project that contains an unrelated file that uses push/poptags directives.

If we define a Beancount file with `pushtag` and `poptag` directives:

pushtag #test

2000-01-01 open Assets:Test
2000-01-01 open Expenses:Test

2001-01-01 * "Test"
  Expenses:Test  1 USD
  Assets:Test

poptag #test

And provide a placeholder configuration:

inputs: []
imports: []

Running bh import fails with a parsing error:

bh import -c empty-config.yaml -b document-with-pushtag.beancount 2>&1

[22:22:59] INFO     Loaded import doc from config.yaml          import_cli.py:72
           INFO     Generated 0 transactions                   import_cli.py:109
           INFO     Deleted 0 transactions                     import_cli.py:110
           INFO     Skipped 0 transactions                     import_cli.py:111
           INFO     Collecting existing imported transactions  import_cli.py:122
                    from Beancount books ...
Traceback (most recent call last):
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 665, in lex
    yield lexer.next_token(lexer_state, parser_state)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 598, in next_token
    raise UnexpectedCharacters(lex_state.text, line_ctr.char_pos, line_ctr.line, line_ctr.column,
lark.exceptions.UnexpectedCharacters: No terminal matches '#' in the current parser context, at line 1 col 9

pushtag #test
        ^
Expected one of:
	* COLON

Previous tokens: Token('METADATA_KEY', 'pushtag')


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/49ks2fr4p22hddm76cjfky1hgy8qfb7c-python3.12-beanhub-cli-2.1.1/bin/.bh-wrapped", line 9, in <module>
    sys.exit(cli())
             ^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
  return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/sq347ia5a61cpc6jildf6w93f853ikad-devenv-profile/lib/python3.12/site-packages/beanhub_cli/import_cli.py", line 126, in main
    existing_txns = list(
                    ^^^^^
  File "/nix/store/psjfv2mma7my7lfb0bhchh0pcljwgvbw-python3.12-beanhub-import-1.2.0/lib/python3.12/site-packages/beanhub_import/post_processor.py", line 52, in extract_existing_transactions
    for bean_path, tree in traverse(
                           ^^^^^^^^^
  File "/nix/store/4afpjpmfa6bjvihyhbbjlnl0r7b6j6a0-python3.12-beancount-parser-1.2.3/lib/python3.12/site-packages/beancount_parser/parser.py", line 54, in traverse
    tree = parser.parse(current_file.read_text())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lark.py", line 655, in parse
    return self.parser.parse(text, start=start, on_error=on_error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parser_frontends.py", line 104, in parse
    return self.parser.parse(stream, chosen_start, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 42, in parse
    return self.parser.parse(lexer, start)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 88, in parse
    return self.parse_from_state(parser_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 111, in parse_from_state
    raise e
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 100, in parse_from_state
    for token in state.lexer.lex(state):
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 674, in lex
    raise UnexpectedToken(token, e.allowed, state=parser_state, token_history=[last_token], terminals_by_name=self.root_lexer.terminals_by_name)
lark.exceptions.UnexpectedToken: Unexpected token Token('TAGS', '#test') at line 1, column 9.
Expected one of:
	* COLON
Previous tokens: [Token('METADATA_KEY', 'pushtag')]

Using a hash-less tag fails in a similar way:

pushtag test

2000-01-01 open Assets:Test
2000-01-01 open Expenses:Test

2001-01-01 * "Test"
  Expenses:Test  1 USD
  Assets:Test

poptag test

bh import -c empty-config.yaml -b document-with-hashless-pushtag.beancount 2>&1

[22:35:55] INFO     Loaded import doc from empty-config.yaml    import_cli.py:72
           INFO     Generated 0 transactions                   import_cli.py:109
           INFO     Deleted 0 transactions                     import_cli.py:110
           INFO     Skipped 0 transactions                     import_cli.py:111
           INFO     Collecting existing imported transactions  import_cli.py:122
                    from Beancount books ...
Traceback (most recent call last):
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 665, in lex
    yield lexer.next_token(lexer_state, parser_state)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 598, in next_token
    raise UnexpectedCharacters(lex_state.text, line_ctr.char_pos, line_ctr.line, line_ctr.column,
lark.exceptions.UnexpectedCharacters: No terminal matches 't' in the current parser context, at line 1 col 9

pushtag test
        ^
Expected one of:
	* COLON

Previous tokens: Token('METADATA_KEY', 'pushtag')


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/49ks2fr4p22hddm76cjfky1hgy8qfb7c-python3.12-beanhub-cli-2.1.1/bin/.bh-wrapped", line 9, in <module>
    sys.exit(cli())
             ^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/q43f205zpdgrkl60ihwzinvzccbyzyig-python3.12-click-8.1.8/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/sq347ia5a61cpc6jildf6w93f853ikad-devenv-profile/lib/python3.12/site-packages/beanhub_cli/import_cli.py", line 126, in main
    existing_txns = list(
                    ^^^^^
  File "/nix/store/psjfv2mma7my7lfb0bhchh0pcljwgvbw-python3.12-beanhub-import-1.2.0/lib/python3.12/site-packages/beanhub_import/post_processor.py", line 52, in extract_existing_transactions
    for bean_path, tree in traverse(
                           ^^^^^^^^^
  File "/nix/store/4afpjpmfa6bjvihyhbbjlnl0r7b6j6a0-python3.12-beancount-parser-1.2.3/lib/python3.12/site-packages/beancount_parser/parser.py", line 54, in traverse
    tree = parser.parse(current_file.read_text())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lark.py", line 655, in parse
    return self.parser.parse(text, start=start, on_error=on_error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parser_frontends.py", line 104, in parse
    return self.parser.parse(stream, chosen_start, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 42, in parse
    return self.parser.parse(lexer, start)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 88, in parse
    return self.parse_from_state(parser_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 111, in parse_from_state
    raise e
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 100, in parse_from_state
    for token in state.lexer.lex(state):
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qldafmcpnlj5f6z6cmlf5bc2g0hc0kix-python3.12-lark-1.2.2/lib/python3.12/site-packages/lark/lexer.py", line 674, in lex
    raise UnexpectedToken(token, e.allowed, state=parser_state, token_history=[last_token], terminals_by_name=self.root_lexer.terminals_by_name)
lark.exceptions.UnexpectedToken: Unexpected token Token('METADATA_KEY', 'test') at line 1, column 9.
Expected one of:
	* COLON
Previous tokens: [Token('METADATA_KEY', 'pushtag')]

Is this expected?

Side-note: it'd be very useful to have a file-name on the error output!

Thanks for this project!

[fangpenlin]

Hi @aisamu, thanks for reporting the issue. Unfortunately we don't support push tag and have no plan to support in the short term. Because push tag makes sorting and formatting way more complex. We think it's a bit unnecessary given that if our users adopt tools like our beanhub-import, they can easily manage tags for the transactions they want.

[aisamu]

Noted!

Just to reinforce, I ran into this while trying to migrate my existing ledgers to beanhub-import!
I'm more than happy to have something handle tags other than myself.

Do you have a suggestion on how to handle existing/old files?
Manual conversion would be a bit tricky given the volume, and excluding them (e.g. via a separate safe-main.beancount) would potentially make things like duplicate detection fail for those accounts/files, right?

[fangpenlin]

Currently, there's no easy way provided to convert your legacy Beancount file with pushtag syntax to not using pushtax instead. But the beancount-parser for parsing the syntax into syntax tree + beancount-black for formatting syntax tree back into beancount file can help.

In beanhub-cli, we provided account renaming feature along with the format subcommand:

https://beanhub-cli-docs.beanhub.io/commands/format/

We actually implemented a transformer to transform tne syntax tree for renaming account or currency:

https://github.com/LaunchPlatform/beanhub-cli/blob/f9a97e92bcbdb7e3ff7fc51bbcee9657503edf34/beanhub_cli/format.py#L75-L86

I think it's possible to create a transformer to inject tags into your new beancount file. I would envision write a simple script to record the line numbers of push tag and the tags to push. Then, write it to a JSON file or something like that one the side. Then, with a custom transformer, read your beancount file after commenting all the push tag syntax and see if the entry are between the line numbers you have recorded. If so, inject tag into the syntax tree and let beancount-black serialize them back into a new beancount file.

Sorry I know this is less than idea. But I am thinking a better way to do it. Another approach might be asking a LLM model to do the changes for you.