Open
Conversation
simonmichael
added
the
investing
There was a problem hiding this comment.
Pull request overview
This PR is a large step toward hledger 2.x, centered on introducing automated lot tracking (enabled via --lots) and aligning output/interop behavior (notably Beancount and Ledger-style lot syntax), with accompanying data model updates and extensive new specs/tests.
Changes:
- Add/extend lot tracking: classification (
_ptype), reduction methods (FIFO/LIFO/HIFO/AVERAGE/SPECID variants), transfers/disposals, and--lotsreporting behavior. - Update core types and plumbing: Posting
ptype→preal, MixedAmountKey expanded to include cost basis/cost combinations, gain-posting handling in balancing, and commodity directive source positions for better errors. - Improve print/export: new
-O ledgeroutput format, expanded-O beancountoutput (tolerance option, price directives, booking method, tag/metadata handling), and updated docs/specs/tests.
Reviewed changes
Copilot reviewed 81 out of 81 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| hledger/test/print/beancount.test | Updates/extends Beancount print expectations |
| hledger/test/ledger-compat/hledger-other.test | Adjusts ledger-compat check expectations |
| hledger/test/json.test | Updates JSON schema expectations (preal) |
| hledger/test/journal/lots.test | Expands/renumbers lots syntax functional tests |
| hledger/test/journal/lots-transfer.test | Adds transfer scenarios and regression coverage |
| hledger/test/journal/lots-transfer-entries.test | Systematic transfer entry variation tests |
| hledger/test/journal/lots-methods.test | Tests reduction/booking method variants |
| hledger/test/journal/lots-dispose.test | Adds disposal scenarios and balancing behavior tests |
| hledger/test/journal/lots-dispose-entries.test | Systematic disposal entry variation tests |
| hledger/test/journal/lots-clopen.test | Tests close/open workflow interactions with lots |
| hledger/test/journal/lots-acquire.test | Adds acquisition scenarios and lotful detection tests |
| hledger/test/journal/lots-acquire-entries.test | Systematic acquisition entry variation tests |
| hledger/test/journal/infer-costbasis.test | Adds cost basis inference behavior tests |
| hledger/test/journal/infer-cost.test | Adds transacted cost inference behavior tests |
| hledger/test/journal/commodity-tags.test | Adds commodities tag: query test |
| hledger/hledger.cabal | Regenerated cabal metadata (hpack version bump) |
| hledger/Hledger/Cli/Commands/Register.hs | Uses preal for virtual posting bracketing |
| hledger/Hledger/Cli/Commands/Print.md | Documents new ledger output format |
| hledger/Hledger/Cli/Commands/Print.hs | Adds ledger output and enhances Beancount output |
| hledger/Hledger/Cli/Commands/Commodities.md | Documents tag: support for commodities query |
| hledger/Hledger/Cli/Commands/Commodities.hs | Enables tag-aware commodity filtering |
| hledger/Hledger/Cli/Commands/Close.md | Notes lots-specific balance assertion behavior |
| hledger/Hledger/Cli/Commands/Close.hs | Skips assertions for lot subaccounts in closing txns |
| hledger/Hledger/Cli/Commands/Check.md | Documents lot subaccount exemption for account checks |
| hledger/Hledger/Cli/Commands/Add.hs | Uses preal when creating postings in add workflow |
| hledger/Hledger/Cli/Commands.hs | Updates command tests to preal |
| hledger/Hledger/Cli/CliOptions.hs | Adds --lots; refactors output-format/rawopts handling |
| hledger/.date.m4 | Updates manual date stamp |
| hledger-web/hledger-web.txt | Updates rendered manual footer date |
| hledger-web/hledger-web.cabal | Regenerated cabal metadata (hpack version bump) |
| hledger-web/hledger-web.1 | Updates manpage header date |
| hledger-web/Hledger/Web/Widget/AddForm.hs | Uses preal when validating postings |
| hledger-web/Hledger/Web/Test.hs | Updates rawOptsToInputOpts call signature |
| hledger-web/.date.m4 | Updates manual date stamp |
| hledger-ui/hledger-ui.txt | Updates rendered manual footer date |
| hledger-ui/hledger-ui.cabal | Regenerated cabal metadata (hpack version bump) |
| hledger-ui/hledger-ui.1 | Updates manpage header date |
| hledger-ui/.date.m4 | Updates manual date stamp |
| hledger-lib/package.yaml | Exposes new modules (AccountType, Lots, Write.Ledger) |
| hledger-lib/hledger-lib.cabal | Regenerated cabal metadata + new exposed modules |
| hledger-lib/Hledger/Write/Ledger.hs | Adds Ledger-style lot syntax transaction renderer |
| hledger-lib/Hledger/Write/Beancount.hs | Enhances Beancount output (prices, metadata filtering, commodities/accounts) |
| hledger-lib/Hledger/Read/TimedotReader.hs | Switches virtual posting field to preal |
| hledger-lib/Hledger/Read/RulesReader.hs | Switches posting field to preal |
| hledger-lib/Hledger/Read/JournalReader.hs | Adds commodity directive sourcepos; uses preal |
| hledger-lib/Hledger/Read/InputOptions.hs | Adds lots_; removes auto_posting_tags_ |
| hledger-lib/Hledger/Query.hs | Adds tag-aware commodity matching helper |
| hledger-lib/Hledger/Data/Types.hs | Core type changes: Commodity sourcepos, CostBasis order, LotId/ReductionMethod, MixedAmountKey expansion, preal |
| hledger-lib/Hledger/Data/Transaction.hs | Adapts posting rendering to new AmountFormat pipeline |
| hledger-lib/Hledger/Data/Timeclock.hs | Uses preal for virtual postings |
| hledger-lib/Hledger/Data/Posting.hs | Updates rendering signatures; uses preal consistently |
| hledger-lib/Hledger/Data/Json.hs | Renames JSON field output to preal |
| hledger-lib/Hledger/Data/JournalChecks.hs | Ignores lot subaccounts in account checks; updates rendering call sites |
| hledger-lib/Hledger/Data/Journal.hs | Adds lots helpers/validation; cost-basis/cost inference utilities |
| hledger-lib/Hledger/Data/Errors.hs | Adds commodity tag excerpt helper for better errors |
| hledger-lib/Hledger/Data/Balancing.hs | Excludes Gain postings from balancing in disposal txns; uses preal |
| hledger-lib/Hledger/Data/Amount.hs | Adds Ledger lot syntax rendering; expands amount keying; preserves {}; reorders cost-basis vs cost display |
| hledger-lib/Hledger/Data/AccountType.hs | New helpers for subtype checks (moved out of Types) |
| hledger-lib/Hledger/Data/AccountName.hs | Updates posting type helpers to PostingRealness |
| hledger-lib/Hledger/Data.hs | Re-exports new modules (AccountType, Lots) |
| hledger-lib/.date.m4 | Updates manual date stamp |
| doc/SPEC-special-postings.md | New spec doc for “special posting” detection |
| doc/SPEC-print.md | New spec doc for print behavior/flags |
| doc/SPEC-lots.md | New/expanded lots specification |
| doc/SPEC-journal.md | New/expanded journal format specification |
| doc/SPEC-finalising.md | New/expanded journal finalising pipeline specification |
| doc/PLAN-doc.md | Planning notes for documentation work |
| doc/PLAN-beancount-output.md | Planning notes for Beancount export |
| doc/NOTE-recompiling.md | Dev note on faster typecheck workflow |
| doc/NOTE-amount-keys.md | Design note on MixedAmountKey/aggregation semantics |
| doc/NOTE-amount-classes.md | Design note on typeclass semantics for amounts |
| doc/NOTE-2.x.md | Notes on 2.x strategy/positioning |
| doc/AI.md | Initial project AI policy document |
| README.md | Describes 2.x branch intent/plans |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Owner
Author
|
PS this brings up a difficult topic. If you don't like the sound of this, please do take a look at https://github.com/simonmichael/hledger/blob/lots/doc/AI.md . Feedback welcome on the mail list thread, here, or in chat. |
This was referenced Mar 1, 2026
simonmichael
force-pushed
the
lots
branch
3 times, most recently
from
fc8a5f7 to
7e849d2
Compare
Owner
Author
|
The lots branch has been updated with many fixes and improved diagnostics, and a It's now pretty much accepting all my sloppy inconsistent real-world journals, except for genuinely wrong/unfinished entries. By that I mean it's checking them carefully enough, and also tolerating enough variation in journal entries, that the lot movements are calculated as intended (no more "insufficient lots" errors caused by some far away entry; problems are generally reported at the earliest relevant location 🤞). So: more real world testing welcome. Also, there's a user-oriented set of example journal entries showing the most common patterns: examples/lots/lot-entries.journal. |
simonmichael
changed the title
This was referenced Mar 13, 2026
simonmichael
force-pushed
the
lots
branch
3 times, most recently
from
7ae25fd to
e857ccb
Compare
When --lots-warn is used, lot calculation is enabled (implies --lots) and lot selection failures (insufficient lots, no lots available, ambiguous selector) produce warnings to stderr instead of aborting. The affected postings pass through unchanged, allowing partial lot processing of journals with known gaps. Warnings use the same rich error format (with transaction excerpt) as the normal error messages.
Change lot debug traces (--debug=5) to show a one-line transaction
summary "FILENAME:LINE DATE DESC" instead of the full transaction
excerpt, e.g.:
lots: -:1 2026-01-15 buy: acquired 10 AAPL {2026-01-15, $50} on assets:stocks
Make --lots-warn also catch pairIndexedTransferPostings failures (mismatched/unmatched transfer postings) as warnings, passing the whole transaction through unchanged. Also use lotDbg helper for classification traces, replacing verbose dbg5 calls that dumped full Transaction records.
…ERAGEALL) *ALL variants select lots per-account like the base methods, but validate that the selected lots would also be chosen first if all accounts' lots were considered together. AVERAGEALL additionally computes the weighted average cost across the global pool rather than per-account.
Lot errors (insufficient lots, no lots available, ambiguous selector) now show which reduction method is active and where it came from (account tag, commodity tag, or default), plus a hledger command to review prior lot operations for the affected account/commodity. Also adds showOtherAccountLots to the "insufficient lots" error (was already shown for "no lots available").
Replace txnErrPrefix with postingErrPrefix at selectLots call sites, so lot errors show the posting line highlighted rather than the transaction header line.
When a posting involves a lotful commodity or account in an asset account but the classifier couldn't determine its role (acquire, dispose, or transfer), it previously passed through silently. Now with --lots-warn, a warning is emitted at the point where classification fails, helping users find the root cause of downstream "insufficient lots" errors. Zero-amount postings (e.g. from balance assertions) are skipped.
Two transfers on the same date moving portions of the same lot to the same destination account would overwrite instead of summing quantities, due to M.union being left-biased. The addLotState helper (previous commit) fixes this by using M.unionWith to sum quantities.
Show -f options and --verbose-tags.
When a transaction contains both a fee (negative lotful posting paired with expenses) and a transfer (negative lotful posting paired with another asset account), the otherAssetReceives heuristic was too broad and classified both as transfer-from. This caused the fee posting to pair with the transfer-to, delivering the wrong amount. Now checks for a non-asset counterpart at the exact same commodity+quantity (hasFeeCounterpart); when found, the posting is classified as dispose instead of transfer-from.
When an equity transfer-to posting (opening balance) creates a lot on the same date as a regular acquire, findDatesNeedingLabels can't predict the collision (it only counts acquire-tagged postings, not transfer-to postings processed as acquires). Rather than erroring with "duplicate lot id", processAcquirePosting now auto-generates a disambiguating label when a collision is detected and the posting has no user-provided label.
`hledger check lots` runs the lot-tracking pipeline (posting classification, lot calculation, disposal balancing). It also enables --verbose-tags, so that any error messages will show the ptype classification tags.
Income statement accounts (Revenue, Expense, Gain) are flow accounts
that should not track lots. A posting like `expenses:fees 0.1 ETSY {$80} @ $90`
(stock fee on disposal) was getting classified as acquire and receiving
a lot subaccount, which is incorrect.
Now shouldClassifyWithCostBasis skips acquire/dispose classification for
income statement account types, while still allowing transfers (to equity etc.).
check lots: strict mode, stops at first lot error (unchanged behavior). check lotswarn: lotswarn mode, lot selection failures are warnings not fatal errors. Only declaration errors and disposal balance failures are fatal. Uses a two-pass approach: quiet first pass detects hard errors, then diagnostic pass shows scoped warnings. Also adds journalCalculateLotsQuiet (suppresses trace/warn output) and parseErrorDate (extracts date from error messages for scoping).
When --lots-warn is active and a lotful acquire posting is declassified because no cost basis or price could be inferred, emit a warning. This catches cases like prices accidentally left inside comments (e.g. `2220 T ; @ $0.01` where the @ is after the semicolon).
Previously, a bare acquire posting (no {cost basis} or @ price) that
couldn't infer a cost was silently declassified — its _ptype tag was
stripped and the posting passed through untracked. This created hidden
balance discrepancies where the raw account balance included the
amount but the lot state didn't, causing "insufficient lots" errors on
later transactions.
Now processAcquirePosting returns an error for this case, and
foldMPostings handles it like dispose errors: hard error with --lots,
warning with --lots-warn. The error surfaces at the actual problem
transaction instead of a later one.
Also fix shouldClassifyBareTransferTo to exclude postings with @ price,
preventing misclassification of acquisitions as transfers.
Also tweak some error messages.
Refactor lot calculation to collect warnings in the return type instead of using a trace-style warnFn callback. journalCalculateLotsImpl now returns Either String (Journal, [String]) where the [String] contains any warning messages. This removes the RankNTypes/forall parameter and eliminates unsafePerformIO from the warning path. check lotswarn now uses journalCalculateLotsImpl directly to get the warnings list, prints them via warnIO, and exits with code 1 if any were found. Previously it always exited 0 when there were no hard errors, even if warnings were printed.
…otswarn All lot selection/classification failures are now hard errors. Remove the --lots-warn flag, lots_warn_ InputOpts field, journalCalculateLotsImpl, journalCalculateLotsQuiet, parseErrorDate, and the Lotswarn check. Simplify journalCalculateLots signature (Bool -> Journal -> Either String Journal). Unclassified lotful postings in asset accounts are now hard errors with --lots. Update tests accordingly.
Zero-amount postings of lotful commodities (e.g. balance assertions like `0 AAPL = 100 AAPL` or balance assignments inferring zero) don't represent lot movements, so they shouldn't trigger the "was not classified" error.
Use index-based posting lookup (makePostingErrorExcerptByIndex) instead of equality-based search, which failed when postings were modified by the balancer after parsing. The error now points to the specific posting line rather than the transaction start.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch is intended to be the basis for the hledger 2.0 preview 1 discussed at #2547 (comment) . I hope to release that soon, ideally with 1.52.
It mainly contains the automated lot tracking functionality I have been working on this month. This adds a major missing feature to hledger. It is working for me so far, but it's experimental and needs user testing and refinement. See also:
and for background:
This also represents the first serious use of AI tools in the project. hledger 1.x was developed with no AI assistance; hledger 2.x will carefully explore AI-assisted development. Most of the new code in this branch was generated with claude code + sonnet/opus, guided by me. I have started an AI policy doc which will hopefully grow to answer common questions.
I have reviewed each commit myself, all tests are passing and many new ones were added, and I have done a fair amount of testing with my own journals. I would love any additional human review but wow that's a lot of commits, so I don't expect too much. If you have time please take a look at code, tests, docs, and/or build and try it out. This branch will be released as a preview and anything can be changed if needed.
AI policy:
Lot tracking:
A little more on this lot tracking design: