Release v0.3.78 — audit scope: 33 issues across rendering, text, files and colour - #1185
Open
yfedoseev wants to merge 45 commits into
Open
Release v0.3.78 — audit scope: 33 issues across rendering, text, files and colour#1185yfedoseev wants to merge 45 commits into
yfedoseev wants to merge 45 commits into
Conversation
A page may declare /DefaultGray, /DefaultRGB or /DefaultCMYK, and ISO 32000-1:2008 8.6.5.6 makes a bare g/rg/k behave as if it had named that override colour space. The declared family and the operand count the content stream supplies are therefore independent: an ordinary one-operand `0.5 g` under /DefaultGray [/DeviceCMYK] reaches the DeviceCMYK projection with a single component. three_as_rgb and four_as_cmyk_native indexed [1..3] and [1..4] with no bounds check. Two of the three dispatch sites guarded for that and the array-name arm did not, so an otherwise ordinary file indexed out of bounds — a host-process abort under the release profile's panic=abort. Move the arity precondition into the two helpers, which degrade to the first component as gray exactly as the guarded sites already did, and drop the now-duplicated guards from those sites. One rule in one place instead of a precondition three callers have to remember. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
A base image whose /SMask declares a zero dimension aborted the host process: the resample loop computed `sw - 1` on 0u32, which underflows to u32::MAX in release and then indexes out of bounds. With panic=abort in the release profile that terminates the caller, from a file that parses. The sibling /Mask loop had been widened to u64 and given a zero guard for exactly this hazard; the /SMask copy had not, because the two are the same operation — resample a single-channel mask onto the base grid and fold it into alpha — written out twice. Extract fold_mask_into_alpha and route both through it, so the guards belong to the operation rather than to one of its two copies. Also validate the image dimensions where they are read. Table 89 requires positive integers and 8.9.5.1 mandates the image-to-user matrix [1/w 0 0 -1/h 0 1], undefined at zero, so a negative or out-of-range value is invalid rather than a number to truncate: `as u32` turned -1 into 4294967295 and 2^32 into 0, both of which flowed into allocation and sampling. The stencil path's image_mask_layout already rejects these with try_from; the general extractor was the outlier. Every caller skips a failed image and logs, so a bad image costs its own page area, not the render. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 7.9.7 describes a number tree as a tree, but nothing in the file format stops a /Kids array from naming an ancestor. The walker recursed with neither a depth cap nor a visited set, so a self-referential node overflowed the stack — and a stack overflow is not a catchable panic: under panic=abort it takes the host process down. Both guards already exist in this crate's two analogous walkers. Apply the same shape here: a visited set keyed on ObjectRef, since a cycle through indirect references is the realistic form and a depth cap alone would still walk a wide one for a long time, plus a depth cap matching the colour-space walker's. Page labelling is an extraction feature, so a malformed tree degrades to the ranges recovered before the cycle, with a warning, rather than failing the open. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 7.9.5: "Although rectangles are conventionally specified by their lower-left and upper-right corners, it is acceptable to specify any two diagonally opposite corners. Applications that process PDF should be prepared to normalize such rectangles in situations where specific corners are required." Rect::from_points built the struct literally while Rect::new — the same type's other constructor, fed a width and a height — normalised. The two disagreed, and the callers needing specific corners got the unnormalised one: /MediaBox [612 792 0 0] produced negative extents, so the page dimensions went negative and pixmap allocation failed. The page did not render at all. Have from_points delegate to new, which fixes every rectangle built from two corners at once, and normalise in get_page_media_box, which returns the four numbers rather than a Rect. Both are the read boundary, so no consumer has to know which diagonal the file used — several had already grown their own local .abs() and min/max to compensate. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
Each entry states the clause, the mechanism, and what a reader would have seen — and none of them claims a surface the code does not have. The release notes for this version are being checked against the code rather than against the previous notes. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 8.11 lets a reader choose whether to honour optional-content state and blesses callers supplying their own, so returning hidden text by default is legitimate. Offering that mechanism and then ignoring it is not: above a 256 KB content stream the extractor took a prescan route that keeps only BT..ET/Do regions and discards the BDC/EMC pairs carrying optional-content membership, so the exclusion silently did nothing. The caller asked, got no error, and got the content. The gate that chose the parser tested excluded *inks* alone. Replace it with one predicate — has_emission_filter — that names every filter needing fully-interpreted state, with the reason, so a filter added later is either listed there or inherits the same hole. This localises the class; it does not close it. 8.11.3 requires that when optional content is hidden the content shall not be drawn while graphics state operations shall still be applied, so visibility is a decision about marking the page taken after interpretation, never a licence to stop parsing. The structural answer is one sequential interpreter with suppression at emission, which retires this predicate along with the prescan branch. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 14.7.4.2 makes an /MCID unique only "within its content
stream", and 14.7.4.3 gives a marked-content reference an /Stm entry naming
that stream so a consumer can tell two apart. A page and a Form XObject may
each number theirs from 0.
The crate models this as (McidScope, mcid) and computes the right key at the
boundary — then threw it away in four places:
- a table-rank lookup retried the page namespace on a miss, so a form's
MCID 0 matched the table's MCID 0;
- ReadingOrderContext::mcid_order carried bare u32, and page_order projected
the scope away under a comment claiming reading-order strategies "don't
disambiguate by content-stream scope";
- both structure-order text assemblers bucketed spans in HashMap<u32, _>,
so a form's span joined the page's bucket and was emitted at the page
element's slot, while the form's own element then found the id already
consumed and emitted nothing.
Result: form text was reordered into a table cell and a table cell went
missing. Widen all four to the key that was already being computed.
Deleting the retry alone was not enough — the spans carried correct scopes and
the corruption persisted, which is how the other three sites were found.
Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 7.7.3.3 Table 30 makes /Rotate a clockwise multiple of 90. Every page transform also carries the PDF y-up to raster y-down flip, so each of the four matrices has a negative determinant — a positive one is a mirror, not a turn. The composite renderer's 270 case was corrected to from_row(0, -s, -s, 0, ..). The separation renderer kept its own copy, from_row(0, s, -s, 0, ..), whose determinant is +s^2, so every ink plate of a /Rotate 270 page came out mirrored while the composite of the same page did not: two renderers of one page disagreed. /Rotate -90 was worse — it had fallen through to the unrotated arm and was at least legible, and normalising the angle without fixing the matrix moved it to mirrored. One shared page_base_transform, so the copies cannot diverge again, with the determinant invariant recorded where it can be checked. The existing plate rotation test used a centred symmetric square on a square page, invariant under every rotation *and* every reflection, which is why this shipped; the new fixture is asymmetric on both axes and asserts plate against composite. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 12.5.5: an appearance entry is either a stream or a subdictionary of appearance states, and when it is the latter /AS names the one to use — the specification's own checkbox example. The renderer accepted /N only when it resolved to Object::Stream, so the subdictionary was rejected and every AcroForm checkbox and radio button rendered blank, while Annotation::appearance_state was parsed and read nowhere in the renderer. With no /AS, or an /AS naming no member, the clause blesses displaying nothing, so those cases stay blank deliberately. Annotation::flags was equally unread, so 12.5.3 Table 165's Hidden and NoView annotations were painted. Both are now skipped. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 Table 39 gives a Type 0 function the defaults /Encode [0 (Size_0 - 1) ...] and /Decode "same as the value of Range". A dictionary that writes those out explicitly means exactly what an absent entry means. The evaluator declined on *presence*, while its own doc comment said it declined a non-default /Encode//Decode — so a conforming file fell back to the 1 - tint grey approximation instead of its real colour. Measured over a 154-document sample, 11 of 122 sampled function dictionaries carried both keys and all 11 held the defaults, including one reachable from a /Separation space in this repository's own fixtures. Compare against the computed defaults instead. A genuinely non-default array is still declined, because this evaluator implements only the default mapping. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
Reconstruction runs only on files that are already malformed, so hostile object numbering is guaranteed on this path rather than merely possible. The synthetic Catalog and /Pages node were numbered max_obj + 1 and max_obj + 2 unchecked; no profile sets overflow-checks, so a file containing 4294967295 0 obj panicked in debug and wrapped in release, and the wrapped number then collided with a real object, placing the synthesized page tree on top of it. Also on this path: parse_object_stream returns a HashMap and the selections walking it are get_or_insert, so the same damaged bytes recovered a different /Root between runs of the same binary. Walk it in ascending object number, the order the sibling non-object-stream path in this file already uses. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…ting Three changes to one file that have to land together, because the first is a prerequisite for observing the second. **Premultiplication.** tiny_skia::Pixmap stores premultiplied RGBA and Pixmap::from_vec takes the bytes verbatim, while the image crate produces straight alpha. A fully transparent pixel therefore kept its full colour and the compositor *added* it: a masked-out region showed the base image's colour blended over the backdrop instead of the backdrop alone. Premultiplying also now happens before the resample, which is the correct order — resampling straight alpha bleeds colour across an alpha edge from pixels contributing none. **/Mask polarity.** 8.9.6.2: with the default /Decode, "a sample value of 0 shall mark the page ... and a 1 shall leave the previous contents unchanged. If the Decode array is [ 1 0 ], these meanings shall be reversed." The stencil path implemented the complement, under a comment citing that very clause — so a reader checking the citation against the code saw two things agree and both be wrong. /Decode is read here rather than fixed separately because the two compose: a /Decode [1 0] mask rendered correctly by accident, two errors cancelling, and correcting polarity alone would have broken those files. Past the end of the stream there is no sample to test, so the masked-out value is used rather than resolving toward paint. **Nesting caps.** A form's own /Resources /XObject may name the form, and a tiling pattern's cell may set the pattern it is painting; neither clause requires an acyclic reference graph. Without a cap that recursion overflows the stack, which under panic=abort is a process abort. Type 3 glyph and soft-mask chains were already capped this way; these two were not. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-2 Algorithm 8 generates one random file encryption key, wraps it into /UE under a key derived from the user password, and requires every string and stream to use that same key. compute_u_and_ue produced and returned exactly that key. EncryptDictBuilder build discarded it. EncryptionWriteHandler::new then called compute_encryption_key, which for revision >= 5 returns generate_random_encryption_key — a second, unrelated random key. So the streams were encrypted under key B while /UE wrapped key A: every AES-256 file this library wrote authenticated and then decrypted every stream to noise, in this library and in any other. Carry the key from the builder to the handler. build_with_key returns it alongside the dictionary and build() delegates, so neither existing signature changes; R<=4 returns None because its key genuinely is a derivation the handler can recompute. This had been read statically by two reviews and executed by neither. The new round-trip test reproduced it first — AES-256 extracting "" while AES-128 and RC4-128 pass — and now covers write-then-read for all three, that the canary is absent from the output bytes, and that a wrong password is still refused. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 7.7.3.4 Table 30: an inheritable attribute comes from the nearest ancestor that specifies it. Two walkers implemented this and disagreed. The eager one snapshotted the inherited map around the recursion and used insert, which is correct. The lazy one used entry().or_insert_with() on a root-first walk, which keeps the value already present — the first seen, i.e. the most distant ancestor — so the root won and every intermediate node was ignored, and its comment claimed the opposite. Because the two disagreed, the same page could resolve differently depending on which one ran. The walker change itself is in the preceding commit: both edits are in document.rs and were staged together, so it is not separable after the fact. These tests cover /MediaBox and /Rotate from the nearest ancestor, a page's own attribute beating every ancestor, no leak between sibling subtrees, and the order-independence property that the two walkers violated. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
The corpus render sweep caught this in the preceding commit's own change. Past the end of a stencil's data there is no sample to test. That commit resolved the case toward transparent, reasoning that the mask's masked-out value was the conservative reading. It is not: for an explicit /Mask, alpha 0 *hides* the base image, so an undecodable mask erases the very content it was only meant to refine. An absent /Mask shows the image; an unreadable one must degrade to the same place. The case is not hypothetical. JBIG2 stencils reach this path still compressed — the decode happens on the other branch — so a 1718-byte stream stands in for a 1.1 MB bitmap and virtually every pixel is past the end. A scanned book in the corpus went from its scan to blank pages: mean tone 134 -> 245 across its pages with coverage held at ~1.0, the signature of an image that stopped painting. Attribution was by bisecting the two hunks: disabling premultiplication left it unchanged, which identified this default as the cause. Restores the previous disposition for that case only. The polarity and /Decode correction, which is what the clause is about, is unaffected and its fixtures still pass. Recorded while here: this path never JBIG2-decodes the stencil it reads, so for these files it is testing bits of compressed data. That is a real defect and a separate one. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…erands InDesign wraps placed artwork in a /PlacedPDF marked-content scope. A placed galley that merely repeats the page body must be suppressed or extract_text emits every word twice. The gate decides by tokenising the text-show operands and measuring how much of the placed text also appears outside. But those operands carry encoded character codes, not text (9.4.3: a show operand is a string of character codes interpreted through the font's encoding). Under Identity-H — the dominant modern encoding, and the one this producer emits — the bytes are two-byte CIDs, no run of ASCII alphanumerics forms, the measured duplication came out 0.0, and the gate kept. Every word of the page was then emitted twice, which is exactly what the suppression exists to prevent. text_duplication_fraction now returns Option and the caller treats None as absence of evidence rather than evidence of absence. Gate 2 above already keeps a placed body that dominates the page whatever its encoding, so what reaches this point is placed text comparable in size to the rest of the page, where a duplicate overlay is the likely reading. The real discriminator is bounding-box overlap, which exists downstream and is not consulted here; decoding the operand through the font would also settle it. Either is a larger change than this gate. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
It was never demonstrated to work and the corpus sweep shows it is harmful. The intent was to have span_in_table consult TableCell::spans — the spans the detector actually placed — instead of re-deriving membership by origin +/-2pt against the detector's centre rule. On synthetic fixtures it produced byte-identical markdown and HTML, so no test could show it doing anything. On the 445-document golden set it moves 38 documents, all in the same direction: MORE text, because the identity match fails and cell spans fall through to prose while the table renders them too. That is duplication, not a fix. It reached the branch by accident: in the preceding commit staged this file along with the intended one, so a later checkout restored the change from the index rather than removing it. The underlying defect is real but differently shaped than recorded — there are three ownership implementations (detector by centre, span_in_table by origin, and assemble_text_from_spans by containment plus a cell-text set), and the observable symptom is governed by the third. Tracked with a reproducer separately; it needs one ownership decision consulted by all three consumers, not an edit to one of them. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
… update ISO 32000-1:2008 7.5.6: "The added trailer shall contain all the entries except the Prev entry (if present) from the previous trailer, whether modified or not. In addition, the added trailer dictionary shall contain a Prev entry giving the location of the previous cross-reference section." Only /Size, /Prev, /Root and /Info were written, so every other entry the original trailer carried was dropped. /ID is the one that matters: Table 15 NOTE 2 warns its absence "might prevent the file from functioning in some workflows that depend on files being uniquely identified", and it is required outright whenever an /Encrypt entry is present. Carried entries are emitted in sorted key order, because a HashMap's iteration order is not stable and two saves of one document must produce the same bytes. /Size and /Prev are recomputed for the update by definition, /Root is written from the source trailer just above, and /Info is either rewritten or inherited through the /Prev chain, so those four are skipped. /Encrypt cannot reach this trailer: an encrypted source is already refused by this path, because appending plaintext objects under a document key would corrupt them. The audit item that prompted this read the trailer writer without that guard and so overstated the exposure; the /ID loss is the real half. Noted while here: this crate's own DocumentBuilder writes no /ID at all, so the fixture is hand-built — a source produced by the builder would have had nothing to carry and would have passed vacuously. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 8.10.2 step (c): the form's /BBox, expressed in form space and transformed by /Matrix, is intersected with the current clipping path before the content stream is executed. Table 78 makes /BBox required for exactly this reason. It was not applied at all, so a form painting outside its own bounding box bled onto the page. Expressed as the operators the clause describes — q, the box, W n, the content, Q — rather than as a second clipping mechanism beside the interpreter's own. combined_transform already carries /Matrix, so the rectangle lands in the right place, and the existing clip stack intersects it with whatever clip is already in force. The box is normalised (7.9.5 permits either diagonal) and a degenerate box declines to clip rather than erasing the form: a zero-area /BBox that cannot be honoured should not delete content. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…ages The corpus render sweep shows 11 pages losing substantial ink, up to 40% of a page: PMC8025825 p2 coverage 0.14150 -> 0.08545, PMC8103263 p1 0.41453 -> 0.33425, PMC8103253 p1 0.36263 -> 0.29524. Those are real losses, not the rounding-level movement the rest of the sweep shows. The clause is not in doubt — ISO 32000-1:2008 8.10.2 step (c) does require the /BBox, mapped through /Matrix, to be intersected with the current clip before the content stream runs. The implementation is what is wrong. Most likely mechanism, stated as the hypothesis it is: the clip was expressed by wrapping the form's operators in a save, the rectangle, a clip-and-end-path, the content, and a restore. Real content streams are frequently unbalanced, and a stray restore inside the form consumes the injected save; the trailing restore then pops one level too many and corrupts the clip stack for everything drawn afterwards. That matches the symptom — loss concentrated in content *after* a form rather than inside it. Doing this safely means giving the nested-stream call its own clip rather than borrowing the operator stack's, which is the RenderScope change the release notes deliberately hold for v0.3.80. Reverting rather than shipping a content-losing change into a release; the finding goes on the issue. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 8.9.6.2: for an image mask with the default /Decode a sample of 0 marks the page, and [1 0] reverses that. The CCITT path pads a truncated stream white and substitutes a blank buffer when both decoders fail — but neither value is neutral, because a sample is a sample. One of the two /Decode readings takes the padding as "paint every pixel", so an unreadable stencil covers its whole footprint in the fill colour. Measured rather than assumed: under the default /Decode a damaged stencil already paints nothing, and it is /Decode [1 0] that covers 100% of the footprint. The audit said exactly this; the issue text generalised it. decompress_ccitt_reporting now returns how many leading rows actually came from the stream, and the stencil path overwrites only the padded tail with the value that draws nothing under that mask's own /Decode. Rows that did decode are kept — this crate deliberately recovers partial content instead of blanking the page — and the page ends up looking as it would with the stencil absent, which is the honest degradation. The decoder already logs the failure. decompress_ccitt keeps its signature and delegates, so the other two callers are untouched. The eight existing CCITT rendering tests still pass, including the one that paints real T.6 data, which is what rules out over-blanking. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
A Catalog and page tree rebuilt for a truncated file are seeded into the object cache and have no byte offset. Reconstruction also pre-populates the scanned-offset map, so a later lookup reports "not found" immediately rather than rescanning the file. Evicting one therefore does not cost a re-parse — it makes the object permanently unreachable, and a large truncated document would open, extract pages 1..k, then fail silently on every subsequent call once the 64 MB FIFO turned over. The seeding site's own comment already said these were reachable only through the cache, and then used the evictable insert. They are now inserted pinned: eviction rotates past them, and clear() — which callers use to force a re-parse after authentication — keeps them, because a pinned object has nothing to re-parse from and clearing it would destroy rather than refresh it. The rotation is bounded so an all-pinned cache terminates instead of cycling its queue. Tested on the cache directly rather than end to end: reproducing the symptom through the public API needs a document large enough to churn 64 MB, which is not a reasonable fixture. Four unit tests cover the pinned entry surviving the flood, clear() keeping it, an all-pinned cache terminating, and — the control that stops the first from passing vacuously — an ordinary entry being evicted under identical pressure. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
outline_glyph returns None for an empty glyph as well as a missing one, so the drop tally could not tell "the font gave us nothing" from "the font says paint nothing". Two populations are legitimately the latter: invisible text (ISO 32000-1:2008 9.3.6 render mode 3), and the glyphless fonts OCR tools emit — Tesseract and ocrmypdf ship a synthetic font, conventionally named with GLYPHLESS, mapping every CID to a non-zero glyph id with an empty outline and a correct /ToUnicode. Every such page rendered and extracted perfectly and still produced one warning per page claiming invisible data loss. src/extractors/text.rs already gates on exactly these two signals for the same population; this diagnostic gated on neither. A diagnostic that fires on the healthy common case trains callers to ignore the channel. The fixture is the font from the neighbouring drop-reporting test — one that genuinely drops a glyph — so the assertions are about the gate rather than about a page that never drops anything. Its control asserts a real drop still warns, and the pre-existing per-page reporting test still passes, which is what rules out over-suppression. Render mode 7 is named by the gate but not asserted, and the reason is worth recording: such a page is rasterised twice, once with a default graphics state and once with mode 7, and the first pass emits before the gate can see the mode. That double rasterise is a separate defect. Pinning the current behaviour would enshrine it and asserting the desired behaviour would fail for a reason this change does not own, so it is documented in the test instead. Modes 3 and the glyphless convention are the populations the report is actually about. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…ode does Both were verified against the code rather than against the previous notes. "Exposed in every language binding ... available across the C ABI and every binding built on it" was false twice over. WASM cannot reach page_bbox at all: it is a method rather than a serialized field, and the two fields it derives from are marked serde(skip), so JavaScript cannot even recompute it. And the accessor is the identity when the run's own rotation_degrees is zero, which means a landscape page stored portrait with /Rotate 90 carrying ordinary horizontal text — the commonest rotated-page shape there is — still reports pre-/Rotate space, exactly what the note promised it would not. "Rotated runs now report a page-space rectangle that matches where they display" reads as a fix to extraction, and it is not one. page_bbox has zero internal consumers: LayoutObjectSpatial for TextSpan still returns the raw mixed-frame bbox, so extract_spans/extract_words are unchanged and extract_text_in_rect still selects on the uncorrected rectangle. The geometry itself is right in all eight rotation/mirror combinations; it is the wiring that does not exist. A user who read the old note and deleted their workaround would have got wrong results. Both entries now state the limits alongside the capability. The wiring and the WASM surface are tracked separately; this commit makes the notes honest about what shipped. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 8.5.4: content outside the clipping path shall not be painted. A clip whose device coordinates exceed the rasteriser's ceiling was discarded outright, so an off-page clip with huge coordinates — which excludes the whole page — painted everything it was hiding. The page went from correctly blank to entirely filled. Annex C.1 is worth quoting against the obvious objection: it does say a conforming reader has arithmetic limits and that exceeding one is an error, so having a MAX_DEVICE_COORD is legitimate. What it does not license is the response. Dropping a clip is neither raising an error nor skipping the construct; it renders something the file did not describe, and a limit must never be resolved in the direction that paints more. The previous behaviour was not simply wrong, which is why this is a split rather than a flip. Two unrasterisable clips need opposite answers: one whose bounds miss the pixmap excludes everything and must clip everything away, while one with enormous coordinates that still encloses the page restricts nothing visible and must be discarded exactly as before — an empty mask there would erase every subsequent draw, which is the failure the old comment describes. device_bounds_miss_pixmap tells them apart; non-finite bounds support no conclusion and keep the existing fallback. The fixture covers all four cases, and the enclosing-clip control is the one that stops this from trading one regression for another. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…efused ISO 32000-1:2008 8.9.5.2 maps each raw sample through Dmin + raw * (Dmax - Dmin) / (2^bpc - 1). At 1 bpc a /Decode [1 0] is a pure inversion, and on packed samples that is a byte-wise NOT — no unpacking and no allocation. The unpacking path refuses buffers past a 256 MiB ceiling, which is a correct bound on a hostile /Width x /Height. But refusing left the samples packed *and the /Decode unapplied*, so a large 1-bpc scan came out as the exact negative of the picture. The code's own comment named this outcome and accepted it on the grounds that no real corpus reaches the cap; a negative page is too loud a wrong answer to accept on that basis, and the byte-wise NOT handled it at any size before the unpacking path existed. Applied on the packed bytes in the refusal arm, with samples_are_raw cleared and decode_folded_in set so the colour-key masker and the plate router both still read the right fact about the buffer. This is one half of the reported issue. The other half — a colour-key /Mask skipped when the image also carries a /Decode — is not fixed here. The flag the masker reads is the correct one: the field documentation is explicit that range-testing stored samples is only meaningful in raw sample space, and every rescale leaves it, not just /Decode. The real answer is pdf.js's ordering, mask against the raw samples and decode afterwards, which means reordering the extractor's decode pipeline and threading an alpha channel out of it. That is a larger change than this one and is left to its issue. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 8.11.4: the /OCProperties /D configuration sets the document's default visibility, and the composite renderer already honours it. The separation renderer had no marked-content arms at all, so the exclusion never reached the plates: a layer omitted from the render was still counted as ink, and two renderers of one page gave contradictory answers about the same document. Measured on the fixture, the baseline plates carry identical ink — 1836000 either way — while the composite correctly differs. 8.11.3 settles how to suppress: "the content shall not be drawn" while "graphics state operations ... shall still be applied". So a painting operator is rewritten to the path-clearing `n` rather than skipped, leaving path bookkeeping and every state change exactly as they would be if the content were visible. XObjects, shadings, inline images and text shows have no path-clearing equivalent and mark the page directly, so those are skipped. The excluded set is computed the same way the composite renderer computes its baseline — compute_default_off_ocgs — because the separation entry points take no render options and the defect is about a layer the *document* hides. Not addressed here, and still open on the issue: the separation renderer also has no arms for inline images or `sh`, so those contribute no ink at all. That is under-reporting rather than wrong reporting, and closing it means teaching this renderer to rasterise images and shadings, which is a different piece of work from honouring visibility. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
Caught by the crate's own CCITT polarity test, which went black-majority under /Decode [1 0]. The preceding commit applied the 1-bpc inversion in the arm where unpacking produced nothing, but that arm covers three different situations: a CCITT buffer that is deliberately kept raw with its /Decode polarity carried in ccitt_params, the 8-bpc identity case with no /Decode, and the one actually intended — an unpack that was attempted and refused past the size cap. Applying the inversion in the first case ran that mapping a second time and turned a mostly-white scan black. The predicate that chose the branch is now named and reused, so the refusal handling cannot be reached by a path that never asked to unpack. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
Text painted 0 0 0 1 k does not convert to (0, 0, 0). ISO 32000-1:2008 10.3.5's naive additive complement would give exact black, but this crate converts through the measured process-ink corners and the 000K corner is approximately (0.137, 0.122, 0.126) — a dark grey. That is the right colour to paint; it is the wrong thing to hand a word processor as an explicit run colour. Three converters — DOCX, PPTX and XLSX — decided "leave the colour unset so the theme applies" by testing for exact (0, 0, 0). So every print-origin PDF stamped a hard-coded dark grey into its output instead of inheriting the destination theme. Nothing covered it, and a word-level corpus diff cannot see it: the text is identical and only the colour attribute changed. All three now share one predicate. It recognises the specific value the CMYK conversion produces rather than applying a blanket tolerance, so an author's deliberate dark grey still carries its colour through — a tolerance wide enough to catch 0.137 would have swallowed that too. The fixture asserts the precondition explicitly (CMYK black is not exact RGB black, else the rule would be unnecessary rather than wrong) alongside the positive and negative cases. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…sent jpeg-decoder applies a 255-x inversion to 4-component JPEG output, and the extractor undoes it to recover the raw DCT samples poppler treats as straight CMYK ink. That undo ran only when an Adobe APP14 marker was present, which is wrong twice over. ISO 32000-1:2008 Table 13: when the marker is absent "the default value of ColorTransform shall be 1 if the image has three components and 0 otherwise". For four components, no marker *is* transform 0 — the very case the undo handles — so the specification equates the two situations the code treated oppositely. And measurement settles it independently of the clause. The new test encodes one CMYK pixel, strips the APP14 segment so the entropy-coded data is byte-identical and the marker is the only difference, and decodes both: jpeg-decoder returns the same samples either way. So a marker-less 4-component JPEG kept the decoder's inversion and rendered as the complement of its ink — the canonical Distiller/WeasyPrint shape coming out near-black. The predicate is now named and shared by both decode paths, and an explicit marker declaring some other transform is still taken at its word. The three pre-existing CMYK JPEG suites still pass, including the contract test that pins the decoder's marker-present direction. Not addressed here: the audit also reports that the four-component test fires for flat /DeviceN, so those DCT images error and vanish. That is a separate gate and stays on the issue. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 Table 30: /CropBox is "a rectangle ... that shall define the visible region of default user space. When the page is displayed or printed, its contents shall be clipped (cropped) to this rectangle." The crop box was parsed onto PageInfo and then consulted by neither renderer, so a cropped scan rendered at full media size showing the margins the file asked to crop away — the everyday case being a scan whose black scanner borders were cropped off. One shared page_render_box, used by the composite renderer and the separation renderer alike, so plates and composite keep the same size and origin. §14.11.2 takes the crop box as its intersection with the media box, so an oversized one does not enlarge the page; a crop box disjoint from the medium describes nothing to show and falls back to the media box rather than rendering an empty page. This changes the output dimensions of every page carrying a crop box smaller than its medium, which is a large and intended behaviour change — it is what every other reader does. The fixture covers the size, the origin offset, the clamp, the disjoint fallback, and that content outside the box stops painting. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
The extractor batches a run of Tm-positioned show operations into one span, which is what keeps a producer that positions every glyph individually from yielding thousands of one-character spans. The continuation test required the same line, the same transform and forward progression, but placed no bound on how far forward: a jump into the next column was accepted, so two show operations separated by empty page were glued into a single span carrying no separator and a width spanning the void between them. ISO 32000-1:2008 Table 108 gives Tm and Td the same effect on the text and text line matrices, differing only in whether the displacement is absolute or relative. Td, TD and T* all end the run outright, so a displacement that ends a run for one operator must end it for the other: continuity is a property of the resulting pen position, not of the operator that moved the pen. The bound is an em rather than a word space deliberately. A producer can leave an intra-word repositioning seam wider than the same font's declared space advance, so no word-space constant separates a seam from a space; only the source-order evidence the span merger reads tells them apart. Everything below an em is left to the merger, and this rule speaks only to gaps too large to be typographic slack at any size. The cost is not only a missing space. Anything reasoning about a span's extent -- table-cell ownership, column detection, reading order -- saw one span straddling the gap, so a cell could no longer claim the text drawn inside it. Corpus: 3 of 419 and 9 of 445 documents change, every one an un-glued word or a removed duplicate; no content lost. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…ntly A table cell's text reached the page twice: once from the table's own rendering and again as flow text beside it. Both halves of the suppression compared spacing that the two sides do not have to agree on. A span leaves the flow only by consuming its tokens from a covering cell's retention budget, which is denominated in the cell's member-span tokens. The budget could already consume a span token that is a substring of one budget token -- the cell builders glue spans while joining them -- but had no path for the mirror case, a span token that is the concatenation of several. Word clustering and the flow assembler break words at different distances, so a cell built from two show operations offers "abc" and "def" against a flow span of "abcdef"; that span was retained and emitted a second time. take_concatenation covers the token exactly or spends nothing, so absorption stays bounded by the material the cell was built from. Markdown's orphan recovery re-emits a span the table did not render, and decided that with a literal substring test against the rendered table. The cell builder joins its member spans with a space while the flow assembler joins the same glyphs with none, so a span the table already rendered failed the test purely on whitespace. Whitespace is a rendering choice of each side; the glyphs are the content. Comparing the squashed glyph sequences fixes it, and the row's pipe delimiters are not whitespace, so they survive the squash and still keep a span that straddles two cells from matching the concatenation of their texts. Corpus: 9 of 445 documents change, each one a duplicate removed; the text survives in every case, spaced as the table renders it. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…omments The inherited-page-attributes test read the page rotation through a `get_page_info` accessor that does not exist, so the target failed to compile. It now reads `get_page_rotation`, which is the accessor the crate actually exposes, and the five tests pass. Two earlier commits in this branch inserted a new function between an existing doc comment and the item it documented, which detached the comment from `cmyk_to_rgb` and from `is_table_separator_line` and left two `empty line after doc comment` lints. Both new functions move below the item they were splitting. Type-checked with `--all-targets` on the default, `rendering`, and `--no-default-features --features fips,icc` combinations. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
… spans A landscape table typeset on an upright page carries a dominant text-matrix rotation, and the row-major assembler only reads it correctly once the spans are rotated upright. That map was applied inside the converters and left as an unwritten convention of whichever local variable held the mapped spans, so every other page-space value a converter compared them against stayed in the frame the file wrote it in. Four user-visible failures follow from the one mismatch. Hyperlinks vanished: ISO 32000-1:2008 section 12.5.2 puts an annotation's /Rect in default user space, and intersecting a page-space rectangle against a mapped span matches nothing. Form values detached from their fields: widget spans are built from page-space /Rect values and were appended to the mapped page spans, so one vector held two coordinate frames and the values sorted to the end of the page instead of beside their own labels. Table cells were emitted twice: extract_page_tables works from page-space words and paths, so no cell could claim the spans it renders and every one was emitted again as flow text beside the grid. preserve_layout placed every span wrong: it writes each bbox straight out as absolute CSS and so needs the frame the page displays in, while the map deliberately moves spans out of it. The frame is now a value rather than a convention. spans_in_reading_frame returns the ReadingFrame it applied, and link rectangles, widget spans and table geometry follow the spans into it; layout mode, which consumes no reading order, does not take the map at all. A consumer can map its own geometry instead of having to know the convention. One thing this does not change: the emitted grid keeps its page-space row and column orientation, because the table detector does not know about the reading frame. The cells and their contents are now correct. Corpus: byte-identical on 419 and 445 documents. The map fires only on an upright page whose text carries a dominant rotation, which no corpus document reaches through these consumers. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
Two columns of quarter-turn text ran together into single lines, in two independent places. Downstream, runs are merged into a rotated line by their offset ACROSS the writing axis, with nothing said about their separation ALONG it, so two columns fused however wide the gutter between them. The upright path splits a line at max(3 x font size, 30 pt). A rotated line is the same line with its axes exchanged -- ISO 32000-1:2008 section 9.4.4 puts the glyph displacement along the text matrix's writing direction -- so it now takes the same rule measured along its own axis. Projecting each word's origin onto the writing direction makes both quarter turns one computation. Upstream, the Tm run-continuation test compared the matrix translation components directly, which assumes the run advances along +x and separates along y. Under a quarter turn the two axes are exchanged, so the perpendicular tolerance collapsed to its 0.5 pt floor and every consecutive glyph of a rotated run became its own span: ten glyphs that batch into one span upright produced ten spans rotated. A frame-correct helper already existed but was ANDed onto the raw comparison rather than replacing it, so it could only veto, never admit. The three raw-matrix questions -- on the line, forward along it, near enough to the run's end -- are now asked once in the run's own frame, and the accumulated width advances along that frame too. Note that the raw form was not merely stricter for a quarter turn: it split a rotated column jump by accident, via a tolerance that had collapsed. Substituting the helper alone would have started gluing those, which is why the gap bound moves into it rather than staying beside it. Vertical writing mode keeps the raw comparison; section 9.7.4.3 gives it an axis convention the (a, b) row does not describe. Corpus: byte-identical on 419 and 445 documents. extract_text and to_markdown body text do not route through the rotated line grouping, and no corpus document sets a rotated run glyph by glyph. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
A 2x2 ruled grid with one show operation per cell needed no fix of its own: the flow assembler glued "North" and "120" across the column rule into one span, which no single cell's bbox contained and so no cell's budget could absorb, and every cell was emitted a second time as prose beneath the grid. Ending the glyph run at the repositioning jump splits that span back into two, each inside its own cell. Pinned separately because it reaches the same defect through a different seam from the split/join case above, and because it is the shape a reporter is most likely to hit -- four cells and six rules, nothing unusual. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
ISO 32000-1:2008 section 8.10.2 step (c): the form's bounding box, mapped through /Matrix, is intersected with the current clipping path before the content stream runs, and Table 78 makes /BBox required for exactly this reason. It was not applied, so a form painting outside its own box bled onto the page. The clip is installed at depth 0 of the nested stream's own clip stack rather than by wrapping the stream's operators in an injected save/restore. A nested stream already gets a fresh stack, and `Q` never pops below depth 0, so the clip holds however unbalanced the form's own q/Q pairs are -- which real content streams frequently are. Both tests for that are in the fixture. This reinstates a change reverted earlier in this branch, and the reason given for that revert was wrong on both counts. The stated mechanism -- a stray restore consuming an injected save and corrupting the parent's clip stack -- cannot happen, because execute_operators builds a fresh clip stack per call and the two never share. And the evidence was misread: the render sweep's "content lost" bucket only reports that coverage went DOWN, which is what a fix that stops over-painting looks like. Adjudicated against PyMuPDF and poppler, all six PMC pages cited in the revert are corrections, not losses: PMC8025825 p2 ours 0.14150 -> 0.08545 pymupdf 0.08509 poppler 0.07779 PMC8103263 p1 ours 0.41453 -> 0.33425 pymupdf 0.33680 PMC8103253 p1 ours 0.36263 -> 0.29524 pymupdf 0.29939 PMC8025752 p2 ours 0.23272 -> 0.17852 pymupdf 0.18125 PMC8103263 p2 ours 0.27355 -> 0.23808 pymupdf 0.24222 PMC8103279 p2 ours 0.09016 -> 0.05555 pymupdf 0.05721 The unclipped output was the outlier by 0.05-0.08 coverage; the clipped output lands within 0.004 of the reference. Annotation appearance streams are exempt. They reach the same renderer, but positioned only by a translation to the annotation's lower-left corner, where section 12.5.5 calls for the mapped bounding box to be fitted to /Rect. Clipping to a box computed under a transform that does not implement that fit trims real content: it was the one page in the sweep the reference judged a regression, and exempting the path removes it. The fit itself is not in this change. Render sweep, 561 pages, adjudicated against PyMuPDF: 7 pages moved, 7 closer to the reference, 0 regressions. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…uns unmerged Two corrections to the two commits before this one, both found by running the suites those commits had not run. The run-continuation gap bound was an em. That is narrower than a wide inter-word gap: a full-width header emitted as two show operations, with 22.6 pt between the end of the first and the start of the second at 12 pt type, was split into two spans, and the column cut then relocated its tail into the second column -- the exact failure the header fixture exists to catch. The bound is now max(3 x font size, 30 pt), which is the threshold the line grouping already uses to decide that two pieces of text belong to different columns. Making the two the same constant is the point: a displacement wide enough to be a column boundary one level down should not be a continuation one level up, and now they cannot disagree. The frame-correct rule is ANDed with the raw-matrix band again rather than replacing it. Substituting reads better and is what the writing-axis helper was built for, but it changes what a quarter-turn run does. The raw band collapses to its 0.5 pt floor under a quarter turn, so rotated runs never merge today; letting them merge concatenates them in content-stream order, which defeats the writing-axis ordering the rotated line grouping performs. A chart label that draws its subscript last then reads "H02" instead of "H2O". Merging rotated runs is still worth doing -- a run set glyph by glyph yields one span per glyph where the same text upright yields one -- but only together with an ordering rule that survives it, and that is not this change. Recorded on the issue. One pre-existing fixture is adjusted rather than satisfied. Its four rotated words sat 40 to 60 pt apart at 10 pt type, which is a column gap, not a word gap: an upright line spaced that way splits into two lines under the same threshold, so the fixture described two lines rather than the one it asserts. The words now sit about 11 pt apart and its subject -- several runs, one visual line -- is unchanged. Verification: 111 text-exposed integration suites and 5828 lib tests pass. Corpus vs main: 1 of 419 signatures, 7 of 445 golden docs. Every diff read individually -- one pair of fused numbers un-glued, six duplicates removed with the text still present. No content lost. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
Closed
3 tasks
yfedoseev
changed the title
yfedoseev
changed the title
All three pass locally and fail on the runners, and each is a real defect in the test rather than in what it tests. The encryption round-trip fixture exercised AES-128 and RC4-128, which do not exist in a FIPS build: RC4 is not an approved cipher and the R4 key derivation is built on MD5, which FIPS 140-3 forbids, so the crate makes the exclusion a build-time one. Those two revisions are now cfg'd out of the FIPS configuration; AES-256 is approved and keeps running everywhere, including the canary check that the output is genuinely encrypted. This was missed locally because the FIPS combination was type-checked but never run. The glyph-drop tests share a process-global warning collector and drain it, so with cargo running a file's tests concurrently one test could drain another's warnings. It is a race, not a platform difference; the runners just have a different core count. They now take turns over the clear/render/drain sequence. And a doc comment linked to `EncryptDictBuilder::build_with_key` from a module where that type is not in scope, which fails the warnings-as-errors documentation build. Now a fully qualified path. Verified with the configurations that caught them: the round trip passes under both default and `--no-default-features --features fips,icc`, the glyph-drop file passes five consecutive runs, and `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps` is clean. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
The pitch-scaled cap on the column-merge threshold caps it at 0.6 of the table's median inter-column gap whenever three or more columns are found. Its premise is that the median gap IS this table's pitch, and that premise was never checked. On a layout with no pitch -- most often prose the detector has mistaken for a table -- the median is the middle of a spread of unrelated gaps, and 0.6 of it forbids merging fragments that are one column. The visible cost is larger than a stray column boundary. A label and the number beside it split apart (`Line 1` -> `Line` | `1`), and because neither half then matches the flow span it was built from, the cell retention budget cannot absorb that span and the row is emitted a second time as prose beneath the table. The gate is `is_regular_lattice`, not the on-pitch ratio alone: that ratio tolerates two off-pitch gaps, which on a handful of columns is every gap there is, so it answers "yes" for any small irregular group. The lattice predicate carries the column-count floor that makes the tolerance mean something. Both now share one `gaps_are_on_pitch` helper so the ratio the cap borrows and the ratio the lattice test applies cannot drift apart. The dense-pitch case the cap exists for is unaffected -- a six-column 20 pt lattice under a 25 pt fixed threshold still keeps its columns distinct, and that test passes unchanged. Found by a v0.3.77-to-HEAD regression sweep over a feature-stratified 913-document set, scoring each surface against PyMuPDF with a multiset Jaccard so duplication counts against the score. Bisect named the commit; this restores seven regressed surfaces, the worst of them 0.667 -> 0.500 before and 0.667 after. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…vention A page whose only content is one image with a CCITT-coded /Mask painted the whole image rectangle: on issue4379 the rendered coverage was 0.39850, and 500x400 / (595x842) is 0.3992 -- the image's own rectangle to four significant figures, so the mask was having no effect at all. ISO 32000-1 section 8.9.6.3 requires the opposite: unmasked areas shall be painted with the base image and masked areas shall not be. Two defects, one on top of the other. `decompress_ccitt_group4` built its parameters with `..Default::default()` and set only the dimensions. Table 11 makes the filter's default K = 0, which is Group 3 one-dimensional, so this function -- whose entire purpose is Group 4 -- asked for the Group 3 decoder every time and never decoded anything. The failure is silent: the caller falls back to the still-compressed bytes, and a mask sampled from those runs past the end of its own data at almost every pixel. It now sets K = -1, which Table 11 defines as pure two-dimensional encoding. With the mask decoding, its polarity was still inverted. The consumer applies section 8.9.6.2 correctly -- sample 0 marks the page, so 0 is opaque -- but the CCITT decoder emits 1 for an inked pixel, which is the complement of the sample value that rule is written against: Table 11 makes `BlackIs1` false the normal PDF convention, in which 0 is black. The decoder's output is normalised at the branch that produces it, so one polarity rule serves both it and a mask whose filter the stream decoder already applied. Settled from the file rather than from a reference renderer: the decoded mask carries 15.39% set bits, and 0.1539 x 0.3992 = 0.0614 of the page, against a rendered 0.05947 once the base image's own content is taken into account. MuPDF (0.0712) and poppler (0.0655) agree, and v0.3.77 rendered 0.05947 -- by a different route, since its fallback for an unreadable mask was transparent and happened to hide the same region. Found by a v0.3.77-to-HEAD render sweep over 1298 pages, adjudicated against two independent references and claiming a verdict only where they agree. Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1136, closes #1137, closes #1138, closes #1139, closes #1140, closes #1141, closes #1142, closes #1143, closes #1144, closes #1145, closes #1146, closes #1147, closes #1148, closes #1149, closes #1151, closes #1152, closes #1153, closes #1154, closes #1155, closes #1156, closes #1157, closes #1158, closes #1159, closes #1160, closes #1161, closes #1162, closes #1163, closes #1164, closes #1165, closes #1166, closes #1167, closes #1184, closes #1150, closes #1189, closes #1190, closes #1191, closes #1194, closes #1195, closes #1196, closes #1197, closes #933, closes #1206, closes #1207, closes #1211, closes #1215, closes #1216
The v0.3.78 audit scope: 33 issues, every one either fixed here or closed
with the evidence that it needed no fix — plus nine more the release
sweep found afterwards (see "Found by the sweep" below).
The release gate
#1160 — AES-256 writing derived a second random key, so a file this
library encrypted could not be decrypted by anything, including itself.
The write handler generated its own file key while
/UEwrapped adifferent one. Reproduced by execution before being fixed: a round-trip
extracted
"". AES-128, RC4 and wrong-password paths are pinned ascontrols.
Aborts on valid files
panic = "abort"in the release profile turns each of these into a deadhost process, not a catchable error.
0.5 gfill aborts the renderer when the default colour space is an array #1146 a bare0.5 gunder a/DefaultGray [/DeviceCMYK]overridereached the four-component projection with one operand and indexed out
of bounds. §8.6.5.6 makes the declared family and the supplied operand
count independent, so the arity precondition now belongs to the
projection helpers rather than to two of their three callers.
/SMaskunderflowedsw - 1tou32::MAX.The sibling
/Maskloop had been hardened for exactly this and the twoare the same operation written twice; they now share one helper.
/PageLabelsnumber tree naming an ancestor overflowed thestack, which is not catchable.
/MediaBoxwritten on the opposite diagonal producednegative extents and the page did not render at all. §7.9.5 says a
rectangle may use either diagonal.
without a depth guard.
Rendering
/CropBoxwas parsed and never used. Pages rendered at mediasize, showing the margins the file asked to crop. Verified page-for-page
against
pdftoppm -cropboxand PyMuPDF: 13 pages change size, every onematching both references exactly.
/BBox. §8.10.2 step(c). The clip is installed at depth 0 of the nested stream's own clip
stack, so it survives however unbalanced the form's
q/Qpairs are —both cases are pinned. This reinstates a change reverted earlier in the
branch; that revert was wrong on both its mechanism and its evidence,
and the commit message sets out the reference measurements that settle
it. Annotation appearance streams are exempt until §12.5.5's fit lands.
/Masktransparency was inverted, and the pixmap was blittedwithout premultiplying — the second had to be fixed before the first
could be measured at all.
(
/ASwas not consulted) and Hidden annotations were drawn./Rotate 270page was mirrored; bothrenderers now share one page transform, which documents the
negative-determinant invariant that made them disagree.
sh, or marked-content arms, so /OC exclusion never reaches ink plates #1165 optional-content exclusion never reached the ink plates, sotwo renderers of one page gave contradictory answers about the same ink.
painting everything it was meant to hide. Resolving past an arithmetic
limit must not resolve in the direction that paints more.
as solid ink.
Text and structure
the grid and once as prose beside it. Three separate causes, each a case
of two sides disagreeing about spacing they do not have to agree on: a
repositioning jump glued two cells into one span that no cell could
claim; the retention budget could absorb a span token contained in one
budget token but not one that concatenated several; and markdown's
orphan recovery compared rendered text literally when the two sides join
their pieces differently. [Bug]: A ruled table's cell text is emitted twice — three disagreeing ownership rules #1184 needed no fix of its own once the first
landed, and its reporter's exact reproducer is pinned.
consumers. Links vanished, form values detached from their fields,
every table cell doubled, and
preserve_layoutplaced every span wrong.The frame is now a value rather than an unwritten convention, so a
consumer maps its own geometry instead of having to know.
rotated line grouping had no along-axis gap test; it now uses the same
max(3 × font size, 30 pt)the upright path uses, measured along itsown writing axis. Both quarter turns are covered;
-90°had none.ancestor and changed answer past the lazy-load threshold, and a
scope-ignoring marked-content fallback swapped Form XObject text into
table cells. The latter needed the MCID key widened to
(scope, id)infour places, not the one the issue named.
set_excluded_layerswas silently ignored above 256 KB./PlacedPDFkeep-gate tokenised encoded bytes, so anIdentity-H page extracted every word twice; it now fails closed.
Files and recovery
/Encryptand/IDfrom thetrailer and appended objects unencrypted.
cache and became permanently unreachable.
and a nondeterministic objstm walk.
/Encodeor/Decodeheld the Table 39 defaults — presence was being tested wherevalue was meant.
Colour
0 0 0 1 kconverts through the process-ink model to a darkgrey, not
(0, 0, 0), so three converters stamped a hard-coded grey intoDOCX/PPTX/XLSX instead of letting the theme supply black. A word-level
corpus diff cannot see this: the text is identical and only the colour
attribute changed.
Table 13 equates with
ColorTransform 0./Decode [1 0]scan rendered as a negative.Documentation
#1144 / #1145 two release notes claimed more than the code does — the
rotated-run bbox correction reaches no internal consumer, and
page_bboxis unreachable from WASM and is the identity on the commonest rotated-page
shape. Both notes now say what is true.
#1150, withdrawn and then fixed
This PR originally left #1150 open, on the grounds that the change it
proposed is a provable no-op on its own reproducer and that forcing it
through duplicated table text as prose. Both of those remain true of the
proposed change. The issue itself was real, and it is now fixed here by
a different mechanism.
to_htmldrops a span a table claims but renders in no cell — thedetector assigns a span to a cell by its bbox centre while ownership is
decided from the span's origin, so the two disagree at a boundary — and
markdown had a recovery pass for exactly this while HTML had none. Adding
one is what duplicated text before, because the guard deciding "the table
did not render this" asked the wrong question twice over: it read
cell.text, which is not the stringrender_cell_htmlproduces, and itcompared whitespace-normalised text, where the two sides disagree about
where the spaces go rather than about the glyphs. The comparison now runs
on glyph sequences from the same span walk the renderer uses, bounded to
a single row.
Measured over 2008 documents, on recovered paragraphs the table already
renders:
At every length where a substring test against table text means anything,
the branch is now at or below v0.3.77 — while the HTML surface drops
4,016 fewer token types than v0.3.77 does, on ten fewer documents, which
is what the recovery pass is for.
Three issues are closed with an open half recorded on them rather than
silently: #1143 (colour-key
/Maskordering needs a decode-pipelinereorder), #1159 (flat
/DeviceNDCT images error and vanish), and #1165(the separation renderer still has no inline-image or
sharms).Verification
Every fix has a synthetic reproducer built in code, named by defect class,
with recorded red-on-baseline evidence and a control that fails if the fix
over-reaches. Four fixtures were rebuilt after the revert-check showed them
vacuous.
are unrelated: one is an uncommitted local WIP file, the other fails
identically on
mainand reads a fixture from outside the repo.--all-targetson default,rendering, and--no-default-features --features fips,icc. Clippy clean.The regression figures above supersede the earlier single-reference
numbers this section used to carry. The change worth keeping is not the
figures but the method: a coverage drop only reports that fewer pixels
cleared a threshold, which is indistinguishable from a fix that stops
over-painting — reading it as a regression is what caused the #1167
revert — and it is equally blind to ink that moved without changing the
pixel count. Flagged pages are now compared against four independent
renderers on both coverage and tone, and against the file's own
arithmetic wherever the format allows it.
Found by the sweep, fixed here
None of these was reported. They came out of the v0.3.77 → v0.3.78
comparison described under "Regression" below, which scores our two arms
against four independent renderers and four independent text extractors
rather than against each other.
§12.5.5 places an appearance by mapping its
/BBoxcorners through/Matrix, taking the enclosing upright rectangle, and fitting that onto/Rect. We translated to the rectangle's lower-left corner and drew theform at whatever size it declared, so a stamp with
/BBox [0 0 512 543]inside a 93 × 98 pt
/Rectcovered a fifth of the page. The arithmeticconvicts without a reference: 512 × 543 is 57% of a 612 × 792 page and
the rectangle is 1.9% of it. Coverage on the three affected pages moves
0.21936 → 0.01503 (panel median 0.01512), 0.35857 → 0.03843 (0.03905)
and 0.15084 → 0.08384 (0.08413). With the appearance in the right
coordinate system the form's
/BBoxclip is meaningful again, so theexemption [Bug]: /BBox is not applied as a clip for form XObjects, transparency groups, or shadings #1167 left for annotations is lifted.
/Maskwas never decoded. The streamdecoder passes
JBIG2Decodethrough untouched, so the compressedbitstream reached the stencil loop, every sample fell past the end of
the buffer, and the "leave the base image visible" fallback disabled the
mask completely. Three pages of one scanned book rendered at mean tone
129–134 where MuPDF and poppler both report 246–251; they now read
246.4 / 251.1 / 248.8 against MuPDF's 245.9 / 251.1 / 248.8.
to_htmlglued words wherever a run stepped backwards. Theinline-flow separator read a negative gap as sub-em kerning and
concatenated, so
It is thecame out astheisIt. A span that endsbefore the previous one begins cannot be its continuation. Recovers
190,381 word tokens across the corpus.
bbox.top()and fell back toxonly on exact equality, so anysub-point difference put two glyphs of one line into different "rows"
and the order degenerated to a pure descending sort — whole lines came
out backwards on OCR text layers.
top()is also the wrong edge: itmoves with the font size, so a line mixing 2 pt punctuation with 8 pt
words has tops further apart than the line spacing while the baselines
agree to a fraction of a point. Now uses the banded-baseline comparator
the single-column path already used. The multi-column branch of
GeometricStrategy, which had noxtiebreak at all, takes the samecomparator.
Three more the sweep exposed are filed and not fixed here, because
none is root-caused and each needs its own investigation: #1198 (a
stencil under
/Decode [1 0]and a bit-packed PNG predictor paints ~2×the panel), #1199 (a small text page renders at 41% of the panel
median), and #1200 (two low-magnitude divergences — JPX +
/SMask,and Type 1 CM glyph weight).
Regression: v0.3.77 vs v0.3.78
Scored against the tag, not
main, over 2008 documents and 3214 pages,with two panels of four independent implementations:
pdftoppm(-cropbox), Ghostscript 10.00pdftotextA difference from a reference is a question, not a verdict — the four
renderers disagree with each other by up to 27% on a single page — so a
value inside the panel's own spread is not a defect, a page whose extent
changed is not comparable by coverage at all, and anything outside the
whole panel is settled from the file and the spec rather than by
tie-break.
Rendering, 3214 pages, adjudicated page by page against all four on both
coverage and mean tone:
The 13 remaining "away" pages are eight documents and every one is
pre-existing — seven render byte-identically in both arms, and the other
two moved by less than half their pre-existing gap. All nine are filed
(#1199, #1200, #1202, #1203, #1204), except
issue4246, which #1198withdrew once tone showed we match MuPDF and Ghostscript to 0.3 of a grey
level.
Reading both quantities matters more than it sounds. Coverage counts pixels
under a threshold, so it measures edge softness as much as ink; mean tone
measures how much ink was laid down. Two of the pages the coverage metric
flagged turned out to be nothing —
issue4246differs from the panel by2.1x on coverage while agreeing to 0.3 of a grey level out of 255 on tone,
and its expected ink can be derived from the file exactly (
0.15388of thestencil's samples set, times the
500 x 400 / (595 x 842) = 0.39918theimage occupies, gives
0.06143; Ghostscript reports0.06144). Threepages the coverage metric could not see were only visible on tone.
The 67 extent changes are the
/CropBoxfix. Coverage is a fraction anda crop changes the denominator, so they are not comparable that way and
were judged against the file instead: all 55 unique pages now render at
exactly the size their
/CropBoxdeclares, accounting for/Rotate.Two of this release's late fixes are visible in that figure. A JPEG 2000
image with no declared
/ColorSpacewas rejected outright and its pagerendered blank (#1211); it now renders at mean tone 105.73 against MuPDF's
105.73. And a Pattern colour space reached through a resource name was not
recognised as one, so a tiling fill painted solid black; that page moves
inside the panel on both coverage and tone. Both were found by adjudicating
pages that render identically in both arms — 2688 of 3214 — which no
regression sweep examines, because unchanged is exactly what hides a defect
we have always had.
.mdand.htmlagainst engines that emit those surfaces. No enginein the render or text panels produces markdown or HTML, so until this run
those two surfaces could only be compared arm-to-arm and against our own
.text. Two do: popplerpdftohtmlandpymupdf4llm. With one engine persurface there is no majority to appeal to, so the gate is two-of-two
instead — a token that BOTH the external engine and our own
.textreportis one the page draws, and the surface must carry it.
Two documents on
.htmlwent from 203 and 129 missing to zero. This runalso found #1206 — all three cell renderers ran a cell's vertically
stacked members together, because they asked only
has_horizontal_gap,which compares x — and it is fixed here.
Text,
.text/.md/.htmlover all 2008 documents, plus a164-document panel sample:
Eight documents improve, four lose ground, and every one of the four was
opened:
source.pdfloses no token type at all (the delta is instancecounts, which §9.4.3 makes formatting);
a4bf0c8a…is correctdehyphenation the reference does not perform;
TAMReview.pdfis #1201;030ae7ca…is #1208, filed for a decision rather than fixed, because ouroutput is what the page paints and the panel's disagreement is the
artefact.
This run also found and fixed #1207, a real regression: dehyphenation
ate a compound's own hyphen, turning
Cross-sectionalintoCrosssectionalandReceiver-operatingintoReceiveroperating. Atypesetter breaking an already-hyphenated word writes the real hyphen and
then U+00AD; stripping the marker before the wrap decision left a bare
Cross-, which the rejoiner then read as the marker and removed in turn.Arm against arm on our own three surfaces, scored by the §9.4.3 rule that
a drawn token must appear at least once and counting separately the types
that disappeared by being joined into a longer token or split into
several:
.text.md.htmlThe
.mdfigure is dominated by the removal of the injected[OCR REQUIRED — page N]prose (#1189, #933) — those documents have notext layer, so the marker was their entire markdown. The rest, on all
three surfaces, are fused compounds that no longer get produced:
balancto,Comvision,insectthe,ありぶ. Every one was readindividually.
Cross-surface, which the panel structurally cannot see: comparing our
own
.htmlagainst our own.texton the same document, the HTML surfacerecovers 190,381 word tokens it had been fusing, and on the document in
#1194 it now reproduces
extract_texttoken for token.