DOC: Execute docs examples in CI #3507

ievgen-kapinos · 2025-10-27T12:52:17Z

In this PR added execution for all python code snippets in documentation via sphinx.ext.doctest

In this PR:

Added a new CI step with Sphinx's doctest build. Effectively we test:
- code blocks marked with testcode directive in *.md files
- code blocks from python's docstrings imported via autoclass directive in *.rst files
Added documentation how to execute code snippets locally
Made all output files unique. So it is possible to check output of all snippets in docs/_build/doctest/pypdf_test directory
Reconfigured CI pipeline steps for all Sphinx builds (html and doctest). Now sphinx-build tool is run in docs directory and we have output file structure in docs/_build that match:
- Read the Docs build. See example for this PR
- local build with make tool as described in Sphinx's official docs
Code snippets now executable. But it make sense to review all used input files and make sure that content match text and images used in documentation. It is not always the case. I consider it as out of scope of this task.
Few tests marked as :skipif: True
- We do not test code which require access to Cloud: AWS and GCP
- We do not test code which require additional dependencies: svgwrite and pdfminer.six. These can be added in requirements/docs.in. Please let me know if we need them.

P.S. Read the Docs has nice feature to view diff between main and PR branch. See diff for this PR

codecov · 2025-10-27T13:01:33Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.10%. Comparing base (3b5c85f) to head (9cefe93).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3507   +/-   ##
=======================================
  Coverage   97.10%   97.10%           
=======================================
  Files          57       57           
  Lines        9711     9711           
  Branches     1759     1759           
=======================================
  Hits         9430     9430           
  Misses        168      168           
  Partials      113      113

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ievgen-kapinos · 2025-10-27T14:05:01Z

Hi @j-t-1 and @stefan6419846

May I ask you to pre-review overall approach?

As for now implemented only execution via CI. So far we have 72 failures. See full list here

If in general this approach is acceptable, then in scope of this PR I'll:

fix all examples in docs. I expect PR will be very big after it. So I'd request pre-review
add description in documentation on how it works and how we can execute it locally

ievgen-kapinos · 2025-10-27T14:17:58Z

FYI Code Blocks in HTML rendered as before after I've replaced python with {testcode}

stefan6419846 · 2025-10-27T15:56:57Z

.github/workflows/github-ci.yaml

        sphinx-build --nitpicky --fail-on-warning --keep-going --show-traceback --builder html docs build/sphinx/html
+    - name: Test docs examples
+      run: |
+        sphinx-build --nitpicky --fail-on-warning --keep-going --show-traceback --builder doctest docs build/sphinx/html


Are all of these parameters required for this call? I am especially not sure about the output directory, as just running doctests should not write any HTML in theory?

These parameters do nothing. I've removed them. Thanks!

As for output directory - it is necessary. Extension sphinx.ext.doctest creates .doctrees sub-folder with *.doctree files. I suppose these files contain documentation in some universal format which than can be transformed to variety of different formats. Previous CI step (when we run sphinx-build ... --builder html ...) also creates this sub-folder. As I see Sphinx is smart enough and looks for outdated files. So it makes sense to point to build/sphinx/html folder.

…le.pdf

ievgen-kapinos · 2025-11-01T10:37:02Z

Hi @stefan6419846 and @j-t-1

This PR is ready for review. Please check out updated description first.

stefan6419846

Thanks for the PR.

I have left some reviews comments where your changes were not clear to me. Please note that some comments might cover other files with similar patterns as well.

docs/dev/documentation.md

docs/modules/PaperSize.rst

docs/conf.py

resources/nup-source.pdf

ievgen-kapinos

@stefan6419846 Thanks a lot for the review! It is big and requires a lot of effort from your side. I appreciate it.

I've covered all your questions. Many of them were just fixed (and marked with 👍). As for the rest, I've added explanations and sometimes questions. Let me now what need to be done next in this PR

stefan6419846 · 2025-11-05T13:35:42Z

docs/user/add-watermark.md

    page.merge_page(stamp, over=False)  # here set to False for watermarking

-writer.write("out.pdf")
+writer.write("out-underlay.pdf")


Do we use "underlay" somewhere else as well? Otherwise I would go with consistent naming like "stamp" for over and "watermark" for under, which seems to be the current naming scheme?

stefan6419846 · 2025-11-05T13:40:58Z

docs/user/suppress-warnings.md

 want the text. You might use pdfminer.six as a fallback and do this:

-```python
+% We prefer not to execute doc examples for third-party package "pdfminer" used only in one code snippet


The package is named pdfminer.six:

Suggested change

% We prefer not to execute doc examples for third-party package "pdfminer" used only in one code snippet

% We prefer not to execute doc examples for third-party package "pdfminer.six" used in one code snippet only

Yeah... missed. It was mentioned 3 lines above 👍

stefan6419846 · 2025-11-05T13:43:54Z

docs/conf.py

+        file = os.path.join(dst_root_dir, file_name)
+        if os.path.isfile(file):
+            if not has_files:
+                print("Docs page was not configured propery for running code examples")


I guess these should print to stderr instead of stdout?

In case of stderr doctest ignores output see official doc. Just double-checked, message appears in output, but test not fail. My intension is opposite, force to use {testsetup} directive if doc examples spit output files.

stefan6419846 · 2025-11-05T13:44:13Z

docs/conf.py

+                print("Deleting unexpected file(s) in " + dst_root_dir)
+                has_files = True
+            print(f"- {{file_name}}")
+            os.remove(file)  # We should not affect other tests


Suggested change

os.remove(file) # We should not affect other tests

os.remove(file) # Avoid side effects on other tests

stefan6419846 · 2025-11-05T13:45:20Z

docs/conf.py

+
+    has_files = False
+    for file_name in os.listdir(dst_root_dir):
+        file = os.path.join(dst_root_dir, file_name)


Please avoid using the (previously) builtin name file to avoid confusion.

Done... thanks. I've learned something new 👍

ievgen-kapinos added 2 commits October 27, 2025 13:33

Enable sphinx.ext.doctest extension

40c08bd

Replace python basic blocks with testcode directive in *.md files

0a8e021

ievgen-kapinos added 2 commits October 27, 2025 14:12

Add new CI step

6d16058

Fix typo

063950f

stefan6419846 reviewed Oct 27, 2025

View reviewed changes

ievgen-kapinos added 22 commits October 27, 2025 18:19

removed unnesessary arguments form new CI step

a8a2a18

Fixed first 30 tests

4b42e33

Fix CI

5d46e69

Fix CI py-pdf#2

497c6e2

Removed garbage

50aa2ab

run doctest from root

a2d191b

Remove garbage

485d6d3

Fix in path

5885cbd

Remove garbage again

8dbfc4a

re-fix

36c97d0

Run Sphinx in docs directory

5c1ba4d

Added comments above skipped test

748b2cd

Remove garbage

24409ba

Remove garbage

4799520

Replaced ../resources/side-by-side-subfig.pdf with self created examp…

bee4ca7

…le.pdf

Fixed 7 test in user/merging-pdfs

2f67992

Fix 2 tests in docs\user\handle-attachments.md

01ff206

Fix more errors in docs\user\extract-text.md

c816752

Normalize

53975f1

less spaces

f4f680d

Fix errors in docs\user\extract-images.md

bd09de3

Fix tests in docs\user\cropping-and-transforming.md

00a7f11

ievgen-kapinos added 8 commits October 31, 2025 20:29

Removed revative filenames

a530f11

Added new docs

9e7083e

Polishing

62792f2

polishing

2c58519

polishing

cd9a5eb

Pilishing

5d11a51

Polishing

6bac8c3

polishing

2035c50

ievgen-kapinos marked this pull request as ready for review November 1, 2025 10:36

ievgen-kapinos added 2 commits November 1, 2025 19:25

Deleted garbage

691ba58

Fix formatting in docs\user\merging-pdfs.md

7d5d9e5

stefan6419846 requested changes Nov 3, 2025

View reviewed changes

Corrections after PR review

79185c4

ievgen-kapinos commented Nov 3, 2025

View reviewed changes

ievgen-kapinos requested a review from stefan6419846 November 3, 2025 21:42

ievgen-kapinos added 7 commits November 4, 2025 15:41

Corrections after PR review

48f9833

Moved each tested file in own sub-folder

e586c79

droppped nup-source.pdf

7ebf235

Removed file prefix. Applied common

2a69f06

removed index from out prefix

97bd980

Polishing

540c13b

Added description for pypdf_test_setup function

4bd5a14

stefan6419846 reviewed Nov 5, 2025

View reviewed changes

ievgen-kapinos added 2 commits November 5, 2025 15:15

Code review fixes

a41917e

Merge branch 'main' into iss-2610-doctest-via-sphinx

9cefe93

	% We prefer not to execute doc examples for third-party package "pdfminer" used only in one code snippet
	% We prefer not to execute doc examples for third-party package "pdfminer.six" used in one code snippet only

	os.remove(file) # We should not affect other tests
	os.remove(file) # Avoid side effects on other tests

DOC: Execute docs examples in CI #3507

Are you sure you want to change the base?

DOC: Execute docs examples in CI #3507

Uh oh!

Conversation

ievgen-kapinos commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ievgen-kapinos commented Oct 27, 2025

Uh oh!

ievgen-kapinos commented Oct 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ievgen-kapinos commented Nov 1, 2025

Uh oh!

stefan6419846 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ievgen-kapinos left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ievgen-kapinos Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ievgen-kapinos Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ievgen-kapinos commented Oct 27, 2025 •

edited

Loading

codecov bot commented Oct 27, 2025 •

edited

Loading

ievgen-kapinos left a comment •

edited

Loading

ievgen-kapinos Nov 5, 2025 •

edited

Loading

ievgen-kapinos Nov 5, 2025 •

edited

Loading