Skip to content

Commit 8ea8b47

Browse files
authored
Deeper inspection, follow simple type hints (#19)
1 parent 55bdfec commit 8ea8b47

File tree

11 files changed

+566
-210
lines changed

11 files changed

+566
-210
lines changed

docs/src/about.rst

Lines changed: 26 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -10,26 +10,32 @@ Caveats
1010
is off, it silently removes directives that would produce output.
1111
- **Only processes literal blocks, not inline code**. Sphinx has great tools
1212
for linking definitions inline, and longer code should be in a block anyway.
13-
- **Doesn't run code or follow importable type hints**. Therefore all
14-
possible resolvable names are not found, and the runtime correctness of code
15-
cannot be validated. Nonsensical operations that would result in errors at
16-
runtime are possible. However, syntax errors are caught while parsing!
17-
- **Parsing is incomplete and performed top-down**. While all Python syntax is
18-
supported, some cases are ambiguous and produce unintuitive results or even
19-
incorrect results when compared to runtime behavior. Firstly, assignments
20-
are not followed. For example, assigning :code:`cal = sphinx_codeautolink`
21-
and using its attributes like :code:`cal.setup()` does not produce a link.
22-
Deleting or assigning to a global variable from an inner scope is
23-
not recognised in outer scopes. This is because the value depends on when
24-
the function is called, which is not tracked. Additionally, variable values
25-
are assumed to be static after leaving an inner scope, i.e. a function
26-
referencing a global variable. This is not the case in Python: values may
27-
change after the definition and impact the function. For more details on the
28-
expected failures, see our test suite on `GitHub <https://github.com/
29-
felix-hilden/sphinx-codeautolink>`_. Please report any unexpected failures!
30-
Because the library's purpose is to highlight the definitions used from
31-
imported modules, these shortcomings are likely minor, and only occur in
32-
practice when a variable shadows or overwrites an imported module.
13+
- **Doesn't run example code**. Therefore all possible resolvable types are not
14+
found, and the runtime correctness of code cannot be validated.
15+
Nonsensical operations that would result in errors at runtime are possible.
16+
However, syntax errors are caught while parsing!
17+
- **Parsing and type hint resolving is incomplete**. While all Python syntax is
18+
supported, some ambiguous cases might produce unintuitive results or even
19+
incorrect results when compared to runtime behavior. We try to err on the
20+
side of caution, but here are some examples of compromises and limitations:
21+
22+
- Only simple assignments of names, attributes and calls to a single name
23+
are tracked and used to resolve later values.
24+
- Only simple return type hints that consists of a single resolved type
25+
(not a string) are tracked through call and attribute access chains.
26+
- Type hints of intersphinx-linked definitions are not available.
27+
- Deleting or assigning to a global variable from an inner scope is
28+
not recognised in outer scopes. This is because the value depends on when
29+
the function is called, which is not tracked. Additionally, variable values
30+
are assumed to be static after leaving an inner scope, i.e. a function
31+
referencing a global variable. This is not the case in Python: values may
32+
change after the definition and impact the function.
33+
Encountering this should be unlikely, because it only occurs in practice
34+
when a variable shadows or overwrites an imported module or its part.
35+
36+
These cases are subject to change when the library matures. For more details
37+
on the expected failures, see our test suite on `GitHub <https://github.com
38+
/felix-hilden/sphinx-codeautolink>`_. Please report any unexpected failures!
3339

3440
Clean Sphinx build
3541
------------------

docs/src/examples.rst

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -184,8 +184,9 @@ and their use in examples does not represent their correct usage.
184184

185185
sphinx-codeautolink
186186
*******************
187-
.. automodule:: sphinx_codeautolink
187+
.. autofunction:: sphinx_codeautolink.setup
188188

189189
sphinx-codeautolink.parse
190190
*************************
191-
.. automodule:: sphinx_codeautolink.parse
191+
.. autoclass:: sphinx_codeautolink.parse.Name
192+
.. autofunction:: sphinx_codeautolink.parse.parse_names

docs/src/index.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,8 +66,8 @@ For a more thorough explanation, see :ref:`about`.
6666

6767
- Only works with HTML documentation
6868
- Only processes literal blocks, not inline code
69-
- Doesn't run code or follow importable type hints
70-
- Parsing is incomplete and performed top-down
69+
- Doesn't run example code
70+
- Parsing and type hint resolving is incomplete
7171

7272
Thanks
7373
------

docs/src/release_notes.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ sphinx-codeautolink adheres to
1010

1111
Unreleased
1212
----------
13+
- Improve code analysis and follow simple type hints (:issue:`5`)
1314
- Improve directive arguments and behavior (:issue:`16`)
1415
- Correctly consume :code:`autolink-skip:: next` (:issue:`17`)
1516

src/sphinx_codeautolink/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,8 @@ def setup(app: Sphinx):
2525
app.add_directive('autolink-skip', AutoLinkSkip)
2626

2727
app.connect('builder-inited', state.build_inited)
28+
app.connect('autodoc-process-docstring', state.autodoc_process_docstring)
2829
app.connect('doctree-read', state.parse_blocks)
2930
app.connect('doctree-resolved', state.generate_backref_tables)
3031
app.connect('build-finished', state.apply_links)
31-
app.connect('autodoc-process-docstring', state.autodoc_process_docstring)
3232
return {'version': __version__}

src/sphinx_codeautolink/extension/__init__.py

Lines changed: 105 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,23 @@
11
"""Sphinx extension implementation."""
22
import json
3-
import posixpath
43

5-
from typing import Dict, List
4+
from dataclasses import dataclass, asdict
5+
from typing import Dict, List, Optional
66
from pathlib import Path
7-
from warnings import warn
87

9-
from sphinx.ext.intersphinx import fetch_inventory, INVENTORY_FILENAME, InventoryAdapter
8+
from sphinx.ext.intersphinx import InventoryAdapter
109

1110
from .backref import CodeRefsVisitor, CodeExample
12-
from .block import CodeBlockAnalyser, link_html
11+
from .block import CodeBlockAnalyser, link_html, Name, NameBreak
12+
13+
14+
@dataclass
15+
class DocumentedObject:
16+
"""Autodoc-documented code object."""
17+
18+
what: str
19+
obj: object
20+
return_type: str = None
1321

1422

1523
class SphinxCodeAutoLink:
@@ -19,20 +27,77 @@ class SphinxCodeAutoLink:
1927

2028
def __init__(self):
2129
self.code_refs: Dict[str, Dict[str, List[CodeExample]]] = {}
30+
self._flat_refs: Dict[str, List[CodeExample]] = {}
2231
self.block_visitors: List[CodeBlockAnalyser] = []
2332
self.do_nothing = False
24-
self._flat_refs = {}
33+
self.objects: Dict[str, DocumentedObject] = {}
34+
self._inventory = {}
2535

26-
@property
27-
def flat_refs(self):
36+
def make_flat_refs(self, app):
2837
"""Flattened version of :attr:`code_refs`."""
29-
if not self._flat_refs:
30-
for refs in self.code_refs.values():
31-
for doc, examples in refs.items():
32-
self._flat_refs.setdefault(doc, []).extend(examples)
38+
if self._flat_refs:
39+
return self._flat_refs
40+
41+
self.parse_transforms(app)
42+
43+
for refs in self.code_refs.values():
44+
for doc, examples in refs.items():
45+
self._flat_refs.setdefault(doc, []).extend(examples)
3346

3447
return self._flat_refs
3548

49+
def parse_transforms(self, app):
50+
"""Construct code_refs and try to link name chains."""
51+
inventory = self.make_inventory(app)
52+
for visitor in self.block_visitors:
53+
refs = {}
54+
for transform in visitor.source_transforms:
55+
filtered = []
56+
for name in transform.names:
57+
key = self.find_in_objects(name)
58+
if not key or key not in inventory:
59+
continue
60+
name.resolved_location = key
61+
filtered.append(name)
62+
refs.setdefault(key, []).append(transform.example)
63+
transform.names = filtered
64+
self.code_refs[visitor.current_document] = refs
65+
66+
def find_in_objects(self, chain: Name) -> Optional[str]:
67+
"""Find the final type that a name refers to."""
68+
comps = []
69+
for comp in chain.import_components:
70+
if comp.name == NameBreak.call:
71+
name = '.'.join(comps)
72+
if name in self.objects and self.objects[name].return_type:
73+
comps = [self.objects[name].return_type]
74+
continue
75+
else:
76+
return
77+
comps.append(comp.name)
78+
return '.'.join(comps)
79+
80+
def make_inventory(self, app):
81+
"""Create object inventory from local info and intersphinx."""
82+
if self._inventory:
83+
return self._inventory
84+
85+
inv_parts = {
86+
k: str(
87+
Path(app.outdir)
88+
/ (app.builder.get_target_uri(v.docname) + f'#{v.node_id}')
89+
)
90+
for k, v in app.env.domains['py'].objects.items()
91+
}
92+
inventory = {'py:class': {
93+
k: (None, None, v, None) for k, v in inv_parts.items()
94+
}}
95+
inter_inv = InventoryAdapter(app.env).main_inventory
96+
transposed = transpose_inventory(inter_inv, relative_to=app.outdir)
97+
transposed.update(transpose_inventory(inventory, relative_to=app.outdir))
98+
self._inventory = transposed
99+
return self._inventory
100+
36101
def build_inited(self, app):
37102
"""Handle initial setup."""
38103
if app.builder.name != 'html':
@@ -54,28 +119,41 @@ def build_inited(self, app):
54119
if not full_path.exists():
55120
continue
56121
self.code_refs[file] = {
57-
obj: [
58-
CodeExample(e['document'], e['ref_id'], e['headings'])
59-
for e in examples
60-
]
122+
obj: [CodeExample(**e) for e in examples]
61123
for obj, examples in ref.items()
62124
}
63125

126+
def autodoc_process_docstring(self, app, what, name, obj, options, lines):
127+
"""Handle autodoc-process-docstring event."""
128+
if self.do_nothing:
129+
return
130+
131+
if app.config.codeautolink_autodoc_inject:
132+
lines.append(f'.. code-refs:: {name}')
133+
134+
d_obj = DocumentedObject(what, obj)
135+
if what in ('class', 'exception'):
136+
d_obj.annotation = name
137+
elif what in ('function', 'method'):
138+
ret_annotation = obj.__annotations__.get('return', None)
139+
if ret_annotation and not hasattr(ret_annotation, '__origin__'):
140+
d_obj.annotation = getattr(ret_annotation, '__name__', None)
141+
self.objects[name] = d_obj
142+
64143
def parse_blocks(self, app, doctree):
65144
"""Parse code blocks for later link substitution."""
66145
if self.do_nothing:
67146
return
68147

69148
visitor = CodeBlockAnalyser(doctree, source_dir=app.srcdir)
70149
doctree.walkabout(visitor)
71-
self.code_refs[visitor.current_document] = visitor.code_refs
72150
self.block_visitors.append(visitor)
73151

74152
def generate_backref_tables(self, app, doctree, docname):
75153
"""Generate backreference tables."""
76154
visitor = CodeRefsVisitor(
77155
doctree,
78-
code_refs=self.flat_refs,
156+
code_refs=self.make_flat_refs(app),
79157
remove_directives=self.do_nothing,
80158
)
81159
doctree.walk(visitor)
@@ -85,50 +163,30 @@ def apply_links(self, app, exception):
85163
if self.do_nothing or exception is not None:
86164
return
87165

88-
inv_file = posixpath.join(app.outdir, INVENTORY_FILENAME)
89-
if not Path(inv_file).exists():
90-
msg = (
91-
'sphinx-codeautolink: cannot locate object inventory '
92-
f' in {INVENTORY_FILENAME}, no links applied'
93-
)
94-
warn(msg, RuntimeWarning)
95-
return
96-
97-
inv = fetch_inventory(app, app.outdir, inv_file)
98-
inter_inv = InventoryAdapter(app.env).main_inventory
99-
transposed = transpose_inventory(inter_inv, relative_to=app.outdir)
100-
transposed.update(transpose_inventory(inv, relative_to=app.outdir))
101-
102166
for visitor in self.block_visitors:
103167
if not visitor.source_transforms:
104168
continue
105169
file = Path(app.outdir) / (visitor.current_document + '.html')
106-
link_html(file, app.outdir, visitor.source_transforms, transposed)
170+
link_html(
171+
file, app.outdir, visitor.source_transforms, self.make_inventory(app)
172+
)
107173

108174
refs_file = Path(app.srcdir) / self.code_refs_file
109175
refs = {}
110176
for file, ref in self.code_refs.items():
111177
refs[file] = {
112-
obj: [
113-
{'document': e.document, 'ref_id': e.ref_id, 'headings': e.headings}
114-
for e in examples
115-
]
178+
obj: [asdict(e) for e in examples]
116179
for obj, examples in ref.items()
117180
}
118181
refs_file.write_text(json.dumps(refs, indent=4), 'utf-8')
119182

120-
def autodoc_process_docstring(self, app, what, name, obj, options, lines):
121-
"""Inject code-refs tables to docstrings."""
122-
if not app.config.codeautolink_autodoc_inject or self.do_nothing:
123-
return
124-
125-
lines.append(f'.. code-refs:: {name}')
126-
127183

128184
def transpose_inventory(inv: dict, relative_to: str):
129185
"""
130186
Transpose Sphinx inventory from {type: {name: (..., location)}} to {name: location}.
131187
188+
Also filters the inventory to Python domain only.
189+
132190
Parameters
133191
----------
134192
inv
@@ -137,7 +195,9 @@ def transpose_inventory(inv: dict, relative_to: str):
137195
if a local file is found, transform it to be relative to this dir
138196
"""
139197
transposed = {}
140-
for _, items in inv.items():
198+
for type_, items in inv.items():
199+
if not type_.startswith('py:'):
200+
continue
141201
for item, info in items.items():
142202
location = info[2]
143203
if not location.startswith('http'):

0 commit comments

Comments
 (0)