wikipedia: sections parsed incorrectly

[SnoopJ]

Handling a link to a section of a Wikipedia article appears to be broken:

<SnoopJ> https://en.wikipedia.org/wiki/Attractor
<mediocrebot> [wikipedia] Attractor | "In the mathematical field of dynamical systems, an attractor is a set of states toward which a system tends to evolve, for a wide variety of starting conditions of the system. System values that get close enough to the attractor values remain close even if slightly disturbed. In finite-dimensional systems, the evolving variable may be represented algebraically as an n-dimensional vector. The attractor is a region […]"
<SnoopJ> https://en.wikipedia.org/wiki/Attractor#Strange_attractor
<mediocrebot> [wikipedia] Error fetching section "Strange_attractor" for page "Attractor".

The requests we make to wikipedia.org are coming back fine and do contain the data we want to see, but the parse result is an empty string, so this is some kind of parser problem.

Here's a self-contained repro using the real data from requesting this section:

from sopel_wikimedia.wikipedia.parser import WikiParser

section = "Strange attractor"
data = {
    "parse": {
        "pageid": 191094,
        "redirects": [],
        "text": {
            "*": '<div class="mw-content-ltr mw-parser-output" lang="en" dir="ltr"><div class="mw-heading mw-heading3"><h3 id="Strange_attractor">Strange attractor</h3><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=Attractor&amp;action=edit&amp;section=1" title="Edit section: Strange attractor"><span>edit</span></a><span class="mw-editsection-bracket">]</span></span></div>\n<style data-mw-deduplicate="TemplateStyles:r1320445320">.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}</style><div role="note" class="hatnote navigation-not-searchable">"Strange attractor" redirects here. For other uses, see <a href="/wiki/Strange_Attractor_(disambiguation)" class="mw-redirect mw-disambig" title="Strange Attractor (disambiguation)">Strange Attractor (disambiguation)</a>.</div>\n<figure class="mw-halign-right" typeof="mw:File/Thumb"><a href="/wiki/File:Lorenz_attractor_yb.svg" class="mw-file-description"><img src="//upload.wikimedia.org/wikipedia/commons/thumb/5/5b/Lorenz_attractor_yb.svg/250px-Lorenz_attractor_yb.svg.png" decoding="async" width="200" height="200" class="mw-file-element" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/5/5b/Lorenz_attractor_yb.svg/330px-Lorenz_attractor_yb.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/5/5b/Lorenz_attractor_yb.svg/500px-Lorenz_attractor_yb.svg.png 2x" data-file-width="750" data-file-height="750" /></a><figcaption>A plot of <a href="/wiki/Lorenz%27s_strange_attractor" class="mw-redirect" title="Lorenz&#39;s strange attractor">Lorenz\'s strange attractor</a> for values&#160;<i>ρ</i>&#160;=&#160;28,&#160;<i>σ</i>&#160;=&#160;10,&#160;<i>β</i>&#160;=&#160;8/3</figcaption></figure>\n<p>An attractor is called <i>strange</i> if it has a <a href="/wiki/Fractal" title="Fractal">fractal</a> structure, that is if it has non-integer <a href="/wiki/Hausdorff_dimension" title="Hausdorff dimension">Hausdorff dimension</a>. This is often the case when the dynamics on it are <a href="/wiki/Chaos_theory" title="Chaos theory">chaotic</a>, but <a href="/wiki/Strange_nonchaotic_attractor" title="Strange nonchaotic attractor">strange nonchaotic attractors</a> also exist. If a strange attractor is chaotic, exhibiting <a href="/wiki/Sensitive_dependence_on_initial_conditions" class="mw-redirect" title="Sensitive dependence on initial conditions">sensitive dependence on initial conditions</a>, then any two arbitrarily close alternative initial points on the attractor, after any of various numbers of iterations, will lead to points that are arbitrarily far apart (subject to the confines of the attractor), and after any of various other numbers of iterations will lead to points that are arbitrarily close together. Thus a dynamic system with a chaotic attractor is locally unstable yet globally stable: once some sequences have entered the attractor, nearby points diverge from one another but never depart from the attractor.<sup id="cite&#95;ref-1" class="reference"><a href="#cite_note-1"><span class="cite-bracket">&#91;</span>1<span class="cite-bracket">&#93;</span></a></sup>\n</p><p>The term <i>strange attractor</i> was coined by <a href="/wiki/David_Ruelle" title="David Ruelle">David Ruelle</a> and <a href="/wiki/Floris_Takens" title="Floris Takens">Floris Takens</a> to describe the attractor resulting from a series of <a href="/wiki/Bifurcation_theory" title="Bifurcation theory">bifurcations</a> of a system describing fluid flow.<sup id="cite&#95;ref-2" class="reference"><a href="#cite_note-2"><span class="cite-bracket">&#91;</span>2<span class="cite-bracket">&#93;</span></a></sup> Strange attractors are often <a href="/wiki/Differentiable_function" title="Differentiable function">differentiable</a> in a few directions, but some are <a href="/wiki/Homeomorphic" class="mw-redirect" title="Homeomorphic">like</a> a <a href="/wiki/Cantor_dust" class="mw-redirect" title="Cantor dust">Cantor dust</a>, and therefore not differentiable. Strange attractors may also be found in the presence of noise, where they may be shown to support invariant random probability measures of Sinai–Ruelle–Bowen type.<sup id="cite&#95;ref-Stochastic&#95;climate&#95;dynamics:&#95;Random&#95;attractors&#95;and&#95;time-dependent&#95;invariant&#95;measures&#95;3-0" class="reference"><a href="#cite_note-Stochastic_climate_dynamics:_Random_attractors_and_time-dependent_invariant_measures-3"><span class="cite-bracket">&#91;</span>3<span class="cite-bracket">&#93;</span></a></sup>\n</p><p>Examples of strange attractors include the <a href="/wiki/Double_scroll_attractor" class="mw-redirect" title="Double scroll attractor">double-scroll attractor</a>, <a href="/wiki/H%C3%A9non_map" title="Hénon map">Hénon attractor</a>, <a href="/wiki/R%C3%B6ssler_attractor" title="Rössler attractor">Rössler attractor</a>, and <a href="/wiki/Lorenz_attractor" class="mw-redirect" title="Lorenz attractor">Lorenz attractor</a>.\n</p>\n<div class="mw-references-wrap"><ol class="references">\n<li id="cite&#95;note-1"><span class="mw-cite-backlink"><b><a href="#cite_ref-1">^</a></b></span> <span class="reference-text"><style data-mw-deduplicate="TemplateStyles:r1238218222">.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\\"""\\"""\'""\'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("//upload.wikimedia.org/wikipedia/commons/6/65/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("//upload.wikimedia.org/wikipedia/commons/d/d6/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("//upload.wikimedia.org/wikipedia/commons/a/aa/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("//upload.wikimedia.org/wikipedia/commons/4/4c/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}</style><cite id="CITEREFGrebogi&#95;Celso,&#95;Ott&#95;Edward,&#95;Yorke&#95;James&#95;A1987" class="citation journal cs1">Grebogi Celso, Ott Edward, Yorke James A (1987). "Chaos, Strange Attractors, and Fractal Basin Boundaries in Nonlinear Dynamics". <i>Science</i>. <b>238</b> (4827): <span class="nowrap">632–</span>638. <a href="/wiki/Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/1987Sci...238..632G">1987Sci...238..632G</a>. <a href="/wiki/Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1126%2Fscience.238.4827.632">10.1126/science.238.4827.632</a>. <a href="/wiki/PMID_(identifier)" class="mw-redirect" title="PMID (identifier)">PMID</a>&#160;<a rel="nofollow" class="external text" href="https://pubmed.ncbi.nlm.nih.gov/17816542">17816542</a>. <a href="/wiki/S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a>&#160;<a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:1586349">1586349</a>.</cite><span title="ctx&#95;ver=Z39.88-2004&amp;rft&#95;val&#95;fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&amp;rft.genre=article&amp;rft.jtitle=Science&amp;rft.atitle=Chaos%2C+Strange+Attractors%2C+and+Fractal+Basin+Boundaries+in+Nonlinear+Dynamics&amp;rft.volume=238&amp;rft.issue=4827&amp;rft.pages=632-638&amp;rft.date=1987&amp;rft&#95;id=info%3Adoi%2F10.1126%2Fscience.238.4827.632&amp;rft&#95;id=https%3A%2F%2Fapi.semanticscholar.org%2FCorpusID%3A1586349%23id-name%3DS2CID&amp;rft&#95;id=info%3Apmid%2F17816542&amp;rft&#95;id=info%3Abibcode%2F1987Sci...238..632G&amp;rft.au=Grebogi+Celso%2C+Ott+Edward%2C+Yorke+James+A&amp;rfr&#95;id=info%3Asid%2Fen.wikipedia.org%3AAttractor" class="Z3988"></span><span class="cs1-maint citation-comment"><code class="cs1-code">{{<a href="/wiki/Template:Cite_journal" title="Template:Cite journal">cite journal</a>}}</code>:  CS1 maint: multiple names: authors list (<a href="/wiki/Category:CS1_maint:_multiple_names:_authors_list" title="Category:CS1 maint: multiple names: authors list">link</a>)</span></span>\n</li>\n<li id="cite&#95;note-2"><span class="mw-cite-backlink"><b><a href="#cite_ref-2">^</a></b></span> <span class="reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1238218222" /><cite id="CITEREFRuelleTakens1971" class="citation journal cs1">Ruelle, David; Takens, Floris (1971). <span class="id-lock-subscription" title="Paid subscription required"><a rel="nofollow" class="external text" href="http://projecteuclid.org/euclid.cmp/1103857186">"On the nature of turbulence"</a></span>. <i>Communications in Mathematical Physics</i>. <b>20</b> (3): <span class="nowrap">167–</span>192. <a href="/wiki/Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/1971CMaPh..20..167R">1971CMaPh..20..167R</a>. <a href="/wiki/Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2Fbf01646553">10.1007/bf01646553</a>. <a href="/wiki/S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a>&#160;<a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:17074317">17074317</a>.</cite><span title="ctx&#95;ver=Z39.88-2004&amp;rft&#95;val&#95;fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&amp;rft.genre=article&amp;rft.jtitle=Communications+in+Mathematical+Physics&amp;rft.atitle=On+the+nature+of+turbulence&amp;rft.volume=20&amp;rft.issue=3&amp;rft.pages=167-192&amp;rft.date=1971&amp;rft&#95;id=https%3A%2F%2Fapi.semanticscholar.org%2FCorpusID%3A17074317%23id-name%3DS2CID&amp;rft&#95;id=info%3Adoi%2F10.1007%2Fbf01646553&amp;rft&#95;id=info%3Abibcode%2F1971CMaPh..20..167R&amp;rft.aulast=Ruelle&amp;rft.aufirst=David&amp;rft.au=Takens%2C+Floris&amp;rft&#95;id=http%3A%2F%2Fprojecteuclid.org%2Feuclid.cmp%2F1103857186&amp;rfr&#95;id=info%3Asid%2Fen.wikipedia.org%3AAttractor" class="Z3988"></span></span>\n</li>\n<li id="cite&#95;note-Stochastic&#95;climate&#95;dynamics:&#95;Random&#95;attractors&#95;and&#95;time-dependent&#95;invariant&#95;measures-3"><span class="mw-cite-backlink"><b><a href="#cite_ref-Stochastic_climate_dynamics:_Random_attractors_and_time-dependent_invariant_measures_3-0">^</a></b></span> <span class="reference-text"><link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1238218222" /><cite id="CITEREFChekroun&#95;M.&#95;D.Simonnet&#95;E.Ghil&#95;M.2011" class="citation journal cs1">Chekroun M. D.; Simonnet E. &amp; <a href="/wiki/Michael_Ghil" title="Michael Ghil">Ghil M.</a> (2011). "Stochastic climate dynamics: Random attractors and time-dependent invariant measures". <i>Physica D</i>. <b>240</b> (21): <span class="nowrap">1685–</span>1700. <a href="/wiki/Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2011PhyD..240.1685C">2011PhyD..240.1685C</a>. <a href="/wiki/CiteSeerX_(identifier)" class="mw-redirect" title="CiteSeerX (identifier)">CiteSeerX</a>&#160;<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.156.5891">10.1.1.156.5891</a></span>. <a href="/wiki/Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.physd.2011.06.005">10.1016/j.physd.2011.06.005</a>.</cite><span title="ctx&#95;ver=Z39.88-2004&amp;rft&#95;val&#95;fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&amp;rft.genre=article&amp;rft.jtitle=Physica+D&amp;rft.atitle=Stochastic+climate+dynamics%3A+Random+attractors+and+time-dependent+invariant+measures&amp;rft.volume=240&amp;rft.issue=21&amp;rft.pages=1685-1700&amp;rft.date=2011&amp;rft&#95;id=https%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fsummary%3Fdoi%3D10.1.1.156.5891%23id-name%3DCiteSeerX&amp;rft&#95;id=info%3Adoi%2F10.1016%2Fj.physd.2011.06.005&amp;rft&#95;id=info%3Abibcode%2F2011PhyD..240.1685C&amp;rft.au=Chekroun+M.+D.&amp;rft.au=Simonnet+E.&amp;rft.au=Ghil+M.&amp;rfr&#95;id=info%3Asid%2Fen.wikipedia.org%3AAttractor" class="Z3988"></span></span>\n</li>\n</ol></div>\n<!-- \nNewPP limit report\nParsed by mw‐api‐ext.eqiad.main‐68bfd79cbb‐m29pv\nCached time: 20251202184134\nCache expiry: 19118\nReduced expiry: true\nComplications: [vary‐revision‐sha1, vary‐revision‐exists]\nCPU time usage: 0.121 seconds\nReal time usage: 0.152 seconds\nPreprocessor visited node count: 171/1000000\nRevision size: 3184/2097152 bytes\nPost‐expand include size: 8994/2097152 bytes\nTemplate argument size: 0/2097152 bytes\nHighest expansion depth: 4/100\nExpensive parser function count: 3/500\nUnstrip recursion depth: 0/20\nUnstrip post‐expand size: 7673/5000000 bytes\nLua time usage: 0.082/10.000 seconds\nLua memory usage: 3176810/52428800 bytes\nNumber of Wikibase entities loaded: 0/500\n-->\n<!--\nTransclusion expansion time report (%,ms,calls,template)\n100.00%  132.620      1 -total\n 67.22%   89.141      3 Template:Cite_journal\n 27.22%   36.099      1 Template:Redirect\n-->\n</div>'
        },
        "title": "Attractor",
    }
}

parser = WikiParser(section.replace("_", " "))
parser.feed(data["parse"]["text"]["*"])
text = parser.get_result()

print(f"{text = }")

[dgw]

Probably unrelated to this bug, but when poking at the API output for enumerating sections in the sample article (prop=sections) I saw a deprecation warning at the top of the JSON:

{
  "warnings": {
    "main": {
      "*": "Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes. Use [[Special:ApiFeatureUsage]] to see usage of deprecated features by your application."
    },
    "parse": {
      "*": "\"prop=sections\" is deprecated since MediaWiki 1.46. Use \"prop=tocdata\" instead."
    }
  },
  "parse": {
    ...
  }
}

So that's something to update at the same time as this fix happens: use prop=tocdata instead of prop=sections. The response format has changed a little, different names for the JSON keys and that sort of thing.

No deprecation warning in the JSON output for the next call fetching section=8, so that's good. Seems that fetching the section text itself is still done the way we are doing it.

[dgw]

I should have thought sooner to test on our regular ol' Sopel, which was still running the built-in version of wikipedia from Sopel 8.0.4. It can fetch the section snippet just fine:

<+dgw> .version wikipedia
<+Sopel> [version] wikipedia v1.0.0
<+dgw> https://en.wikipedia.org/wiki/Attractor#Strange_attractor
<+Sopel> [wikipedia] Error fetching section "Strange_attractor" for page "Attractor".
<+dgw> .version wikipedia
<+Sopel> [version] wikipedia v8.0.4 (built in)
<+dgw> https://en.wikipedia.org/wiki/Attractor#Strange_attractor
<+Sopel> [wikipedia] Attractor - Strange attractor | "A plot of Lorenz's strange attractor for values ρ = 28, σ = 10, β = 8/3 An attractor is called strange if it has a fractal structure, that is if it has non-integer Hausdorff dimension. This is often the case when the dynamics on it are chaotic, but strange nonchaotic attractors also exist. If a strange attractor is chaotic, exhibiting sensitive dependence on initial conditions, then any two arbitrarily […]"

This could be an issue with reorganizing the plugin into submodules?