Fix 2021 publications sync

Author: benquist

.github/workflows/sync-google-doc-cv.yml

@@ -22,15 +22,17 @@ jobs:
           python-version: "3.12"
 
       - name: Sync CV from Google Docs export
-        env:
-          GOOGLE_DOC_CV_ID: ${{ secrets.GOOGLE_DOC_CV_ID }}
         run: |
           bash scripts/sync_google_doc_cv.sh
 
       - name: Sync new publications from CV DOIs
         run: |
           python scripts/sync_publications_from_cv.py
 
+      - name: Rebuild publications include from public Google Doc HTML
+        run: |
+          python scripts/rebuild_publications_include_from_doc.py
+
       - name: Sync publications HTML from CV
         run: |
           python scripts/sync_publications_html.py
@@ -52,6 +54,7 @@ jobs:
                   assets/cv/publications_sync_report.txt \
                   _data/cv_sync.yml \
                   _bibliography/papers.bib \
+                  scripts/rebuild_publications_include_from_doc.py \
                   _includes/publications_full_from_doc.md
           git commit -m "chore(cv): sync CV and publications from Google Doc"
           git push

_includes/publications_full_from_doc.md

@@ -581,6 +581,10 @@ <h2>Complete Publication List</h2>
 				label: 'Conservation Impacts',
 				threshold: 2,
 				matchers: [
+					{ pattern: /deregulation/i, weight: 2 },
+					{ pattern: /increasing fire|landscape fire|wildfire/i, weight: 2 },
+					{ pattern: /amazonian biodiversity/i, weight: 3 },
+					{ pattern: /drought.{0,30}biodiversity|biodiversity.{0,30}drought/i, weight: 2 },
 					{ pattern: /conservation planning/i, weight: 3 },
 					{ pattern: /conserving terrestrial biodiversity/i, weight: 3 },
 					{ pattern: /biodiversity, carbon and water/i, weight: 3 },

_pages/publications.html

@@ -1,57 +1,12 @@
-Publications Sync Report — 2026-06-06 06:54 UTC
+Publications Sync Report — 2026-06-06 07:56 UTC
 ============================================================
 
 CV entries parsed: 332
 In Press → published patches applied: 0
+Missing-paper insertions applied: 0
 DOIs verified OK: 23
 DOI failures / anomalies: 3
-Papers in CV but NOT found in HTML: 43
-
-MISSING FROM WEBSITE HTML (in CV but not found in _includes/publications_full_from_doc.md)
-----------------------------------------
-  [2025 #11] Gaudard, J., Telford, R. J., Chacon‐Labella, J., Dawson, H. R., Enquist, B. J., Töpper, J. P., ... & Halbritter, A. H. (2025). fluxible:An R …
-  [2024 #27] Westgeest, Adrianus J., François Vasseur, Brian J. Enquist, Rubén Milla, Alicia Gómez‐Fernández, David Pot, Denis Vile, and Cyrille Violle. …
-  [2023 #50] Jónsdóttir, Ingibjörg S., et al. "Intraspecific trait variability is a key feature underlying high Arctic plant community resistance to clim …
-  [2022 #61] Chaplin-Kramer, R., Brauman, K. A., Cavender-Bares, J., Díaz, S., Duarte, G. T., Enquist, B. J., ... & Zafra-Calvo, N. (2022). Conservation …
-  [2022 #62] Feng, X., Enquist, B. J., Park, D. S., Boyle, B., Breshears, D. D., Gallagher, R. V., ... & López‐Hoffman, L. (2022)  A review of the hetero …
-  [2022 #63] Boyle, B. L., Maitner, B. S., Barbosa, G. G., Sajja, R. K., Feng, X., Merow, C., ... & Enquist, B. J. (2022). Geographic name resolution ser …
-  [2022 #64] Maitner, B. S., Park, D. S., Enquist, B. J., & Dlugosch, K. M. (2022). Where we've been and where we're going: the importance of source comm …
-  [2022 #65] Aguirre-Gutiérrez, Jesús, et al. Functional susceptibility of tropical forests to climate change. Nature Ecology & Evolution (2022): 1-12. …
-  [2022 #66] Wieczynski, D.J., Díaz, S., Durán, S.M., Fyllas, N.M., Salinas, N., Martin, R.E., Shenkin, A., Silman, M.R., Asner, G.P., Bentley, L.P., Mal …
-  [2022 #67] Lourenço Junior J, Enquist BJ, von Arx G, Sonsin Oliveira J, Morino K, Thomaz LD, Milanez CR. (2022)  Hydraulic tradeoffs underlie local var …
-  [2022 #68] Gatti, R.G. et al. (2022) The number of tree species on Earth. Proceedings of the National Academy of Sciences, U.S.A  DOI: 10.1073/pnas.211 …
-  [2022 #69] Maitner BS, Park DS, Enquist BJ, Dlugosch KM. (2022) Both source and recipient range phylogenetic community structure can predict the outcom …
-  [2022 #70] Brun, P., Violle, C., Mouillot, D., Mouquet, N., Enquist, B.J., Munoz, F., Münkemüller, T., Ostling, A., Zimmermann, N.E. and W. Thuiller  ( …
-  [2021 #71] Enquist, B. J. (2021)  How Pandemics Rapidly Reshape the Evolutionary & Ecological Landscape, In: The Complex Alternative: Complexity Scient …
-  [2021 #72] Feng X, Merow C, Liu Z, Park DS, Roehrdanz PR, Maitner B, Newman EA, Boyle BL, Lien A, Burger JR, Pires MM. P. M. Brando, M.B. Bush, C.N.H. …
-  [2021 #74] Huang, C.Y., Durán, S.M., Hu, K.T., Li, H.J., Swenson, N.G. and B.J. Enquist (2021) Remotely sensed assessment of increasing chronic and epi …
-  [2021 #75] Chaplin-Kramer, R., Brauman, K.A., Cavender-Bares, J., Díaz, S., Duarte, G.T., Enquist, B.J., Garibaldi, L.A., Geldmann, J., Halpern, B.S., …
-  [2020 #93] Vandvik, V., Halbritter, A.H., Yang, Y., He, H., Zhang, L., Brummer, A.B., Klanderud, K., Maitner, B.S., Michaletz, S.T., Sun, X. and Telfor …
-  [2019 #104] Durán, S.M., Martin,R.E., Díaz, S., Maitner, B.S., Malhi, Y., Salinas, N., Shenkin, A., Silman, M.L., Wieczynski, D.J., Asner, G.P., Bentl …
-  [2019 #112] Weiser, M.D., Ning. D., Buzzard, V., Michaletz, S.T., He., Z., Enquist, B.J., Waide, R.B., Zhou, J., and M. Kaspari. Thermal disruption of s …
-  [2019 #118] Šímová, I., Sandel, B., Enquist, B.J., Michaletz, S.T., Kattge, J., Violle, C., McGill, B.J., Blonder, B., Engemann, K., Peet, R.K. and Wise …
-  [2018 #120] Bjorkman, A.D. et al. (2018) (50+ international authors)  Plant functional trait change across a warming tundra biome Nature, 562:57–62. …
-  [2017 #146] Blonder, B., Moulton, D.E., Blois, J., Enquist, B.J., Graae, B.J., Macias-Fauria, M., McGill, B.J., Nogué, S., Ordonez, A., Sandel, B., and …
-  [2016 #164] Feakins, S.J, T. Peters, M.S. Wu, A. Shenkin, N. Salinas, C.A.J. Girardin, L. P. Bentley, B. Blonder, B. J. Enquist, R.E. Martin, G. P. Asne …
-  [2016 #173] Enquist, B.J. (2016) Biology Distilled. Nature, 531:34. …
-  [2015 #179] Duncanson, L.I., Dubayah, R.O, and  B. J. Enquist  Assessing the general patterns of forest structure: Quantifying tree and forest allometri …
-  [2015 #180] Blonder, B., Vasseur, F., Violle, C., Shipley, B., Enquist, B.J. and D. Vile. Testing models for the leaf economics spectrum with leaf and w …
-  [2015 #183] Grady, J.M., Enquist,B.J., Dettweiler-Robinson, E., Wright, N. A.,  and F. A. Smith (2015)  TECHNICAL COMMENTS: Response to Comments on “Evi …
-  [2014 #191] Grady, J.M., Enquist, B.J., Dettweiler-Robinson, E., Wright, N.A. and F.A. Smith. (2014) Evidence for mesothermy in dinosaurs.Science, 344:1 …
-  [2014 #192] Lamanna, C. A., Blonder, B., Violle, C. Kraft, N. J. B., Sandel, B. Simova, I. Donoghue, J., Svenning, J.C., McGill, B.J. Boyle, B. Dolins, …
-  [2014 #194] Marquet,P.A., A.P. Allen, , J. H. Brown, J. Dunne, B.J. Enquist, J. Gillooly, P.A. Gowaty, J. L. Green, D. Storch, J. Harte, S. P. Hubbell, …
-  [2014 #198] Blonder, B., Lamanna, C., Violle, C, and B.J. Enquist (2014) The n-dimensional hypervolume. Global Ecology and Biogeography, 23:595-609. …
-  [2012 #213] Enquist, B. J., Boyle, B. (2012): SALVIAS – the SALVIAS vegetation inventory database. – In: Dengler, J., Oldeland, J., Jansen, F., Chytrý, …
-  [2011 #225] Enquist, B.J. (2011) Forest annual carbon cost: comment. Ecology, 92:1994-1998. …
-  [2011 #226] Kattge, J., Díaz, S., Lavorel S. et al. (2011) TRY – a global database of plant traits. Global Change Biology, 17:2905-2935. …
-  [2011 #232] Stark, S.C., Bentley, L. P., and B.J. Enquist (2011) Response to Coomes & Allen (2009) ‘Testing the metabolic scaling theory of tree growth’ …
-  [2009 #237] Price, C. A. and J. Enquist (2009) Comment on Coomes et al. “Scaling of xylem vessels and veins within the leaves of oak species”. Biology L …
-  [2009 #240] Poore, B., Lamanna, C., Ebersole, J.J., and J. Enquist. (2009) Controls on radial growth of Mountain Big Sagebrush and implications for clim …
-  [2007 #253] Enquist, B.J. (2007). Journal Club – An ecologist wonders how biotic feedback matters to global-change research. Nature 450:139. …
-  [2007 #255] Enquist, B.J. and S.C. Stark (2007) Correspondence – Follow Thompson to make biology a capital-S Science. Nature 446:611. …
-  [2003 #293] West, G.B., V. M. Savage, J. Gillooly, J. Enquist, William H. Woodruff and James H. Brown. (2003). Brief Communication – But Why DoesMetabol …
-  [2002 #307] West, G.B., Savage, V.M., Gillooly, J., Enquist, B.J. Woodruff, W.H. and J.H. Brown (2002). Red herrings and rotten fish. arXiv:physics/0211 …
-  [1999 #321] Enquist, B.J., Brown, J.H. & West, G.B. (1999) Plant energetics and population density Response. Nature 398:572-573. …
+Papers in public Google Doc but NOT found in HTML: 0
 
 DOI FAILURES / ANOMALIES
 ----------------------------------------

assets/cv/publications_sync_report.txt

@@ -327,3 +327,13 @@ Outcome: Completed coordinated tri-agent review-only audit (no code edits). Cros
 Prompt: Update research tab to say closed-loop and open-path chamber-based approaches; add co2fluxtent (https://github.com/PaulESantos/co2fluxtent) alongside fluxible in research.md flux tooling paragraph and as a new section in software.md.
 Source session: VS Code Copilot Chat
 Outcome: (1) research.md flux measurement sentence updated to cite both closed-loop and open-path approaches; (2) co2fluxtent described as a companion open-path package after the fluxible paragraph; (3) software.md new co2fluxtent section added below fluxible with GitHub link and install snippet.
+
+43. Date: 2026-06-06
+Prompt: For https://enquistlab.github.io/publications/ add the missing 2021 Nature paper "How deregulation, drought and increasing fire impact Amazonian biodiversity" and ensure it groups under Tropical Ecology and Conservation Impacts.
+Source session: VS Code Copilot Chat
+Outcome: Added a targeted repair path in `scripts/sync_publications_html.py` for the missing Feng et al. (2021) Nature citation, reran the sync so `_includes/publications_full_from_doc.md` now contains the paper, and expanded the `conservation-impacts` topic matchers in `_pages/publications.html` with Amazonian biodiversity / deregulation / fire signals. The paper now exists in the master publication list and is classified for both Tropical Ecology and Conservation Impacts. Updated `assets/cv/publications_sync_report.txt` during validation.
+
+44. Date: 2026-06-06
+Prompt: All 2021 papers appear to be missing from the publications page; keep the website publications synced to the public Google Doc.
+Source session: VS Code Copilot Chat
+Outcome: Added `scripts/rebuild_publications_include_from_doc.py` to rebuild `_includes/publications_full_from_doc.md` from the public Google Doc HTML export while regrouping entries by inferred citation year, which restored an explicit 2021 year block and the missing 2021 papers. Updated `.github/workflows/sync-google-doc-cv.yml` to run the rebuild script before the sync checker. Adjusted `scripts/sync_publications_html.py` so its missing-paper report now compares the site include against the public Google Doc HTML source of truth. Final validation report: `Papers in public Google Doc but NOT found in HTML: 0`.

chat_provenance_log.md

@@ -0,0 +1,233 @@
+#!/usr/bin/env python3
+"""
+Rebuild the publications include from the current cleaned include plus the
+public Google Doc HTML export.
+
+This script fixes two recurring drift problems:
+1. Items can sit under the wrong year block if the Google Doc year heading
+   formatting changes.
+2. New items can appear in the public Google Doc but not in the website include.
+
+The strategy is conservative:
+- Start from the existing cleaned website include.
+- Reassign each existing <li> to the year inferred from the citation text.
+- Parse the public Google Doc HTML export, clean the anchor URLs, and add any
+  missing items by year.
+- Render a normalized include with one explicit year block per inferred year.
+"""
+
+import re
+import urllib.parse
+import urllib.request
+from collections import OrderedDict
+from html import escape, unescape
+from html.parser import HTMLParser
+from pathlib import Path
+
+
+DOC_ID = "1OMolpDfY6c73qRgUFgYh0XbMS8EykUoy_hvrI8fnTn0"
+DOC_HTML_URL = f"https://docs.google.com/document/d/{DOC_ID}/export?format=html"
+PUBS_HTML = Path("_includes/publications_full_from_doc.md")
+SECTION_LABEL = "Peer Reviewed Journal Aricles"
+SECTION_END_LABELS = (
+    "Other Publications",
+    "Book Chapters",
+    "Books",
+)
+
+SECTION_RE = re.compile(r"<p>\s*([^<]+?)\s*</p><ol>(.*?)</ol>", re.IGNORECASE | re.DOTALL)
+LI_RE = re.compile(r"<li>.*?</li>", re.IGNORECASE | re.DOTALL)
+YEAR_RE = re.compile(r"\((19|20)\d{2}\)|\b(19|20)\d{2}\b")
+
+
+def clean_google_href(href):
+    href = unescape(href or "").strip()
+    parsed = urllib.parse.urlparse(href)
+    if parsed.netloc.endswith("google.com") and parsed.path == "/url":
+        q = urllib.parse.parse_qs(parsed.query).get("q")
+        if q:
+            return q[0]
+    return href
+
+
+def strip_tags(html_text):
+    text = re.sub(r"<[^>]+>", "", html_text)
+    text = text.replace("\xa0", " ")
+    return re.sub(r"\s+", " ", unescape(text)).strip()
+
+
+def infer_year(html_text):
+    text = strip_tags(html_text)
+    match = YEAR_RE.search(text)
+    if not match:
+        return None
+    year_text = match.group(0).strip("()")
+    return int(year_text)
+
+
+def title_key(html_text):
+    text = strip_tags(html_text)
+    match = re.search(r"\(\d{4}\)[.\s]+(.+)", text)
+    if match:
+        text = match.group(1)
+    text = re.sub(r"[^a-z0-9]+", " ", text.lower())
+    return re.sub(r"\s+", " ", text).strip()[:160]
+
+
+def normalize_li(html_text):
+    html_text = html_text.replace("\xa0", " ")
+    html_text = re.sub(r"\s+", " ", html_text)
+    html_text = re.sub(r">\s+", ">", html_text)
+    html_text = re.sub(r"\s+<", "<", html_text)
+    html_text = re.sub(r"\s+([,.;:)])", r"\1", html_text)
+    html_text = re.sub(r"([(" + "\[] )\s+", r"\1", html_text)
+    return html_text.strip()
+
+
+class DocPublicationParser(HTMLParser):
+    def __init__(self):
+        super().__init__()
+        self.items = []
+        self.in_li = False
+        self.parts = []
+
+    def handle_starttag(self, tag, attrs):
+        if tag == "li":
+            self.in_li = True
+            self.parts = []
+            return
+
+        if self.in_li and tag == "a":
+            attr_map = dict(attrs)
+            href = clean_google_href(attr_map.get("href", ""))
+            self.parts.append(
+                '<a href="{}" target="_blank" rel="noopener noreferrer">'.format(
+                    escape(href, quote=True)
+                )
+            )
+
+    def handle_endtag(self, tag):
+        if self.in_li and tag == "a":
+            self.parts.append("</a>")
+            return
+
+        if tag == "li" and self.in_li:
+            item = normalize_li("<li>" + "".join(self.parts) + "</li>")
+            if strip_tags(item):
+                self.items.append(item)
+            self.in_li = False
+            self.parts = []
+
+    def handle_data(self, data):
+        if self.in_li:
+            self.parts.append(escape(data, quote=False))
+
+    def handle_entityref(self, name):
+        if self.in_li:
+            self.parts.append(escape(unescape(f"&{name};"), quote=False))
+
+    def handle_charref(self, name):
+        if self.in_li:
+            self.parts.append(escape(unescape(f"&#{name};"), quote=False))
+
+
+def load_doc_html():
+    req = urllib.request.Request(DOC_HTML_URL, headers={"User-Agent": "publications-include-sync/1.0"})
+    with urllib.request.urlopen(req, timeout=60) as resp:
+        return resp.read().decode("utf-8", errors="replace")
+
+
+def extract_doc_section(doc_html):
+    start = doc_html.find(SECTION_LABEL)
+    if start == -1:
+        raise RuntimeError(f"Could not find section label: {SECTION_LABEL}")
+
+    end_candidates = [doc_html.find(label, start + len(SECTION_LABEL)) for label in SECTION_END_LABELS]
+    end_candidates = [idx for idx in end_candidates if idx != -1]
+    end = min(end_candidates) if end_candidates else len(doc_html)
+    return doc_html[start:end]
+
+
+def parse_doc_items_by_year(doc_html):
+    parser = DocPublicationParser()
+    parser.feed(extract_doc_section(doc_html))
+
+    items_by_year = OrderedDict()
+    keys_by_year = {}
+    for item in parser.items:
+        year = infer_year(item)
+        if year is None:
+            continue
+        key = title_key(item)
+        if year not in items_by_year:
+            items_by_year[year] = []
+            keys_by_year[year] = set()
+        if key and key not in keys_by_year[year]:
+            items_by_year[year].append(item)
+            keys_by_year[year].add(key)
+    return items_by_year
+
+
+def parse_existing_include(include_text):
+    items_by_year = OrderedDict()
+    keys_by_year = {}
+
+    for match in SECTION_RE.finditer(include_text):
+        section_year = match.group(1).strip()
+        body = match.group(2)
+        for li_match in LI_RE.finditer(body):
+            item = normalize_li(li_match.group(0))
+            inferred = infer_year(item)
+            if inferred is None:
+                try:
+                    inferred = int(section_year)
+                except ValueError:
+                    continue
+            key = title_key(item)
+            if inferred not in items_by_year:
+                items_by_year[inferred] = []
+                keys_by_year[inferred] = set()
+            if key and key not in keys_by_year[inferred]:
+                items_by_year[inferred].append(item)
+                keys_by_year[inferred].add(key)
+    return items_by_year, keys_by_year
+
+
+def render_include(items_by_year):
+    ordered_years = sorted(items_by_year.keys(), reverse=True)
+    lines = [
+        "<!-- Auto-generated from shared Google Doc: peer-reviewed publication section -->",
+        f"<p>{SECTION_LABEL}</p>",
+    ]
+    for year in ordered_years:
+        items = "".join(items_by_year[year])
+        lines.append(f"<p>{year}</p><ol>{items}</ol>")
+    return "\n".join(lines) + "\n"
+
+
+def main():
+    include_text = PUBS_HTML.read_text(encoding="utf-8") if PUBS_HTML.exists() else ""
+    existing_by_year, keys_by_year = parse_existing_include(include_text)
+
+    doc_html = load_doc_html()
+    doc_by_year = parse_doc_items_by_year(doc_html)
+
+    additions = 0
+    for year, items in doc_by_year.items():
+        if year not in existing_by_year:
+            existing_by_year[year] = []
+            keys_by_year[year] = set()
+        for item in items:
+            key = title_key(item)
+            if key and key not in keys_by_year[year]:
+                existing_by_year[year].append(item)
+                keys_by_year[year].add(key)
+                additions += 1
+
+    rendered = render_include(existing_by_year)
+    PUBS_HTML.write_text(rendered, encoding="utf-8")
+    print(f"[include] Wrote {PUBS_HTML} with {len(existing_by_year)} year blocks and {additions} added item(s).")
+
+
+if __name__ == "__main__":
+    main()