Codex/fix arxiv dedup

[laoyunzai]

No description provided.

[Copilot]

Pull request overview

This PR fixes arXiv deduplication/cross-list handling end-to-end (spider → record utilities → stats check → markdown conversion), while also hardening the GitHub Pages UI against XSS and making forks/local runs less reliant on owner-specific generated config.

Changes:

• Canonicalize arXiv IDs (strip versions), deduplicate across category pages, and preserve the “requested category” for cross-listed papers.
• Add safe HTML/URL rendering helpers and apply escaping/safe-linking across the UI.
• Update automation/local scripts (weekday schedule, concurrency, atomic writes, data-branch preparation) and adjust local-only auth override flow.

Reviewed changes

Copilot reviewed 27 out of 28 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
to_md/convert.py	Uses matched_category* for categorization and improves JSONL/template handling.
template.md	Updates documentation about weekday scheduling.
statistic.html	Loads local auth override and safe rendering helpers.
setup-local-auth.sh	Writes a local-only auth override file instead of modifying tracked config.
settings.html	Loads local auth override and ensures DATA_CONFIG is available to settings JS.
run.sh	Hardens local workflow (strict mode, module execution, file list generation).
README.md	Clarifies client-side auth limitations and weekday scheduling.
login.html	Loads local auth override for local testing.
js/statistic.js	Uses repo-derived GitHub API URL, broader date regex, and escapes rendered content/URLs.
js/settings.js	Avoids innerHTML injection for tags and uses repo-derived GitHub API URL.
js/safe-render.test.js	Adds Node tests for escaping/highlighting and safe URL handling.
js/safe-render.js	Introduces escapeHtml, safe highlighter, and safe external URL helpers.
js/data-config.js	Derives repo owner/name from the GitHub Pages URL and URL-encodes data paths.
js/auth-config.js	Reframes auth as a client-side UI gate and defaults to disabled.
js/app.js	Uses repo-derived GitHub API URL, safer regex/highlighting, escapes UI strings, and sanitizes external links.
index.html	Loads local auth override and safe rendering helpers for the main app.
daily_arxiv/tests/test_spider.py	Tests cross-list category retention and cross-category dedup in the spider.
daily_arxiv/tests/test_record_utils.py	Tests canonical ID dedup/version merging and history filtering.
daily_arxiv/tests/test_converter.py	Tests main_category() behavior for cross-lists and empty categories.
daily_arxiv/tests/test_check_stats.py	Tests multi-day dedup outcomes and file normalization/deletion behavior.
daily_arxiv/daily_arxiv/spiders/arxiv.py	Normalizes IDs, dedups across pages, and records requested category metadata.
daily_arxiv/daily_arxiv/record_utils.py	Adds canonicalization + dedup/filter helpers shared across the pipeline.
daily_arxiv/daily_arxiv/check_stats.py	Refactors dedup into testable functions with atomic writes and CLI args.
ai/test_compliance.py	Tests fail-closed compliance adapter behavior.
ai/enhance.py	Adds atomic output writes, dedup before enhancement, compliance adapter, and GitHub metadata caching.
ai/compliance.py	Adds a small adapter that fails closed on compliance-service errors.
.gitignore	Ignores local env/auth overrides and temp files.
.github/workflows/run.yml	Weekday schedule, concurrency/permissions, restores data history, and pushes only AI-enhanced outputs to data.

Suppressed comments (1)

js/statistic.js:470

• After URL-encoding data-keyword, the click handler should decode it before passing to showRelatedPapers, otherwise the keyword parameter will be percent-encoded.

    container.querySelectorAll('.keyword-item').forEach(item => {
      item.addEventListener('click', () => showRelatedPapers(item.dataset.keyword));
    });

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.