Rework the broken links report (#11)

* Rework the broken links report

The client asked for a file holding, per item: the yodaat id, the title and
abstract in their original language, the authors, a link to the item and the
broken link itself. Three of those six columns were missing, and the link we
did report was often not the one they meant.

Extraction is now driven per item kind. Dedicated link fields (a publication's
url, an org's website/facebook/logo, a dataset's full_data_source) are taken
verbatim, and dataset series source_urls - 1031 links the report never covered
- are pulled out of the series JSON. Free text is still scanned, since for some
records the link in the abstract is the one that matters, but with a full
RFC 3986 character set: the old pattern stopped at '#' and was clipping 2060
URLs mid-path, then reporting the stump as broken. Each row now says where its
link came from, so 'קישור למקור' is distinguishable from a link in the
abstract. Wherever a field is translated, the untranslated Hebrew one is used.

Checking no longer trusts a bare HEAD. Half the rows in the current report are
codes servers hand to robots rather than dead links - 403s, 405s, Facebook's
400 - and a spot check of 60 of them found 12 that answer a GET perfectly well.
So a bot-wall code or a 5xx now triggers a real GET, results are sorted into
תקין / שבור / חסום לבדיקה אוטומטית / תקלה זמנית, and a connection error whose
host no longer resolves is called what it is: a dead domain.

Results are cached to disk for less than a day. Within a run that removes the
39% of fetches that are duplicate URLs; across runs it keeps a failure streak,
so a site that was down this morning reads differently from one that has been
gone a week.

Finally, the report is written as xlsx alongside the csv. Excel was rendering
the BOM-less UTF-8 as gibberish, which is what prompted the request; the
workbook is right-to-left, sorted with the certainly-dead links first, and has
clickable links, a frozen header and an autofilter.

Also dropped the global URL dedup, which was hiding 115 items whose broken link
happened to be shared with another record, and gave each unwound row its own
dict - DF.parallelize pickles rows from a background thread, so the shared
mutated one was a race waiting to happen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Add limit_rows, so a direct run is a usable smoke test

broken_links_flow() takes a limit_rows that goes straight to DF.load, and the
__main__ path passes 10 - enough to exercise every branch in a couple of
minutes rather than a couple of hours. A limited run gets its own checkpoint,
so a sample never ends up being served from, or serving, the cache of a full
run.

Running it that way turned up two things. On macOS the direct run hung: the
default start method there is spawn, DF.parallelize hands each worker a
closure, and a closure does not survive spawn, so the workers never came up and
the fetcher waited forever for results that were never coming. Fork is the
default wherever the pipeline actually runs, which is why this has never bitten
anyone; __main__ now asks for it explicitly.

The other was cosmetic: some servers answer without a reason phrase, and the
report was showing a bare '404:' with nothing after the colon, which reads like
a bug to whoever opens the file. Fall back to the standard phrase for the code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Publish the xlsx inside the datapackage directory

It was being written to data/broken_links.xlsx, one level above the
datapackage.json describing the same rows, so it was not picked up as part of
the published package. Both paths now derive from a single DUMP_PATH, which
leaves no way for them to drift apart again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: akariv