fix: add lxml dependency to environment2.yml and update argparse namespace in html_crawler.py

lukeblevins · lukeblevins · commit bd75bc919a83 · 2025-04-28T11:14:28.000-04:00
diff --git a/environment2.yml b/environment2.yml
@@ -16,6 +16,7 @@ dependencies:
   - huggingface_hub==0.23.5 
   - inflect==7.0.0
   - langdetect==1.0.9
+  - lxml==5.4.0
   - matplotlib==3.7.3
   - multiprocess==0.70.16
   - networkx==3.1
diff --git a/poligrapher/scripts/html_crawler.py b/poligrapher/scripts/html_crawler.py
@@ -93,7 +93,7 @@ def main(url, output):
         format="%(asctime)s [%(levelname)s] %(message)s", level=logging.INFO
     )
 
-    args = argparse.Namespace(url=url, output=output)
+    args = argparse.Namespace(url=url, output=output, no_readability_js=False)
     access_url = url_arg_handler(args.url)
 
     if access_url is None:

Original file line number	Diff line number	Diff line change
`@@ -93,7 +93,7 @@ def main(url, output):`
`93`	`93`	`format="%(asctime)s [%(levelname)s] %(message)s", level=logging.INFO`
`94`	`94`	`)`
`95`	`95`
`96`		`- args = argparse.Namespace(url=url, output=output)`
	`96`	`+ args = argparse.Namespace(url=url, output=output, no_readability_js=False)`
`97`	`97`	`access_url = url_arg_handler(args.url)`
`98`	`98`
`99`	`99`	`if access_url is None:`