-
Notifications
You must be signed in to change notification settings - Fork 107
LBC publisher integrated #814
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| from fundus.publishers.base_objects import Publisher, PublisherGroup | ||
| from fundus.publishers.lb.lbc_group import LBCGroupParser | ||
| from fundus.scraping.url import NewsMap, RSSFeed, Sitemap | ||
|
|
||
| class LB(metaclass=PublisherGroup): | ||
| default_language= "ar" | ||
|
|
||
| LBCGroup=Publisher( | ||
| name = "LBC", | ||
| domain = "https://www.lbcgroup.tv", | ||
| parser = LBCGroupParser, | ||
| sources=[ | ||
| RSSFeed("https://www.lbcgroup.tv/Rss/latest-news/en"), | ||
| NewsMap("https://www.lbcgroup.tv/newssitemap.xml"), | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This this source is multi-lingual, please override the language with |
||
| Sitemap("https://www.lbcgroup.tv/sitemap.xml"), | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this sitemap can safely be removed, at least I didn't find any articles that weren't also part of the Newsmap. And more problematically, there were numerous pages that were not relevant to Fundus. |
||
| ], | ||
| ) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| import datetime | ||
| import re | ||
| from typing import List, Optional | ||
|
|
||
| from lxml.etree import XPath | ||
|
|
||
| from fundus.parser import ArticleBody, BaseParser, Image, ParserProxy, attribute | ||
| from fundus.parser.utility import ( | ||
| extract_article_body_with_selector, | ||
| generic_author_parsing, | ||
| generic_date_parsing, | ||
| generic_topic_parsing, | ||
| image_extraction, | ||
| ) | ||
|
|
||
|
|
||
| class LBCGroupParser(ParserProxy): | ||
| class V1(BaseParser): | ||
| content_container_selector = XPath("//div[@class='LongDesc']/div[1]/div[1]") | ||
|
|
||
| # We tell the parser utility that the content container itself ('.') | ||
| # should be treated as the main text block, allowing extraction of text nodes. | ||
| _paragraph_selector = XPath(".") | ||
|
|
||
| # There are no subheadlines (like <h2>) in your snippet. | ||
| _subheadline_selector = None | ||
|
|
||
| @attribute | ||
| def body(self) -> Optional[ArticleBody]: | ||
| # Use the defined content_selector to locate the block of text. | ||
| return extract_article_body_with_selector( | ||
| self.precomputed.doc, | ||
| content_selector=self._content_container_selector, | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This line does not really make sense and causes the program to crash. You have defined the selector as |
||
| paragraph_selector=self._paragraph_selector, | ||
| subheadline_selector=self._subheadline_selector, | ||
| # Optionally, remove elements like the banner injection and the 'Reuters' credit | ||
| # if you want a cleaner body, but we'll focus on text for now. | ||
| ) | ||
|
|
||
| @attribute | ||
| def title(self) -> Optional[str]: | ||
| return self.precomputed.meta.get("og:title") | ||
|
|
||
| @attribute | ||
| def authors(self) -> List[str]: | ||
| return generic_author_parsing(self.precomputed.ld.bf_search("author")) | ||
|
|
||
| @attribute | ||
| def publishing_date(self) -> Optional[datetime.datetime]: | ||
| return generic_date_parsing(self.precomputed.ld.bf_search("datePublished")) | ||
|
|
||
| @attribute | ||
| def images(self) -> List[Image]: | ||
| return image_extraction( | ||
| doc=self.precomputed.doc, | ||
| paragraph_selector=self._paragraph_selector, | ||
| author_selector=XPath("./ancestor::figure//footer"), | ||
| size_pattern=re.compile(r"/rs:fill:(?P<width>[0-9]+):"), | ||
| ) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the default language is arabic, please add the following parameter to to each English source
languages={"en"}. Also, since there is an arabic RSSFeed, please add that as well:RSSFeed("https://www.lbcgroup.tv/Rss/latest-news/ar")There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can add all RSSFeeds from https://www.lbcgroup.tv/rss/ar and https://www.lbcgroup.tv/rss/en