news-fetch is an open-source, easy-to-use news crawler that extracts structured information from almost any news website 🌐. It can recursively follow internal hyperlinks and read RSS feeds to fetch both recent and archived articles 📚. You only need to provide the root URL of the news website to crawl it completely 🔍. News-fetch combines the power of multiple state-of-the-art libraries and tools.
I built this tool to minimize NaN or empty values when scraping data from various news websites 🚀. It's platform-independent and written in Python 3, making it easy for programmers and developers to access news data for their applications 💻.
news-fetch extracts the following attributes from news articles. You can also check out an example JSON file generated by news-please.
- 📰 Headline
- ✍️ Author(s)
- 📅 Publication date
- 🗞️ Publication
- 📂 Category
- 🌍 Source domain
- 📑 Article content
- 📝 Summary
- 🔑 Keywords
- 🌐 URL
- 🌐 Language
Use the package manager pip to install the required dependencies:
pip install -r requirements.txtYou can download it by clicking the green download button.
To scrape all the news details, use the newspaper function:
Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change.
Make sure to update tests as appropriate.