A console application for scraping news stories from a provided Url and exporting the results as Json.
- The application requires .NET Core 3.1, which can be downloaded here.
- Compile the solution.
- Open a command line window such as
cmdorpowershell, and navigate to the build folder. - Run
NewsScraper.exewith the following parameters:
-p n --posts n Sets the number of posts to scrape. Should be an integer between 1 and 100. Optional, defaults to 10.
-u url --url url Sets the url to scrape the news items from. Optional, defaults to https://news.ycombinator.com/
Simplifies the process of parsing command line arguments.
Provides an API for downloading and parsing Html documents into a queryable node model. Used by the scraper to locate and extract the data.
Used to serialize the final output as Json.
Contains simple, powerful helper functions for validating and maniuplating Urls.