Web scraper - Refine results into favourable format

[navan0]

• web scrapper modify to scrape favorable results

Method

• Get the information from possible #tags from various channels like StackOverflow, Askubuntu, and other community channels.
• Favourable output is CSV
• Put question title as feature and command in answer as label.
• Set a third column verified as true if there is a verified answer. Otherwise false.
• Update timestamp

Schema

• Tags like 50 most frequently used bash commands.
• Remove questions with other programming languages as subtasks.

Tasks

• Scrap unstructured data from askubuntu and stackoverflow (10000 datapoints)

[navan0]

http://cheat.sh/ seems a good resource for the dataset collection

[blacksmithop]

Approached the issue using urllib and re
Criteria for getting question - Most Voted
You can find the implmentation here
Step 1: Iterate through a list of tags and pass it to Tag_To_Url
Output: List of urls to questions (can cover n number of pages)
Step 2: Pass the list from Tag_To_Url to Link_To_JSON
Output: The title, a boolean flag (is_verified) , list of code snippets (belonging to answers only)

[navan0]

@blacksmithop I think it's better to make a PR if you tested, then we can test and review it. it's a better approach when we are contributing.

@GopikrishnanSasikumar please have a look