Add Wikipedia as a new data source

[oree-xx]

Description

Currently, Quantifying uses Google Custom Search and GitHub APIs. I would like to add Wikipedia as a new source for fetching data. So we would have wikipedia_fetch.py under scripts directory.

Implementation

• Integrate the Wikipedia API as a source for it to retrieve the count of Wikipedia articles containing references to specific Creative Commons licenses or keywords.
• Implement fetching functions similar to existing github_fetch.py so the data flow remains consistent.
• Update documentation to reflect Wikipedia as a supported data source.

API documentation: https://www.mediawiki.org/wiki/API:Siteinfo#Rightsinfo

Available statistics:

• Number of articles
• Number of pages
• Number of edits
• Number of users
• Number of images

I would update the implementation as I continue working on it.

[levy-web]

Hi @oree-xx, I'd be happy to take it on this task.

[oree-xx]

ohh hi @levy-web I opened the issue for me to work on it. But if you have any suggestion to help me on it, I would love it.

[oree-xx]

Hi @TimidRobot @possumbilities I saw that I would have to create a query plan for wikipedia. Also the GCS has a python script to generate it own query plan. Can I use gcs_query_plan.csv directly or can I just copy the query plan and paste it in wikipedia_query_plan? Or do I have to write a python script also for wikipedia.

Then I used functions from both GCS and Github fetch script for wikipedia. Looking at how wikipedia does not use credentials (tokens, keys) just base url and I have to use a query plan unlike github that uses github tools. So is it fine I adapt functions from both of them that I think fits the flow for wikipedia or I use functions from just one script.

[TimidRobot]

@oree-xx I expect you won't need a query plan for Wikipedia (it is required for GCS due to the quota). Please do choose the best parts from both scripts as you think appropriate.

[oree-xx]

@TimidRobot okay, I am trying to understand how the query works. In github_fetch.py, there are github tools with different licenses. I am guessing the tool is used to tell github what to query and GCS query plan also is used to tell GCS what to query. But because of the quota in GCS we have a query plan. So do I just list the licenses I want to search in wikipedia just like the github tools.