Initial commit

Author: felixrech

.gitignore

@@ -0,0 +1,2 @@
+dist/
+src/hys_scraper.egg-info/

LICENSE

@@ -0,0 +1,68 @@
+# 'Have your Say' scraper
+
+![Python version](https://img.shields.io/badge/python-%3E%3D3.6-blue?logo=python) [![PyPI version](https://badge.fury.io/py/hys_scraper.svg)](https://badge.fury.io/py/hys_scraper) [![GPLv3 license](https://img.shields.io/github/license/felixrech/hys_scraper)](https://github.com/felixrech/hys_scraper/blob/master/LICENSE)
+
+A small utility to scrape the European Commission's 'Have your Say' plattform ([https://ec.europa.eu/info/law/better-regulation/have-your-say](https://ec.europa.eu/info/law/better-regulation/have-your-say)). Can scrape an initiative's feedback submissions, attachments of these submissions, and the by country and by category statistics.
+
+## Installation
+
+```bash
+pip3 install hys_scraper
+```
+
+Tested to work with Python 3.9 on a Linux machine and Google Colab notebooks.
+
+## Getting started
+
+To get started, you will need the publication id of the initiative you want to scrape. To get this, simply navigate to the initiative on 'Have your Say' and look at the URL - the number at the end is the publication id you will use in the next step. For example, for the [AIAct commission adoption initiative](https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12527-Artificial-intelligence-ethical-and-legal-requirements/feedback_en?p_id=24212003), the publication id would be `24212003`.
+
+To scrape an initiative the following is sufficient (replace `24212003` with the publication id of the initiative you want to scrape):
+
+```bash
+python3 -m hys_scraper 24212003
+```
+
+This will create a new folder in your current working directory with the following layout:
+
+```
+24212003_requirements_for_artificial_intelligence/
+├── attachments
+│   ├── 2488672.pdf
+│   ├── 2596917.pdf
+│   └── ...
+├── attachments.csv
+├── categories.csv
+├── countries.csv
+└── feedbacks.csv
+
+1 directory, 263 files
+```
+
+## Advanced usage
+
+The command line interface has a few more arguments. For example instead of having `hys_scraper` create a folder in the local working directory to save results into, you can also manually specify a target directory.
+
+```
+$ python3 -m hys_scraper -h
+Scrape feedback and statistics from the European Commission's 'Have your Say' plattform.
+
+positional arguments:
+  PID                   The publication id - what comes after 'p_id=' in the initiative's URL.
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --dir target_dir, --target_dir target_dir
+                        Directory to save the feedback and statistics dataframes to. Defaults to creating a new
+                        folder in the current working directory.
+  --no_attachments      Whether to skip the download of attachments.
+  --sleep_time t        Minimum time between consecutive HTTP requests (in seconds).
+```
+
+Alternatively, you can also access `hys_scraper` from Python:
+
+```python
+from hys_scraper import HYS_Scraper
+feedbacks, countries, categories = HYS_Scraper("24212003").scrape()
+```
+
+Similar options are available as for the command line interface, check out `help(HYS_Scraper)` for details.

README.md

@@ -0,0 +1,23 @@
+[build-system]
+requires = ["setuptools>=61.0.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "hys_scraper"
+version = "0.1.3"
+description = "Scrape feedback and statistics from the European Commission's 'Have your Say' plattform."
+readme = "README.md"
+classifiers = [
+    "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 3",
+]
+dependencies = [
+    "requests",
+    "pandas",
+]
+requires-python = ">=3.6"
+
+
+[project.scripts]
+realpython = "hys_scraper.__main__:main"

pyproject.toml

@@ -0,0 +1,26 @@
+#
+# This file is autogenerated by pip-compile with python 3.9
+# To update, run:
+#
+#    pip-compile pyproject.toml
+#
+certifi==2022.9.24
+    # via requests
+charset-normalizer==2.1.1
+    # via requests
+idna==3.4
+    # via requests
+numpy==1.23.4
+    # via pandas
+pandas==1.5.0
+    # via hys-scraper (pyproject.toml)
+python-dateutil==2.8.2
+    # via pandas
+pytz==2022.4
+    # via pandas
+requests==2.28.1
+    # via hys-scraper (pyproject.toml)
+six==1.16.0
+    # via python-dateutil
+urllib3==1.26.12
+    # via requests

requirements.txt

@@ -0,0 +1,3 @@
+from hys_scraper.hys_scraper import HYS_Scraper
+
+__version__ = "0.1.0"

src/hys_scraper/__init__.py

@@ -0,0 +1,56 @@
+import argparse
+
+from hys_scraper import HYS_Scraper
+
+
+def main():
+    """Runs the 'Have your Say' parser with command line arguments."""
+    parser = argparse.ArgumentParser(
+        prog="python3 -m hys_scraper",
+        description="Scrape feedback and statistics from the European Commission's "
+        + "'Have your Say' plattform.",
+    )
+    parser.add_argument(
+        "publication_id",
+        metavar="PID",
+        type=str,
+        help="The publication id - what comes after 'p_id=' in the initiative's URL.",
+    )
+    parser.add_argument(
+        "--dir",
+        "--target_dir",
+        metavar="target_dir",
+        type=str,
+        default=None,
+        help="Directory to save the feedback and statistics dataframes to. "
+        + "Defaults to creating a new folder in the current working directory.",
+    )
+    parser.add_argument(
+        "--no_attachments",
+        action="store_true",
+        help="Whether to skip the download of attachments.",
+    )
+    parser.add_argument(
+        "--sleep_time",
+        metavar="t",
+        type=int,
+        default=None,
+        help="Minimum time between consecutive HTTP requests (in seconds).",
+    )
+
+    # Deviate from scraper's default values only if user specified something
+    args = parser.parse_args()
+    kwargs = {}
+    if args.dir is not None:
+        kwargs["target_dir"] = args.dir
+    if args.no_attachments:
+        kwargs["download_attachments"] = False
+    if args.sleep_time is not None:
+        kwargs["sleep_time"] = args.sleep_time
+
+    # Scrape using the user-set parameters
+    HYS_Scraper(args.publication_id, **kwargs).scrape()
+
+
+if __name__ == "__main__":
+    main()