fix rk, update docs, fix json gen, update html viz

claromes · claromes · commit d693cb26de14 · 2025-05-25T06:57:19.000-03:00
diff --git a/.gitignore b/.gitignore
@@ -1,6 +1,9 @@
 *.csv
 *.json
 *.html
+*.txt
+
+test.py
 
 waybacktweets/__pycache__
 waybacktweets/api/__pycache__
diff --git a/README.md b/README.md
@@ -1,7 +1,6 @@
 # Wayback Tweets
 
-[![PyPI](https://img.shields.io/pypi/v/waybacktweets)](https://pypi.org/project/waybacktweets) [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.12528447.svg)](https://doi.org/10.5281/zenodo.12528447) [![Streamlit App](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://waybacktweets.streamlit.app) [![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1tnaM3rMWpoSHBZ4P_6iHFPjraWRQ3OGe?usp=sharing)
-
+[![PyPI](https://img.shields.io/pypi/v/waybacktweets)](https://pypi.org/project/waybacktweets) [![PyPI Downloads](https://static.pepy.tech/badge/waybacktweets)](https://pepy.tech/projects/waybacktweets)
 
 Retrieves archived tweets CDX data from the Wayback Machine, performs necessary parsing (see [Field Options](https://claromes.github.io/waybacktweets/field_options.html)), and saves the data in HTML, for easy viewing of the tweets using the iframe tags, CSV, and JSON formats.
 
@@ -11,21 +10,50 @@ Retrieves archived tweets CDX data from the Wayback Machine, performs necessary
 pip install waybacktweets
 ```
 
-## Quickstart
-
-### Using Wayback Tweets as a standalone command line tool
-
-waybacktweets [OPTIONS] USERNAME
+## CLI
 
 ```shell
-waybacktweets --from 20150101 --to 20191231 --limit 250 jack
+Usage: waybacktweets [OPTIONS] USERNAME
+
+  USERNAME: The Twitter username without @
+
+Options:
+  -c, --collapse [urlkey|digest|timestamp:XX]
+                                  Collapse results based on a field, or a
+                                  substring of a field. XX in the timestamp
+                                  value ranges from 1 to 14, comparing the
+                                  first XX digits of the timestamp field. It
+                                  is recommended to use from 4 onwards, to
+                                  compare at least by years.
+  -f, --from DATE                 Filtering by date range from this date.
+                                  Format: YYYYmmdd
+  -t, --to DATE                   Filtering by date range up to this date.
+                                  Format: YYYYmmdd
+  -l, --limit INTEGER             Query result limits.
+  -rk, --resumption_key TEXT      Allows for a simple way to scroll through
+                                  the results. Key to continue the query from
+                                  the end of the previous query.
+  -mt, --matchtype [exact|prefix|host|domain]
+                                  Results matching a certain prefix, a certain
+                                  host or all subdomains.
+  -v, --verbose                   Shows the log.
+  --version                       Show the version and exit.
+  -h, --help                      Show this message and exit.
+
+  Examples:
+
+      Retrieve all tweets: waybacktweets jack
+
+      With options and verbose output: waybacktweets --from 20200305 --to 20231231 --limit 300 --verbose jack
+
+  Documentation:
+
+      https://claromes.github.io/waybacktweets/
 ```
 
-### Using Wayback Tweets as a Web App
-
-[Open the application](https://waybacktweets.streamlit.app), a prototype written in Python with the Streamlit framework and hosted on Streamlit Cloud.
+## Module
 
-### Using Wayback Tweets as a Python Module
+[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1tnaM3rMWpoSHBZ4P_6iHFPjraWRQ3OGe?usp=sharing)
 
 ```python
 from waybacktweets import WaybackTweets, TweetsParser, TweetsExporter
@@ -37,29 +65,52 @@ archived_tweets = api.get()
 
 if archived_tweets:
     field_options = [
+        "archived_urlkey",
         "archived_timestamp",
-        "original_tweet_url",
+        "parsed_archived_timestamp",
         "archived_tweet_url",
+        "parsed_archived_tweet_url",
+        "original_tweet_url",
+        "parsed_tweet_url",
+        "available_tweet_text",
+        "available_tweet_is_RT",
+        "available_tweet_info",
+        "archived_mimetype",
         "archived_statuscode",
+        "archived_digest",
+        "archived_length",
+        "resumption_key",
     ]
 
     parser = TweetsParser(archived_tweets, USERNAME, field_options)
     parsed_tweets = parser.parse()
 
     exporter = TweetsExporter(parsed_tweets, USERNAME, field_options)
     exporter.save_to_csv()
+    exporter.save_to_json()
+    exporter.save_to_html()
 ```
 
+## Web App
+
+[![Streamlit App](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://waybacktweets.streamlit.app)
+
+A prototype written in Python with the Streamlit framework and hosted on Streamlit Cloud.
+
+> [!NOTE]
+> Starting from version 1.0, the web app will not receive all updates from the official package. To access all features, prefer the package via PyPI.
+
 ## Documentation
 
 - [Wayback Tweets documentation](https://claromes.github.io/waybacktweets)
 - [Wayback CDX Server API (Beta) documentation](https://archive.org/developers/wayback-cdx-server.html)
 
 ## Acknowledgements
 
-- Tristan Lee (Bellingcat's Data Scientist) for the idea of the application.
+- Tristan Lee (Bellingcat's Data Scientist) for the idea.
 - Jessica Smith (Snowflake's Community Growth Specialist) and Streamlit/Snowflake team for the additional server resources on Streamlit Cloud.
-- OSINT Community for recommending the application.
+- OSINT Community for recommending the package and the application.
 
-> [!NOTE]
-> If the Streamlit application is down, please check the [Streamlit Cloud Status](https://www.streamlitstatus.com/).
+## License
+
+[GPL-3.0](LICENSE.md)
diff --git a/docs/conf.py b/docs/conf.py
@@ -5,7 +5,7 @@
 project = "Wayback Tweets"
 release, version = get_version("waybacktweets")
 rst_epilog = f".. |release| replace:: v{release}"
-copyright = f"2023 - {datetime.datetime.now().year}, Claromes · Icon by The Doodle Library · Title font by Google, licensed under the Open Font License · Pre-release: v{release}"  # noqa: E501
+copyright = f"2023 - {datetime.datetime.now().year}, Claromes · Icon by The Doodle Library · Title font by Google, licensed under the Open Font License · Release: v{release}"  # noqa: E501
 author = "Claromes"
 
 # -- General configuration ---------------------------------------------------
diff --git a/docs/field_options.rst b/docs/field_options.rst
@@ -40,3 +40,5 @@ The package performs several parses to facilitate the analysis of archived tweet
 - ``archived_digest``: (`str`) The ``SHA1`` hash digest of the content, excluding the headers. It's usually a base-32-encoded string.
 
 - ``archived_length``: (`int`) The compressed byte size of the corresponding WARC record, which includes WARC headers, HTTP headers, and content payload.
+
+- ``resumption_key``: (`str`) Allows for a simple way to scroll through the results. Key to continue the query from the end of the previous query.
diff --git a/docs/handson.rst b/docs/handson.rst
@@ -19,4 +19,3 @@ Hands-On Examples
       :target: https://colab.research.google.com/drive/1tnaM3rMWpoSHBZ4P_6iHFPjraWRQ3OGe?usp=sharing
       :alt: Open In Collab
 
-.. raw:: html
diff --git a/docs/index.rst b/docs/index.rst
@@ -39,23 +39,21 @@ Command-Line Interface
 
     cli
 
-Streamlit Web App
--------------------
+API Reference
+---------------
 
 .. toctree::
     :maxdepth: 2
 
-    streamlit
-
+    api
 
-API Reference
----------------
+Streamlit Web App
+-------------------
 
 .. toctree::
     :maxdepth: 2
 
-    api
-
+    streamlit
 
 Additional Information
 -----------------------
diff --git a/docs/outputs.rst b/docs/outputs.rst
@@ -14,10 +14,14 @@ This format allows for easy viewing of the archived tweets, through the use of t
 
 - ``original_tweet_url``: (`str`) The original tweet URL.
 
-- ``parsed_tweet_url``: (`str`) The original tweet URL after parsing. Old URLs were archived in a nested manner. The parsing applied here unnests these URLs, when necessary.  Check the :ref:`utils`.
+- ``parsed_tweet_url``: (`str`) The original tweet URL after parsing. Old URLs were archived in a nested manner. The parsing applied here unnests these URLs when necessary. Refer to the :ref:`utils` for more details.
 
 Additionally, other fields are displayed.
 
+.. note::
+
+    The iframes (accordions) are best viewed in Firefox.
+
 CSV
 --------
 
diff --git a/docs/quickstart.rst b/docs/quickstart.rst
@@ -12,13 +12,6 @@ waybacktweets [OPTIONS] USERNAME
 
     waybacktweets --from 20150101 --to 20191231 --limit 250 jack
 
-Web App
--------------
-
-Using Wayback Tweets as a Streamlit Web App.
-
-`Open the application <https://waybacktweets.streamlit.app>`_, a prototype written in Python with the Streamlit framework and hosted on Streamlit Cloud.
-
 Module
 -------------
 
@@ -35,14 +28,34 @@ Using Wayback Tweets as a Python Module.
 
     if archived_tweets:
         field_options = [
+            "archived_urlkey",
             "archived_timestamp",
-            "original_tweet_url",
+            "parsed_archived_timestamp",
             "archived_tweet_url",
+            "parsed_archived_tweet_url",
+            "original_tweet_url",
+            "parsed_tweet_url",
+            "available_tweet_text",
+            "available_tweet_is_RT",
+            "available_tweet_info",
+            "archived_mimetype",
             "archived_statuscode",
+            "archived_digest",
+            "archived_length",
+            "resumption_key",
         ]
 
         parser = TweetsParser(archived_tweets, USERNAME, field_options)
         parsed_tweets = parser.parse()
 
         exporter = TweetsExporter(parsed_tweets, USERNAME, field_options)
         exporter.save_to_csv()
+        exporter.save_to_json()
+        exporter.save_to_html()
+
+Web App
+-------------
+
+Using Wayback Tweets as a Streamlit Web App.
+
+`Open the application <https://waybacktweets.streamlit.app>`_, a prototype written in Python with the Streamlit framework and hosted on Streamlit Cloud.
diff --git a/docs/streamlit.rst b/docs/streamlit.rst
@@ -1,6 +1,10 @@
 Web App
 =========
 
+.. note::
+
+   Starting from version 1.0, the web app will not receive all updates from the official package. To access all features, prefer the package via PyPI.
+
 The application is a prototype hosted on Streamlit Cloud, serving as an alternative to the command line tool.
 
 `Open the application <https://waybacktweets.streamlit.app>`_.
@@ -13,8 +17,6 @@ Filters
 
 - Limit: Query result limits.
 
-- Resumption Key: Allows for a simple way to scroll through the results. Key to continue the query from the end of the previous query.
-
 - Only unique Wayback Machine URLs: Filtering by the collapse option using the ``urlkey`` field and the URL Match Scope ``prefix``
 
 
diff --git a/pyproject.toml b/pyproject.toml
@@ -1,13 +1,14 @@
 [tool.poetry]
 name = "waybacktweets"
-version = "1.0rc1"
+version = "1.0"
 description = "Retrieves archived tweets CDX data from the Wayback Machine, performs necessary parsing, and saves the data."
 authors = ["Claromes <support@claromes.com>"]
 license = "GPLv3"
 readme = "README.md"
 repository = "https://github.com/claromes/waybacktweets"
 keywords = [
     "twitter",
+    "X",
     "tweet",
     "internet-archive",
     "wayback-machine",
@@ -16,13 +17,14 @@ keywords = [
     "command-line",
 ]
 classifiers = [
-    "Development Status :: 4 - Beta",
+    "Development Status :: 5 - Production/Stable",
     "Intended Audience :: Developers",
     "Intended Audience :: Science/Research",
     "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
     "Natural Language :: English",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
     "Topic :: Software Development",
     "Topic :: Utilities",
 ]
diff --git a/waybacktweets/_cli.py b/waybacktweets/_cli.py
@@ -138,7 +138,7 @@ def main(
             matchtype,
         )
 
-        print(f"Retrieving the archived tweets of @{username}...")
+        print("Retrieving...")
         archived_tweets = api.get()
 
         if archived_tweets:
diff --git a/waybacktweets/api/export.py b/waybacktweets/api/export.py
@@ -3,7 +3,6 @@
 """
 
 import datetime
-import os
 import re
 from typing import Any, Dict, List, Optional
 
@@ -93,6 +92,17 @@ def save_to_csv(self) -> None:
 
         print(f"Saved to {csv_file_path}")
 
+    def generate_json(self) -> str:
+        """
+        Generates JSON data from the DataFrame (without saving to a file).
+
+        Returns:
+            The JSON-formatted string of the DataFrame.
+        """
+
+        json_data = self.dataframe.to_json(orient="records", lines=False)
+        return json_data
+
     def save_to_json(self) -> None:
         """
         Saves the DataFrame to a JSON file.
@@ -106,14 +116,11 @@ def save_to_html(self) -> None:
         """
         Saves the DataFrame to an HTML file.
         """
-        json_path = f"{self.filename}.json"
-
-        if not os.path.exists(json_path):
-            self.save_to_json()
+        json_data = self.generate_json()
 
         html_file_path = f"{self.filename}.html"
 
-        html = HTMLTweetsVisualizer(self.username, json_path, html_file_path)
+        html = HTMLTweetsVisualizer(self.username, json_data, html_file_path)
 
         html_content = html.generate()
         html.save(html_content)
diff --git a/waybacktweets/api/parse.py b/waybacktweets/api/parse.py
diff --git a/waybacktweets/api/request.py b/waybacktweets/api/request.py
diff --git a/waybacktweets/api/visualize.py b/waybacktweets/api/visualize.py
diff --git a/waybacktweets/utils/utils.py b/waybacktweets/utils/utils.py

-Original file line number
+Diff line change
@@ @@ -1,6 +1,9 @@ @@
 *.csv
 *.json
 *.html
 +*.txt
++
 +test.py
 waybacktweets/__pycache__
 waybacktweets/api/__pycache__
Original file line number	Diff line number	Diff line change
`@@ -19,4 +19,3 @@ Hands-On Examples`
`19`	`19`	`:target: https://colab.research.google.com/drive/1tnaM3rMWpoSHBZ4P_6iHFPjraWRQ3OGe?usp=sharing`
`20`	`20`	`:alt: Open In Collab`
`21`	`21`
`22`		`-.. raw:: html`