X-Timeline Scraper

A Python client to scrape tweets from X (formerly Twitter) timelines using a cURL command.

---

Introduction

This project provides a Python client to scrape tweets from X (formerly Twitter) timelines using a cURL command. It leverages asynchronous programming for efficient data retrieval and includes features for parsing tweet data.

Table of Contents 🗂

• Installation
• Usage
• Citation
• Contributing
• License

Installation ⚙️

To install the X-Timeline Scraper, you can use pip:

pip install xtimeline

Usage ⌨️

To use the X-Timeline Scraper, you need to provide a cURL command that accesses the desired X timeline. The instructions can be found in curl_example.txt. Then, you can use the XTimelineClient class to fetch and parse tweets.

Here's a simple example of how to use the client:

import asyncio
from xclient import XTimelineClient

async with XTimelineClient(
        "curl.txt", persist_last_id_path="state/last_id.txt"
    ) as xc:
        tweets = await xc.fetch_tweets(update_last_id=False)
        for t in tweets:
            print(t.to_markdown())

You can also stream new tweets in real-time:

import asyncio
from xclient import XTimelineClient
async with XTimelineClient(
        "curl.txt", persist_last_id_path="state/last_id.txt"
    ) as xc:
        async for t in xc.stream(interval_s=5.0):
            print(t.to_markdown())

Citation ✍️

If you use this project in your research, please cite as follows:

@misc{project_name,
  author  = {Stephan Akkerman},
  title   = {X-Timeline Scraper},
  year    = {2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/StephanAkkerman/x-timeline-scraper}}
}

Contributing 🛠

Contributions are welcome! If you have a feature request, bug report, or proposal for code refactoring, please feel free to open an issue on GitHub. We appreciate your help in improving this project.

License 📜

This project is licensed under the MIT License. See the LICENSE file for details.