Persian Telegram News NLP

A small toolkit that turns a raw Telegram channel export into a clean, analysis-ready Persian text corpus. The new Python package living under src/ wraps the one-off notebook code into reusable modules, a command line interface, and optional Telethon helpers for direct downloads.

---

Project layout

.
├─ data/
│  ├─ processed/                 # default location for generated CSVs (ignored by git)
│  └─ ...                        # place your raw exports here
├─ notebooks/
│  └─ NLP_processing.ipynb       # legacy exploratory analysis notebook
├─ src/
│  └─ persian_news_nlp/
│     ├─ __init__.py             # public API exports
│     ├─ __main__.py             # enables `python -m persian_news_nlp`
│     ├─ analysis.py             # descriptive statistics helpers
│     ├─ cleaning.py             # normalization/token utilities
│     ├─ cli.py                  # argparse-based entry point
│     ├─ config.py               # dataclasses describing pipeline options
│     ├─ io.py                   # JSON/CSV readers and writers
│     ├─ pipeline.py             # orchestration logic for the ETL
│     └─ telegram.py             # optional Telethon downloader
├─ stopwords.data                # UTF-8 stopword list (one token per line)
├─ requirements.txt              # base + optional dependencies
└─ tests/
   └─ test_pipeline.py           # regression test for the core pipeline

The raw export (channel_messages.json) stays outside of version control. The pipeline writes cleaned artifacts to data/processed/filtered_messages.csv, data/processed/final_results.csv, and data/processed/top_tokens.csv.

---

Installation

python3 -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -U pip
pip install -r requirements.txt

hazm and telethon are optional but recommended. The package falls back to regex-based normalization when hazm is missing.

---

Command line usage

1. Drop your Telegram export at channel_messages.json (or point to a custom path).
2. Run the CLI:

python -m persian_news_nlp \
  --input channel_messages.json \
  --stopwords stopwords.data \
  --output-dir data/processed \
  --start 2022-09-05T00:00:00+00:00 \
  --end   2022-11-05T00:00:00+00:00 \
  --top-n 40

Flags of note:

• --stopwords (optional) – UTF-8 file with one token per line.
• --start/--end – ISO-8601 timestamps to constrain the window.
• --keep-empty – retain messages even if they turn empty after cleaning.
• --top-n – number of frequent tokens to store in top_tokens.csv.

The command prints the paths of the generated CSVs on success.

---

Python API

from pathlib import Path
from persian_news_nlp import DateRange, PipelineConfig, run_pipeline

config = PipelineConfig(
    input_path=Path("channel_messages.json"),
    output_dir=Path("data/processed"),
    stopwords_path=Path("stopwords.data"),
    date_range=DateRange(start=None, end=None),
)

result = run_pipeline(config, top_n_tokens=25)
result.write_outputs(config)

PipelineResult exposes three ready-to-use pandas.DataFrame objects:

• filtered_messages: sorted messages with raw/clean text side by side.
• daily_summary: daily activity counts with average token and character lengths.
• top_tokens: frequency table for the most common cleaned tokens.

---

Optional Telegram download helper

persian_news_nlp.telegram wraps the Telethon client for convenience. Provide your API credentials (https://my.telegram.org) and run inside an async context:

import asyncio
from pathlib import Path
from persian_news_nlp.telegram import TelegramCredentials, download_channel_history

creds = TelegramCredentials(api_id=12345, api_hash="abc...", phone="+9891...")
asyncio.run(download_channel_history("https://t.me/bbcpersian", creds, Path("channel_messages.json")))

The helper relies on Telethon’s interactive login the first time you run it (a .session file is created locally).

---

Legacy notebook workflow

The original notebooks/NLP_processing.ipynb remains for exploratory analysis (word clouds, classifiers, etc.). Update the imports to use the new package modules where possible—e.g., reuse pipeline.run_pipeline to bootstrap clean data before diving into custom modeling.

---

Tests

A smoke test lives under tests/. Run it (after installing pytest) to validate the pipeline:

pytest

---

Notes and cautions

• Review outputs before sharing — Telegram exports often contain sensitive user content.
• UTC datetimes are preserved; adjust to local time zones if needed.
• Remember to keep channel_messages.json untracked or encrypted when collaborating.