Skip to content

README.md

Latest commit

 

History

History
148 lines (123 loc) · 5.25 KB

README.md

File metadata and controls

148 lines (123 loc) · 5.25 KB

TARAZ

Project Structure

TARAZ/
├── data/                                    # Datasets
│   └── annotations/
│       └── Iran_data.json                   # Iran cultural knowledge dataset (500 Q&A pairs)
├── evaluation/                              # Evaluation system
│   ├── __init__.py                          # Module initialization
│   ├── evaluation_utils.py                 # Core evaluation utilities
│   ├── persian_evaluation.py               # Persian-specific evaluation logic
│   └── evaluation_results/                 # Evaluation reports
├── models/                                  # Model scripts and configs
│   ├── HooshvareLab-bert-fa-base-uncased/   # Persian BERT model scripts
│   ├── MaralGPT-Maral-7B-alpha-1/          # MaralGPT model scripts
│   └── ViraIntelligentDataMining-PersianLLaMA-13B/  # PersianLLaMA model scripts
├── model_cache/                             # Large model files (gitignored)
│   ├── HooshvareLab-bert-fa-base-uncased/   # BERT model weights
│   ├── MaralGPT-Maral-7B-alpha-1/          # MaralGPT model weights
│   └── ViraIntelligentDataMining-PersianLLaMA-13B/  # PersianLLaMA model weights
├── results/                                 # Processing results
│   └── annotations/
│       ├── HooshvareLab-bert-fa-base-uncased/  # BERT embeddings output (500 files)
│       ├── MaralGPT-Maral-7B-alpha-1/         # MaralGPT text responses output (500 files)
│       └── ViraIntelligentDataMining-PersianLLaMA-13B/  # PersianLLaMA text responses output (500 files)
├── requirements.txt
├── setup_external_deps.sh                  # External dependencies setup
├── utils.py                                # General utilities
└── README.md

Setup

  1. Create and activate virtual environment:

    python3 -m venv .venv
source .venv/bin/activate

  2. Install dependencies:

    pip install -r requirements.txt

  3. Set up environment variables:

    cp .env.example .env

    Edit .env and fill in the appropriate values.

  4. Download models:

    # Download BERT model
cd models/HooshvareLab-bert-fa-base-uncased
python download_model.py

# Download MaralGPT model
cd ../MaralGPT-Maral-7B-alpha-1
python download_model.py

# Download PersianLLaMA model
cd ../ViraIntelligentDataMining-PersianLLaMA-13B
python download_model.py

  5. Run examples:

    # BERT examples
cd models/HooshvareLab-bert-fa-base-uncased
python example_usage.py

# MaralGPT examples
cd ../MaralGPT-Maral-7B-alpha-1
python example_usage.py

# PersianLLaMA examples
cd ../ViraIntelligentDataMining-PersianLLaMA-13B
python example_usage.py

Iran Dataset Processing

Process the Iran cultural knowledge dataset (500 Q&A pairs) with different models:

BERT Embeddings

Generate embeddings for both Farsi and English questions:

cd models/HooshvareLab-bert-fa-base-uncased
python process_iran_data.py --limit 5    # Test with 5 entries
python process_iran_data.py              # Process all 500 entries

MaralGPT Text Responses

Generate actual text responses to questions:

cd models/MaralGPT-Maral-7B-alpha-1
python process_iran_data.py --limit 5    # Test with 5 entries
python process_iran_data.py              # Process all 500 entries

PersianLLaMA Text Responses

Generate text responses using the 13B parameter PersianLLaMA model:

cd models/ViraIntelligentDataMining-PersianLLaMA-13B
python process_iran_data.py --limit 5    # Test with 5 entries
python process_iran_data.py              # Process all 500 entries

Output Structure

  • BERT results: results/annotations/HooshvareLab-bert-fa-base-uncased/
    • Individual JSON files with 768-dimensional embeddings
    • Both CLS token and mean pooled embeddings
  • MaralGPT results: results/annotations/MaralGPT-Maral-7B-alpha-1/
    • Individual JSON files with generated text responses (7B model)
    • Performance metrics and token counts
  • PersianLLaMA results: results/annotations/ViraIntelligentDataMining-PersianLLaMA-13B/
    • Individual JSON files with generated text responses (13B model)
    • Performance metrics and token counts

Evaluation System

BLend-compatible soft exact match evaluation for Persian cultural knowledge:

Method

  • Soft exact match: Lemmatization-based semantic matching
  • Persian lemmatization: Hazm library for root word extraction
  • English fallback: spaCy lemmatization when Persian fails
  • Weighted scoring: Human annotator confidence weighting
  • Binary + weighted metrics: Pass/fail and confidence-adjusted scores

Usage Example

cd models/MaralGPT-Maral-7B-alpha-1
python analyze_annotation_matches.py

Output

  • Location: evaluation/evaluation_results/
  • Format: JSON with detailed per-entry results
  • Metrics: Binary accuracy, weighted accuracy, match statistics

Notes

  • Model weights are stored in model_cache/ and excluded from version control
  • Scripts and configurations are kept in models/ for version tracking
  • Results are saved in structured JSON format with comprehensive metadata