Skip to content

erenirmak/ModelBench

BranchesTags

Repository files navigation

🤖 ModelBench - AI Model Comparison Tool

A professional Flask-based web application for comparing multiple AI models from Hugging Face Hub with advanced analytics, interactive visualizations, and a modern HTML/CSS/JavaScript interface.

✨ Key Differences from Streamlit Version

  • Professional Web Interface: Built with Flask, HTML5, CSS3, and JavaScript
  • Floating Color Picker Modal: Adobe Photoshop-style color picker with HSV controls
  • Advanced Charts: Interactive Plotly charts (bar, line, scatter)
  • Responsive Design: Modern, mobile-friendly interface
  • RESTful API Backend: Clean separation of backend logic and frontend
  • Better Performance: Lightweight frontend with efficient state management

🌟 Features

🔍 Model Search & Discovery

  • Smart Search: Search for models on Hugging Face Hub with real-time autocomplete suggestions
  • Model Validation: Automatically validates model availability and accessibility
  • Support for Multiple Formats: Handles both single and sharded (multi-file) SafeTensors models

💾 Memory Analysis & Estimation

  • Accurate Memory Calculation: Estimates total memory requirements for models including:
    • Parameter counts
    • Memory breakdown by data type (dtype)
    • Per-component memory analysis
  • Data Type Breakdown: Detailed visualization of memory usage by dtype:
    • Shows memory in GB
    • Displays percentage distribution
    • Provides parameter counts per dtype
  • Support for Complex Architectures:
    • Transformer models
    • Sentence transformer models with dense embeddings
    • Multi-component architectures

📊 Side-by-Side Comparison

  • Flexible Model Selection: Compare 1 to 5 models simultaneously
  • Interactive Comparison: Dynamically adjust the number of models to compare
  • Memory Comparison Charts: Visual bar charts comparing memory requirements across models
  • Detailed Comparison Table: Tabular view showing model IDs, memory usage, and parameter counts

🎨 Memory Visualization

  • Multiple Chart Types: Choose between bar, line, and scatter charts
  • Dtype Breakdown Charts: Bar charts showing memory distribution by data type
  • Interactive Plotly Charts: Zoom, pan, and hover for detailed information
  • Responsive Design: Optimized for different screen sizes
  • Chart Customization:
    • Select chart type (bar, line, scatter)
    • Floating modal color picker window (like Photoshop/Adobe tools)
    • Hue slider with gradient bar
    • Saturation and brightness controls
    • Real-time color preview
    • Display HEX, HSV, and RGB values
    • Apply or Close the modal
    • Adjust chart height (300-800 pixels)
    • Customize chart title
    • Toggle gridlines on/off
    • Toggle legend visibility

💾 Data Export & Reporting

  • CSV Export: Download comparison results in CSV format with full details
  • JSON Export: Export structured JSON data for programmatic use
  • HTML Export: Generate beautiful, interactive HTML reports with:
    • Professional styling and gradient design
    • Executive summary with key metrics
    • Detailed comparison tables
    • Per-model analysis cards with breakdown charts
    • Interactive Plotly bar charts for visual comparison
    • Print-friendly layout
    • Responsive design for all devices
    • Embedded JavaScript visualization
  • Timestamped Files: Automatically timestamped exports to avoid overwriting
  • Comprehensive Data:
    • Model identifiers
    • Total memory and parameters
    • Per-dtype memory breakdown
    • Percentage distributions

⚙️ Settings & Cache Management

  • Dynamic Model Count: Slider to select number of models to compare (1-5)
  • Cache Management System:
    • View cached models
    • Clear individual model cache
    • Clear all cache at once
    • Speeds up repeated comparisons
  • Chart Customization Panel (Sidebar Expander):
    • Multiple chart types (bar, line, scatter)
    • Floating HSV color picker modal window
    • Hue slider with gradient bar visualization
    • Saturation and brightness controls
    • Real-time color preview
    • Display HEX, HSV, and RGB values
    • Apply or Close buttons
    • Adjustable chart height
    • Custom title input
    • Gridline toggle
    • Legend visibility toggle
  • Sample Data: Quick-load sample models for testing

🚀 User Interface & Experience

  • Modern Gradient Design: Beautiful gradient backgrounds and styling
  • Responsive Layout: Works seamlessly on different devices
  • Empty State Guidance: Helpful prompts when no data is loaded
  • Progress Indicators: Spinning loaders and success messages
  • Error Handling: Clear error messages and warnings for troubleshooting

🔐 Authentication & Security

  • Hugging Face Token Support: Optional HF_TOKEN for private model access
  • Secure API Communication: Uses HTTPS and proper authentication headers
  • HTTP/2 Support: Modern protocol for faster API communication

⚡ Performance Optimizations

  • Asynchronous Operations: Uses asyncio for concurrent API calls
  • Concurrent Downloads: Configurable MAX_CONCURRENCY for parallel model loading
  • Caching: Streamlit @st.cache_data for avoiding redundant API calls
  • Connection Pooling: Optimized HTTP client with connection limits
  • Configurable Timeouts: REQUEST_TIMEOUT configuration for reliability

📋 Data Import Features

  • Sample Models: Quickly load popular models (Llama-2-7b, Mistral-7B, Nous-Hermes-2-7b)
  • Custom Model Input: Enter any Hugging Face model ID
  • Bulk Operations: Load multiple models for batch comparison

🛠️ Technical Features

  • Model Format Support:
    • Single-file SafeTensors (.safetensors)
    • Sharded SafeTensors (model.safetensors.index.json)
  • Component Analysis: Breaks down memory by model components
  • Dtype Analysis: Supports all SafeTensors dtype formats
  • Memory Unit Conversions: Automatic conversion between bytes, GB, and human-readable formats
  • Large Number Formatting: Automatic K, M, B, T suffix formatting for readability

📱 User Interface Components

  • Model Selection Columns: Individual input fields for each model
  • Status Indicators: Real-time validation and success/error messages
  • Interactive Expanders: Collapsible sections for memory details
  • Action Buttons: Compare, Load Samples, Clear All functions
  • Data Tables: Multiple comparison tables with different perspectives
  • Download Buttons: Easy export with one-click functionality

🔧 Configuration & Customization

  • Environment Variables:
    • HF_TOKEN: Hugging Face authentication token
    • REQUEST_TIMEOUT: API request timeout (default: 10.0 seconds)
    • MAX_WORKERS: Maximum concurrent workers (default: CPU count + 4)
  • Adjustable Parameters: MAX_METADATA_SIZE and other constants
  • Customizable Styling: CSS styling for modern UI appearance

📦 Requirements

  • Python >=3.10
  • Flask and Flask-CORS
  • Pandas and Plotly for data processing and visualization
  • httpx for async HTTP requests
  • hf-mem for model metadata analysis
  • python-dotenv for environment configuration

🚀 Getting Started

Installation

# Navigate to the project directory
cd ModelBench

# Install dependencies
pip install -r requirements.txt
# or
uv sync

Configuration

Create a .env file in the project root:

# Optional: Add your Hugging Face token for private models
HF_TOKEN=your_token_here

# Optional: Configure request timeout (in seconds)
REQUEST_TIMEOUT=10.0

# Optional: Configure max concurrent workers
MAX_WORKERS=8

# Flask configuration (optional)
FLASK_ENV=development
FLASK_DEBUG=True

Running the Application

# From the project root directory
python app.py

The application will start at http://localhost:5000

📖 Usage Guide

  1. Set Model Count: Use the sidebar slider to choose how many models to compare (1-5)
  2. Enter Model IDs: Type model IDs from Hugging Face Hub (e.g., meta-llama/Llama-2-7b)
  3. Use Suggestions: Click on autocomplete suggestions for quick selection
  4. Customize Chart: Open the Chart Customization panel to adjust visualization options
  5. Pick Colors: Click the 🎨 Pick button to open the floating color picker modal
  6. Compare Models: Click "Compare Models" to fetch and analyze selected models
  7. View Results: Charts, tables, and detailed memory breakdowns appear automatically
  8. Export Data: Download results in CSV, JSON, or HTML format

🎨 Professional Color Picker

  • Floating Modal Window: Centered overlay modal with dark backdrop
  • HSV Controls:
    • Hue slider (0-360°) for color selection
    • Saturation slider (0-100%) for intensity
    • Brightness slider (0-100%) for lightness
  • Real-Time Preview: Color swatch updates instantly
  • Multiple Formats: Display HEX, HSV, and RGB values
  • Professional Styling: Adobe Photoshop-like interface

🎯 Use Cases

  • Model Selection: Compare memory requirements before deployment
  • Resource Planning: Estimate GPU/CPU requirements for your infrastructure
  • Model Evaluation: Analyze different model sizes and architectures
  • Documentation: Generate and export comparison reports
  • Research: Understand model memory characteristics and trade-offs
  • Optimization: Identify opportunities for model pruning or quantization

📂 Project Structure

ModelBench/
├── app.py                          # Flask backend with API routes
├── requirements.txt                # Python dependencies
├── pyproject.toml                  # Project configuration
├── .env                            # Environment variables (create from .env.sample)
├── .env.sample                     # Example environment configuration
├── README.md                       # This file
├── templates/
│   └── index.html                 # Main HTML template
└── static/
    ├── css/
    │   └── style.css              # Responsive styling (500+ lines)
    └── js/
        ├── main.js                # Main application logic (300+ lines)
        ├── color-picker.js        # Color picker modal implementation
        └── chart.js               # Chart visualization with Plotly

🔄 Workflow

  1. Selection Phase: User enters model IDs or uses suggestions
  2. Analysis Phase: Backend fetches and analyzes model metadata
  3. Visualization Phase: Charts and tables display automatically
  4. Comparison Phase: Side-by-side analysis and metrics
  5. Export Phase: Download results in preferred format

💾 API Endpoints

/ (GET)

Main page serving the HTML interface

/api/search (POST)

Search for models on Hugging Face Hub

Request: { "query": "llama", "limit": 10 }
Response: { "status": "success", "models": [...] }

/api/analyze (POST)

Analyze a model's memory requirements

Request: { "model_id": "meta-llama/Llama-2-7b" }
Response: { "status": "success", "data": {...} }

/api/export/csv (POST)

Export comparison data as CSV

/api/export/json (POST)

Export comparison data as JSON

/api/export/html (POST)

Export comparison data as HTML report

🎨 Frontend Architecture

State Management

  • Centralized appState object for all application data
  • Model cache for fast repeated comparisons
  • Chart configuration management

Component Structure

  • Sidebar: Settings, customizations, cache management
  • Main Content: Model inputs, comparison results, exports
  • Color Picker Modal: Floating window for color selection
  • Chart Section: Interactive Plotly visualizations
  • Table Section: Detailed comparison data

⚡ Performance Features

  • Async API Calls: Efficient backend processing
  • Client-Side Caching: Avoid redundant API calls
  • Lazy Loading: Sections appear only when needed
  • Responsive Charts: Mobile-friendly visualizations
  • Efficient State Management: Minimal re-renders

🔐 Security Features

  • CORS Protection: Flask-CORS for secure cross-origin requests
  • Input Validation: Backend validation of all inputs
  • Error Handling: Graceful error messages for users
  • Token Support: Optional HF_TOKEN for private models

📱 Browser Compatibility

  • Chrome/Chromium (latest)
  • Firefox (latest)
  • Safari (latest)
  • Edge (latest)
  • Mobile browsers (responsive design)

🐛 Troubleshooting

Port Already in Use

# Change port in app.py
app.run(debug=True, port=5001)

Module Import Errors

# Reinstall dependencies
pip install --upgrade -r requirements.txt

API Connection Issues

  • Verify internet connection
  • Check HF_TOKEN if accessing private models
  • Verify REQUEST_TIMEOUT setting

🚀 Future Enhancements

  • Database integration for comparison history
  • User authentication and saved profiles
  • Advanced filtering and sorting
  • Model benchmark integration
  • Real-time API updates
  • Dark mode toggle
  • Custom color schemes
  • Batch model comparisons

📝 License

Refer to the LICENSE file in the project directory.

🙋 Support

For issues or questions:

  1. Check browser console for error messages (F12)
  2. Verify server is running (http://localhost:5000)
  3. Check Flask logs for backend errors
  4. Ensure all dependencies are installed
  5. Verify model IDs are in correct format (owner/model-name)

ModelBench Flask Edition v1.0 | Built with Flask, HTML5, CSS3, JavaScript & ❤️

About

Compare VRAM Requirements of Huggingface Models

Resources

Readme

License

MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages