Deep Reinforcement Learning for Stock Trading

A comprehensive comparative study of deep reinforcement learning algorithms (PPO, A2C, DDPG) and their ensemble for automated stock trading on Dow Jones 30 stocks.

📋 Project Overview

This project implements and compares multiple state-of-the-art deep reinforcement learning algorithms for financial trading:

• PPO (Proximal Policy Optimization): A policy gradient method that balances exploration and exploitation
• A2C (Advantage Actor-Critic): An actor-critic method with advantage function estimation
• DDPG (Deep Deterministic Policy Gradient): A model-free off-policy algorithm for continuous action spaces
• Ensemble Agent: Combines predictions from all three agents using majority voting

The agents are trained on historical stock data from 28 Dow Jones Industrial Average stocks, with technical indicators including RSI, MACD, CCI, ADX, and OBV to inform trading decisions.

🎯 Key Features

• Multi-Agent Trading System: Train and compare multiple RL algorithms simultaneously
• Custom Trading Environment: Gymnasium-compatible environment with realistic market simulation
• Technical Indicators: 5 technical indicators (RSI, MACD, CCI, ADX, OBV) for enhanced decision-making
• Comprehensive Evaluation: Sharpe ratio, maximum drawdown, total return, and win rate metrics
• Visualization Suite: Net worth tracking, performance comparison charts, and detailed analytics
• Modular Architecture: Clean separation of concerns with dedicated modules for agents, evaluation, and utilities

📊 Dataset

• Stocks: 28 companies from Dow Jones 30 (AAPL, MSFT, JPM, etc.)
• Period: 2009-01-01 to 2020-05-08 (~11.5 years)
• Data Split:
  • Training: 2009-2015 (7 years)
  • Validation: 2016 (1 year)
  • Testing: 2017-2020 (3.5 years)
• Features: Open, High, Low, Close, Volume + 5 technical indicators per stock

🏗️ Project Structure

├── agents/                 # RL agent implementations
│   ├── base_agent.py      # Abstract base class for all agents
│   ├── ppo_agent.py       # PPO agent
│   ├── a2c_agent.py       # A2C agent
│   ├── ddpg_agent.py      # DDPG agent
│   └── ensemble_agent.py  # Ensemble agent
├── config/                 # Configuration files
│   ├── config.py          # Global parameters
│   └── get_config.py      # Configuration loader
├── data/                   # Data storage
│   ├── raw/               # Raw stock data (CSV)
│   └── processed/         # Processed data with indicators
├── envs/                   # Trading environment
│   └── trading_env.py     # Gymnasium-compatible trading environment
├── evaluation/             # Model evaluation and metrics
│   ├── tester.py          # Agent testing functions
│   ├── metrics.py         # Performance metrics calculation
│   └── visualizer.py      # Visualization functions
├── results/                # Training outputs
│   └── run_YYYYMMDD_HHMMSS/
│       ├── models/        # Trained model weights
│       ├── metrics/       # Performance metrics (CSV + Pickle)
│       ├── figures/       # Visualization charts
│       └── final_summary.txt
├── src/                    # Core pipeline
│   ├── pipeline.py        # Main training and evaluation pipeline
│   └── trainer.py         # Agent training logic
├── utils/                  # Utility functions
│   ├── io_utils.py        # Model and metrics I/O
│   ├── data_loader.py     # Data loading and validation
│   ├── directory_manager.py  # Directory management
│   └── report_generator.py   # Report generation
└── main.py                 # Entry point

🚀 Getting Started

Prerequisites

• Python 3.8+
• pip package manager

Installation

1. Clone the repository:

git clone https://github.com/bengisu-sahin/A-Comparative-Study-of-Tabular-and-Deep-Reinforcement-Learning-Methods-for-Financial-Trading.git
cd A-Comparative-Study-of-Tabular-and-Deep-Reinforcement-Learning-Methods-for-Financial-Trading

2. Install dependencies:

pip install -r requirements.txt

Usage

Run the complete training and evaluation pipeline:

python main.py

This will:

1. Load preprocessed stock data with technical indicators
2. Train PPO, A2C, and DDPG agents
3. Create an Ensemble agent combining all three
4. Evaluate all agents on test data
5. Generate performance metrics and visualizations
6. Save results to results/run_YYYYMMDD_HHMMSS/

📈 Pipeline Workflow

The automated pipeline executes the following steps:

1. Data Loading: Loads training, validation, and test datasets for 28 stocks
2. Agent Training: Trains PPO, A2C, and DDPG agents with validation
3. Model Saving: Saves trained models to disk
4. Testing: Evaluates all agents (including Ensemble) on test data
5. Performance Analysis: Computes Sharpe ratio, max drawdown, total return, win rate
6. Result Storage: Saves all metrics, charts, and reports

📊 Output & Results

Each training run creates a timestamped directory with:

Models

• ppo_agent.zip - Trained PPO model
• a2c_agent.zip - Trained A2C model
• ddpg_agent.zip - Trained DDPG model

Metrics

• agent_comparison.csv - Performance comparison table
• Individual agent metrics (CSV + Pickle format)
• Detailed step-by-step balance and net worth tracking

Visualizations

• net_worth_comparison.png - Line chart comparing agent performance over time
• metrics_comparison.png - Bar chart comparing key metrics

Reports

• final_summary.txt - Comprehensive summary of training and evaluation

🔧 Configuration

Modify config/config.py to customize:

• Tickers: Stock symbols to trade
• Training Parameters: Learning rates, timesteps, batch sizes
• Environment Settings: Initial balance, transaction costs, action thresholds
• Technical Indicators: Periods for RSI, MACD, CCI, ADX
• Risk Metrics: Risk-free rate for Sharpe ratio calculation

📚 Key Technologies

• Stable-Baselines3: Deep RL algorithm implementations
• Gymnasium: Environment interface standard
• Pandas: Data manipulation and analysis
• NumPy: Numerical computing
• Matplotlib: Data visualization
• yfinance: Historical market data retrieval

🎓 Algorithms Explained

PPO (Proximal Policy Optimization)

A policy gradient method that uses a clipped surrogate objective to prevent large policy updates, ensuring stable training.

A2C (Advantage Actor-Critic)

An actor-critic method that uses the advantage function to reduce variance in policy gradient estimates.

DDPG (Deep Deterministic Policy Gradient)

An off-policy algorithm designed for continuous action spaces, combining DQN and actor-critic approaches.

Ensemble Agent

Averages the continuous action values from PPO, A2C, and DDPG to create a robust trading strategy that leverages the strengths of all algorithms.

📝 Performance Metrics

The system evaluates agents using:

• Sharpe Ratio: Risk-adjusted return metric
• Maximum Drawdown: Largest peak-to-trough decline
• Total Return: Overall profit/loss percentage
• Win Rate: Percentage of profitable trades
• Final Net Worth: End portfolio value

👤 Author

Bengisu Sahin

• GitHub: @bengisu-sahin
• GitHub: @aydanurakca