Global Terrorism Analysis: A Comprehensive Data Mining Study

Technical University of Munich (TUM) Data Mining Lab Project Advanced Analytics on Global Terrorism Database (GTD) 1970-2016

🎯 Project Overview

This repository contains a comprehensive data mining and machine learning analysis of global terrorism incidents spanning from 1970 to 2016. The project leverages the Global Terrorism Database (GTD) containing over 156,000 terrorist incidents worldwide, applying advanced statistical methods, clustering algorithms, and predictive modeling to understand patterns, trends, and characteristics of terrorist activities.

📊 Dataset Information

Global Terrorism Database (GTD)

• Source: National Consortium for the Study of Terrorism and Responses to Terrorism (START)
• Time Period: 1970-2016
• Total Incidents: 156,772
• Attributes: 137 variables
• Geographic Coverage: Global (all continents)
• Update Frequency: Annual

Key Variables Analyzed

• Temporal: Year, month, day, extended incidents
• Geographic: Country, region, city, coordinates
• Attack Details: Attack type, weapon type, target type
• Casualties: Number killed, wounded, hostages
• Perpetrators: Group names, number of perpetrators
• Textual: Summary, motive, additional notes

🏗️ Repository Structure

global-terrorism-analysis/
├── README.md                           # This documentation
├── requirements.txt                    # Python dependencies
├── environment.yml                     # Conda environment
├── .gitignore                          # Git ignore rules
│
├── data/                               # Data directory
│   ├── raw/                           # Original datasets
│   │   ├── globalterrorismdb_0616dist.xlsx
│   │   ├── Codebook.pdf
│   │   └── .gitkeep
│   ├── processed/                     # Cleaned datasets
│   │   ├── terrorism.csv
│   │   ├── final_group_names.csv
│   │   ├── terrorism_50_train_test.csv
│   │   ├── terrorism_50_val.csv
│   │   ├── terrorism_red_cat_for_random_forest.csv
│   │   ├── terrorism_red_cat_with_country.csv
│   │   └── .gitkeep
│   └── external/                      # External data sources
│       ├── Fossil Fuels.csv
│       ├── Fuel Imports.csv
│       ├── National Income.csv
│       ├── Population.csv
│       └── .gitkeep
│
├── notebooks/                         # Jupyter notebooks
│   ├── 01_data_exploration/           # EDA and data understanding
│   │   ├── missing_data_analysis.ipynb
│   │   ├── frequency_analysis.ipynb
│   │   ├── heatmaps_analysis.ipynb
│   │   └── text_mining_analysis.ipynb
│   ├── 02_visualization/             # Advanced visualizations
│   │   ├── joint_plots_analysis.ipynb
│   │   ├── missing_data_visualization.ipynb
│   │   ├── group_activity_analysis/
│   │   ├── group_name_research/
│   │   └── hostage_analysis.ipynb
│   ├── 03_clustering/                 # Clustering analysis
│   │   ├── Clustering/
│   │   └── terrorist_chapter_analysis/
│   └── 04_prediction/                 # Predictive modeling
│       ├── Prediction (nkill + nwound).ipynb
│       ├── Prediction (nkill).ipynb
│       ├── Prediction (nwound).ipynb
│       ├── Prediction group.ipynb
│       ├── Data Cleaning for Prediction [nkill, nwound].ipynb
│       └── Create 50-50 Split.ipynb
│
├── src/                               # Source code
│   ├── data/                          # Data processing modules
│   │   ├── __init__.py
│   │   └── preprocessing.py
│   ├── analysis/                      # Analysis modules
│   │   ├── __init__.py
│   │   └── clustering.py
│   └── utils/                         # Utility functions
│       └── __init__.py
│
├── scripts/                           # R and Python scripts
│   ├── data_processing.R
│   ├── create_dataset_predict_gname.R
│   └── terrorism_red_cat_for_nkill_pred.csv
│
├── reports/                           # Analysis reports
│   ├── figures/                       # Generated figures
│   │   ├── wordclouds/
│   │   ├── .gitkeep
│   │   └── [various PNG files]
│   └── videos/                        # Animated visualizations
│       ├── week2_visualizations/
│       ├── week3_analysis/
│       ├── week4_clustering/
│       └── .gitkeep
│
├── models/                            # Trained models
│   ├── prediction_results/
│   └── .gitkeep
│
├── tests/                            # Test files
│   └── test_data_processing.py
│
└── docs/                             # Documentation
    └── methodology.md

🔬 Research Methodology

Phase 1: Data Exploration & Preprocessing

• Missing Data Analysis: Comprehensive assessment of data completeness
• Frequency Analysis: Temporal, geographic, and categorical distributions
• Text Mining: Word frequency analysis and cloud generation
• Data Quality: Validation and cleaning procedures

Phase 2: Advanced Analytics & Visualization

• Joint Plot Analysis: Multi-dimensional data relationships
• Group Activity Analysis: Terrorist organization behavior patterns
• Animated Visualizations: Time-series animations of terrorist activities
• Geographic Analysis: Spatial distribution and patterns

Phase 3: Clustering Analysis

• Terrorist Group Clustering: Unsupervised learning to identify group patterns
• Attack Type Clustering: Categorization of attack methodologies
• Target Type Clustering: Classification of attack targets
• Weapon Type Clustering: Weapon usage pattern analysis

Phase 4: Predictive Modeling

• Casualty Prediction: Models to predict number of killed and wounded
• Group Prediction: Classification of responsible terrorist groups
• Feature Engineering: Creation of predictive variables
• Model Validation: Cross-validation and performance assessment

📈 Key Findings

Temporal Patterns

• Peak Activity: 2014-2015 showed highest terrorist activity
• Seasonal Trends: Certain months and days show higher incident rates
• Long-term Trends: Evolution of terrorist tactics over decades

Geographic Distribution

• Regional Hotspots: Middle East, South Asia, and Africa
• Country Analysis: Iraq, Afghanistan, and Pakistan as major targets
• Urban vs Rural: Concentration of attacks in urban areas

Attack Characteristics

• Weapon Preferences: Explosives and firearms most common
• Target Types: Private citizens, military, and government facilities
• Attack Methods: Bombings and armed assaults dominate

Group Analysis

• Major Organizations: ISIS, Taliban, Boko Haram among top groups
• Group Specialization: Different groups favor different attack types
• Geographic Focus: Groups tend to operate in specific regions

🛠️ Technical Implementation

Data Processing Pipeline

# Data loading and preprocessing
data = load_gtd_data('data/raw/globalterrorismdb_0616dist.xlsx')
cleaned_data = preprocess_data(data)
features = engineer_features(cleaned_data)

Machine Learning Models

• Clustering: K-means, hierarchical clustering, DBSCAN
• Classification: Random Forest, SVM, Neural Networks, XGBoost
• Regression: Linear regression, ensemble methods, deep learning
• Text Mining: TF-IDF, word embeddings, sentiment analysis

Visualization Framework

• Static Plots: Matplotlib, Seaborn, ggplot2
• Interactive Visualizations: Plotly, Bokeh, D3.js
• Animated Charts: Time-series animations, geographic flows
• Dashboard: Streamlit, Dash, or Jupyter widgets

📊 Analysis Results

Clustering Results

• Group Clusters: Identified distinct terrorist group categories
• Attack Patterns: Clustered similar attack methodologies
• Target Clusters: Grouped similar target types
• Weapon Clusters: Categorized weapon usage patterns

Predictive Model Performance

• Casualty Prediction: Achieved significant accuracy in predicting casualties
• Group Classification: Successfully classified responsible groups
• Feature Importance: Identified key predictive variables
• Model Validation: Cross-validated results for reliability

Text Mining Insights

• Common Themes: Identified recurring motives and justifications
• Group Communications: Analyzed group statements and propaganda
• Target Descriptions: Understanding of target selection criteria
• Weapon Descriptions: Analysis of weapon acquisition and usage

🎥 Visualizations and Media

Animated Visualizations

• Attack Types Over Time: Evolution of attack methodologies
• Geographic Spread: Global distribution of terrorist activities
• Casualty Trends: Changes in attack lethality over time
• Group Activity: Rise and fall of terrorist organizations

Interactive Dashboards

• Regional Analysis: Interactive exploration by geographic region
• Temporal Analysis: Time-based filtering and analysis
• Group Comparison: Side-by-side group behavior analysis
• Attack Type Distribution: Interactive categorization tools

👥 Contributors

This project was developed collaboratively by multiple contributors from the TUM Data Mining Lab:

Individual Contributions

• Data Processing: Collaborative effort on data cleaning and preprocessing
• Analysis Development: Joint work on statistical analysis and machine learning
• Visualization Creation: Team effort on charts, graphs, and animations
• Model Development: Collaborative approach to predictive modeling
• Documentation: Shared responsibility for code documentation and analysis

🚀 Getting Started

Prerequisites

• Python: Version 3.8+ with data science libraries
• R: Version 4.0+ with required packages
• Jupyter Notebook: For interactive analysis
• Git: Version control

Installation

# Clone the repository
git clone <repository-url>
cd global-terrorism-analysis

# Create conda environment
conda env create -f environment.yml
conda activate terrorism-analysis

# Or use pip
pip install -r requirements.txt

# Install R packages
R -e "install.packages(c('dplyr', 'tm', 'wordcloud', 'openxlsx', 'ggplot2', 'plotly'))"

Running the Analysis

# Activate environment
conda activate terrorism-analysis

# Start Jupyter notebook
jupyter notebook

# Run data processing
python scripts/data_processing.py

# Run specific analysis
jupyter notebook notebooks/01_data_exploration/missing_data_analysis.ipynb

📚 Methodology Details

Data Preprocessing

1. Data Loading: Excel file parsing and initial validation
2. Missing Data Handling: Systematic approach to missing values
3. Feature Engineering: Creation of derived variables
4. Data Validation: Quality checks and consistency verification

Statistical Analysis

1. Descriptive Statistics: Comprehensive data summaries
2. Correlation Analysis: Relationship identification between variables
3. Distribution Analysis: Understanding data distributions
4. Outlier Detection: Identification and handling of anomalies

Machine Learning Pipeline

1. Data Splitting: Training, validation, and test sets
2. Feature Selection: Identification of predictive variables
3. Model Training: Multiple algorithm implementation
4. Model Evaluation: Performance assessment and comparison
5. Hyperparameter Tuning: Optimization of model parameters

🔍 Research Applications

Academic Research

• Terrorism Studies: Understanding terrorist behavior patterns
• Security Analysis: Identifying threat patterns and trends
• Policy Research: Informing counter-terrorism strategies
• Criminology: Understanding criminal organization behavior

Practical Applications

• Security Planning: Risk assessment and mitigation strategies
• Intelligence Analysis: Pattern recognition and threat identification
• Resource Allocation: Optimizing security resource deployment
• Early Warning Systems: Predictive threat assessment

📄 Data Sources and References

Primary Data Source

• Global Terrorism Database (GTD): National Consortium for the Study of Terrorism and Responses to Terrorism (START)
• Codebook: Comprehensive documentation of all variables and coding schemes
• Data Quality: Regular updates and validation by START researchers

Additional Data Sources

• Country Data: Population and economic indicators
• Fuel Data: Energy and resource information
• Income Data: Economic development metrics

📄 License

This project is for academic research purposes. Please cite appropriately when using any findings or methodologies from this work.

---

This repository represents a comprehensive analysis of global terrorism data, combining statistical methods, machine learning techniques, and advanced visualization to understand patterns and trends in terrorist activities worldwide.