What makes a tiger so strong is that it lacks humanity
Advanced audio analysis framework powered by thermodynamic gas molecular processing for real-time meaning synthesis in audio analysis. Designed specifically for electronic music with focus on neurofunk and drum & bass genres, Heihachi uses gas molecular equilibrium restoration to understand audio content without pattern storage, achieving unprecedented efficiency and consciousness-aware audio processing capabilities.
Heihachi employs thermodynamic gas molecular processing to understand audio content through real-time equilibrium restoration rather than pattern matching. Audio input creates perturbations in neural gas molecular ensembles, and meaning emerges through the pathway taken to restore equilibrium - eliminating the need for template storage while providing direct emotional and consciousness state analysis.
Based on thermodynamic principles and consciousness research, audio perception operates like gas molecules in a system - input disturbances create perturbations, and meaning is synthesized through the restoration pathway back to equilibrium. Our system leverages this understanding through:
- Real-Time Equilibrium Restoration: Audio input perturbs gas molecular ensembles, creating unique restoration pathways that represent meaning
- Minimum Variance Synthesis: The system finds the restoration pathway requiring minimum thermodynamic variance, which naturally corresponds to the most probable meaning
- Direct Emotional Prediction: Gas molecular states directly map to emotional responses (valence, arousal, tension, flow) without intermediate processing
- Consciousness State Tracking: Monitor user engagement and comprehension through gas molecular equilibrium patterns
- Zero Storage Requirements: No pattern templates needed - meaning synthesized fresh from equilibrium restoration every time
Heihachi's gas molecular system naturally provides consciousness modeling without external dependencies. The thermodynamic equilibrium states directly correspond to consciousness patterns, enabling:
Direct Benefits:
- Unified Processing: Gas molecular equilibrium handles both audio analysis and consciousness modeling in one system
- Real-Time Emotional Response: Direct mapping from molecular states to emotional coordinates (valence, arousal, tension, flow)
- User State Tracking: Monitor engagement and comprehension through equilibrium restoration patterns
- Streamlined Architecture: No external probabilistic reasoning systems needed - everything emerges from thermodynamic principles
- Overview
- ⚛️ Gas Molecular Audio Processing
- 🧠 Consciousness-Aware Processing
- 🦀 Rust-Powered Architecture
- Features
- Installation
- Usage
- Gas Molecular Processing Usage
- Real-Time Emotional Analysis
- Theoretical Foundation
- Core Components
- REST API
- HuggingFace Integration
- Experimental Results
- Performance Optimizations
- Applications
- Future Directions
- License
- Citation
Heihachi's approach represents a fundamental shift in audio analysis - instead of comparing audio against stored patterns, the system synthesizes meaning in real-time through thermodynamic equilibrium restoration. Audio input creates perturbations in gas molecular ensembles, and the meaning emerges naturally from the restoration pathway.
Key Components:
- Gas Molecular Ensemble: Neural gas molecules with thermodynamic properties that respond to audio input
- Perturbation Analysis: Audio creates forces that disturb the molecular equilibrium state
- Equilibrium Restoration: The system finds the minimum variance pathway back to equilibrium
- Meaning Synthesis: The restoration pathway itself represents the audio's meaning and emotional content
- Audio Input: Raw audio data enters the gas molecular processing system
- Molecular Perturbation: Audio frequencies create forces that disturb gas molecular positions and velocities
- Equilibrium Restoration: The system calculates the pathway requiring minimum thermodynamic variance to restore equilibrium
- Direct Meaning Extraction: The restoration pathway directly encodes the audio's meaning, emotional content, and user impact
- Real-Time Response: Emotional coordinates (valence, arousal, tension, flow) are directly read from the molecular state
This approach is based on the principle that consciousness operates like a thermodynamic system - perturbations create meaning through the specific pathway taken to restore equilibrium, eliminating the need for pattern storage while providing unprecedented insight into audio content and user emotional response.
Heihachi uses a unified gas molecular processing architecture that eliminates complex dependencies and external reasoning systems. All audio analysis, consciousness modeling, and emotional prediction happen directly within the thermodynamic framework.
The gas molecular system provides:
- Unified Framework: Single system handles audio analysis, consciousness modeling, and emotional prediction
- Real-Time Processing: Sub-50ms latency for all operations through thermodynamic optimization
- Zero Storage: No pattern databases or templates needed - meaning synthesized fresh every time
- Direct Emotional Mapping: Molecular states directly correspond to emotional coordinates
- Consciousness Tracking: User engagement and comprehension monitoring through equilibrium patterns
Processing Optimization:
- Audio Analysis: <20ms through equilibrium restoration
- Emotional Prediction: <5ms direct molecular state reading
- Consciousness Tracking: <10ms equilibrium pattern analysis
- End-to-End Latency: <35ms total processing time
Architecture Advantages:
- Simplified Deployment: No external dependencies or complex integrations
- Reduced Memory: 10³-10⁵× reduction in storage requirements
- Linear Scaling: Performance scales linearly with processing load
- Real-Time Guarantee: Consistent sub-50ms response times
Heihachi's Rust backend is optimized specifically for gas molecular processing:
Performance Benefits:
- 15-25x speed improvements through thermodynamic optimization
- Memory safety with zero-cost gas molecular abstractions
- Parallel processing for large molecular ensemble calculations
- Real-time capabilities for live equilibrium restoration
Architecture:
- Rust Core: Gas molecular physics, equilibrium restoration, thermodynamic calculations
- Python Interface: PyO3 bindings for audio analysis and consciousness modeling
- Web Interface: Real-time gas molecular visualization and processing monitoring
- REST API: Unified access to gas molecular processing and emotional analysis
The gas molecular architecture maximizes Rust's strengths:
- Rust: Thermodynamic calculations, molecular physics, real-time equilibrium restoration
- Python: Audio analysis wrapper, consciousness modeling interface, visualization tools
- TypeScript: Gas molecular state visualization, real-time processing monitoring
Heihachi implements novel approaches to audio analysis by combining neurological models of rhythm processing with advanced signal processing techniques. The system is built upon established neuroscientific research demonstrating that humans possess an inherent ability to synchronize motor responses with external rhythmic stimuli. This framework provides high-performance analysis for:
- Detailed drum pattern recognition and visualization
- Bass sound design decomposition
- Component separation and analysis
- Comprehensive visualization tools
- Neural-based feature extraction
- Memory-optimized processing for large files
- ⚛️ Gas Molecular Processing: Real-time audio meaning synthesis through thermodynamic equilibrium restoration
- 🧠 Direct Consciousness Modeling: Built-in emotional prediction and user state tracking without external dependencies
- 🦀 Rust-Powered Performance: 15-25x speed improvements through thermodynamic optimization
- ⚡ Sub-35ms Processing: Ultra-low latency for real-time audio analysis and emotional response
- 🎯 Zero Storage Architecture: No pattern templates or databases - meaning synthesized fresh every time
- 📊 Direct Emotional Mapping: Real-time valence, arousal, tension, and flow prediction from molecular states
- High-performance audio file processing with thermodynamic efficiency
- Batch processing for large electronic music datasets
- Memory optimization through elimination of pattern storage
- Parallel processing for gas molecular ensemble calculations
- Real-time gas molecular visualization and processing monitoring
- Interactive consciousness state exploration and emotional response tracking
- Progress tracking for equilibrium restoration processes
- Export options for molecular states and emotional predictions
- Comprehensive CLI with gas molecular processing commands
- HuggingFace integration optimized with thermodynamic preprocessing
- Consciousness State Validation: Real-time user engagement and comprehension monitoring
- Unified Processing Architecture: Single system handles audio analysis, consciousness modeling, and emotional prediction
# Clone the repository
git clone https://github.com/yourusername/heihachi.git
cd heihachi
# Run the setup script
python scripts/setup.pyThe setup script supports several options:
--install-dir DIR Installation directory
--dev Install development dependencies
--no-gpu Skip GPU acceleration dependencies
--no-interactive Skip interactive mode dependencies
--shell-completion Install shell completion scripts
--no-confirm Skip confirmation prompts
--venv Create and use a virtual environment
--venv-dir DIR Virtual environment directory (default: .venv)
If you prefer to install manually:
# Create and activate virtual environment (optional)
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Install the package
pip install -e .# Process a single audio file
heihachi process audio.wav --output results/
# Process a directory of audio files
heihachi process audio_dir/ --output results/
# Batch processing with different configurations
heihachi batch audio_dir/ --config configs/performance.yaml# Start interactive command-line explorer with processed results
heihachi interactive --results-dir results/
# Start web-based interactive explorer
heihachi interactive --web --results-dir results/
# Compare multiple results with interactive explorer
heihachi compare results1/ results2/
# Show only progress demo
heihachi demo --progress-demo# Export results to different formats
heihachi export results/ --format json
heihachi export results/ --format csv
heihachi export results/ --format markdownThe basic command structure is:
python -m src.main [input_file] [options]Where [input_file] can be either a single audio file or a directory containing multiple audio files.
| Option | Description | Default |
|---|---|---|
input_file |
Path to audio file or directory (required) | - |
-c, --config |
Path to configuration file | ../configs/default.yaml |
-o, --output |
Path to output directory | ../results |
--cache-dir |
Path to cache directory | ../cache |
-v, --verbose |
Enable verbose logging | False |
# Process a single audio file
python -m src.main /path/to/track.wav
# Process an entire directory of audio files
python -m src.main /path/to/audio/folder
# Use a custom configuration file
python -m src.main /path/to/track.wav -c /path/to/custom_config.yaml
# Specify custom output directory
python -m src.main /path/to/track.wav -o /path/to/custom_output
# Enable verbose logging
python -m src.main /path/to/track.wav -vAfter processing, the results are saved to the output directory (default: ../results). For each audio file, the following is generated:
- Analysis data: JSON files containing detailed analysis results
- Visualizations: Graphs and plots showing various aspects of the audio analysis
- Summary report: Overview of the key findings and detected patterns
# Start gas molecular audio analysis
heihachi gas-analyze audio.wav --real-time
# Process with consciousness tracking
heihachi gas-analyze audio.wav --consciousness-tracking --emotional-prediction
# Batch processing with gas molecular optimization
heihachi gas-batch audio_directory/ --molecular-ensemble-size 1000- Audio Input: Feed audio data into the gas molecular processing system
- Molecular Perturbation: System calculates how audio disturbs the gas molecular equilibrium
- Equilibrium Restoration: Find the minimum variance pathway back to equilibrium
- Meaning Extraction: The restoration pathway directly represents the audio's meaning
- Emotional Prediction: Read emotional coordinates directly from molecular states
from heihachi.gas_molecular import GasMolecularProcessor, ConsciousnessTracker
# Initialize gas molecular system
gas_processor = GasMolecularProcessor(ensemble_size=1000)
# Process audio through gas molecular system
molecular_state = gas_processor.process_audio("audio.wav")
# Get equilibrium restoration pathway
restoration_path = molecular_state.restore_equilibrium()
# Extract meaning and emotional response
meaning = restoration_path.extract_meaning()
emotions = restoration_path.get_emotional_response()
# Track user consciousness state
consciousness = ConsciousnessTracker()
user_state = consciousness.track_state(molecular_state)
print(f"Emotional Response: valence={emotions.valence}, arousal={emotions.arousal}")
print(f"User Engagement: {user_state.engagement_level}")Molecular State Mapping:
- Valence: Positive/negative emotional tone from molecular equilibrium patterns
- Arousal: Energy level from molecular movement and perturbation strength
- Tension: Stress/relaxation from variance in equilibrium restoration
- Flow State: Processing smoothness from restoration pathway efficiency
- Engagement: User attention level from consciousness state patterns
- Comprehension: Understanding depth from equilibrium stability
Direct Processing Benefits:
- No Training Required: Works immediately without machine learning training
- Universal Application: Same system works for any audio content
- Real-Time Response: Sub-35ms emotional prediction
- Memory Efficient: No pattern storage - fresh synthesis every time
- Consciousness Aware: Built-in user state tracking and engagement monitoring
# Start Heihachi with optimized gas molecular processing
heihachi start --gas-molecular --ensemble-size 1000
# Start with custom molecular configuration
heihachi start --gas-molecular --config configs/gas_molecular.yaml
# Development mode with real-time molecular visualization
heihachi dev --molecular-visualization --real-time-emotional-trackingfrom heihachi.gas_molecular import GasMolecularProcessor, EmotionalStateTracker
# Initialize gas molecular system
gas_processor = GasMolecularProcessor(ensemble_size=1000)
emotion_tracker = EmotionalStateTracker()
# Process audio through gas molecular system
molecular_state = gas_processor.process_audio("audio.wav")
# Get equilibrium restoration pathway
restoration_path = molecular_state.restore_equilibrium()
# Extract meaning and emotional response directly from molecular state
meaning = restoration_path.extract_meaning()
emotional_state = restoration_path.get_emotional_coordinates()
# Track user consciousness state through molecular patterns
user_consciousness = emotion_tracker.track_consciousness(molecular_state)
# Generate consciousness-informed analysis results
analysis_results = gas_processor.generate_analysis(
molecular_state=molecular_state,
restoration_path=restoration_path,
consciousness_state=user_consciousness
)Real-Time Consciousness Monitoring:
# Monitor consciousness emergence during gas molecular processing
consciousness_stream = gas_processor.stream_consciousness_analysis()
for molecular_state, consciousness_level in consciousness_stream:
if consciousness_level > 0.7: # High consciousness threshold
# Trigger enhanced emotional analysis
enhanced_emotions = emotion_tracker.enhanced_emotional_analysis(
molecular_state, consciousness_level
)
gas_processor.apply_consciousness_enhancement(enhanced_emotions)Thermodynamic State Analysis:
# Access molecular thermodynamic states
thermo_analysis = gas_processor.thermodynamic_analysis(audio_input)
# Perturbation analysis
perturbation_state = thermo_analysis.calculate_perturbations(audio_input)
# Equilibrium restoration pathway
restoration_state = thermo_analysis.find_minimum_variance_pathway(
perturbation_state
)
# Direct meaning extraction from pathway
meaning_coordinates = thermo_analysis.extract_meaning_coordinates(
restoration_state
)Emotional Response Quantification:
# Get precise emotional coordinates from molecular states
emotional_analysis = gas_processor.quantify_emotions(molecular_state)
print(f"Valence (positive/negative): {emotional_analysis.valence}")
print(f"Arousal (energy level): {emotional_analysis.arousal}")
print(f"Tension (stress/relaxation): {emotional_analysis.tension}")
print(f"Flow State (processing smoothness): {emotional_analysis.flow}")
print(f"User Engagement: {emotional_analysis.engagement}")
print(f"Comprehension Level: {emotional_analysis.comprehension}")# Monitor gas molecular processing performance
performance_stats = gas_processor.get_performance_stats()
print(f"Molecular Perturbation Analysis: {performance_stats.perturbation_ms}ms")
print(f"Equilibrium Restoration: {performance_stats.restoration_ms}ms")
print(f"Meaning Synthesis: {performance_stats.synthesis_ms}ms")
print(f"Emotional Extraction: {performance_stats.emotional_ms}ms")
print(f"Total Processing Time: {performance_stats.total_ms}ms")Gas Molecular Processing Settings:
# configs/gas_molecular.yaml
gas_molecular:
ensemble_size: 1000
temperature: 298.15 # Room temperature in Kelvin
equilibrium_threshold: 0.001
# Processing settings
processing:
perturbation_sensitivity: 0.1
restoration_algorithm: "minimum_variance"
real_time_processing: true
consciousness_tracking: true
# Performance settings
performance:
max_molecular_calculations: 50
batch_processing: true
optimization_enabled: true
# Emotional mapping settings
emotional:
valence_sensitivity: 0.2
arousal_sensitivity: 0.15
tension_sensitivity: 0.1
flow_sensitivity: 0.05The framework is built upon established neuroscientific research demonstrating that humans possess an inherent ability to synchronize motor responses with external rhythmic stimuli. This phenomenon, known as beat-based timing, involves complex interactions between auditory and motor systems in the brain.
Key neural mechanisms include:
-
Beat-based Timing Networks
- Basal ganglia-thalamocortical circuits
- Supplementary motor area (SMA)
- Premotor cortex (PMC)
-
Temporal Processing Systems
- Duration-based timing mechanisms
- Beat-based timing mechanisms
- Motor-auditory feedback loops
Research has shown that low-frequency neural oscillations from motor planning areas guide auditory sampling, expressed through coherence measures:
Where:
-
$C_{xy}(f)$ represents coherence at frequency$f$ -
$S_{xy}(f)$ is the cross-spectral density -
$S_{xx}(f)$ and$S_{yy}(f)$ are auto-spectral densities
In addition to the coherence measures, we utilize several key mathematical formulas:
- Spectral Decomposition: For analyzing sub-bass and Reese bass components:
- Groove Pattern Analysis: For microtiming deviations:
- Amen Break Detection: Pattern matching score:
- Reese Bass Analysis: For analyzing modulation and phase relationships:
- Transition Detection: For identifying mix points and transitions:
- Similarity Computation: For comparing audio segments:
- Segment Clustering: Using DBSCAN with adaptive distance:
- Automated drum pattern recognition
- Groove quantification
- Microtiming analysis
- Syncopation detection
- Multi-band decomposition
- Harmonic tracking
- Timbral feature extraction
- Sub-bass characterization
- Sound source separation
- Transformation detection
- Energy distribution analysis
- Component relationship mapping
- Pattern matching and variation detection
- Transformation identification
- Groove characteristic extraction
- VIP/Dubplate classification
- Robust onset envelope extraction with fault tolerance
- Dynamic time warping with optimal window functions
- Neurofunk-specific component separation
- Bass sound design analysis
- Effect chain detection
- Temporal structure analysis
- Multi-band similarity computation
- Transformation-aware comparison
- Groove-based alignment
- Confidence scoring
- Multi-band onset detection
- Adaptive thresholding
- Feature-based peak classification
- Confidence scoring
- Pattern-based segmentation
- Hierarchical clustering
- Relationship analysis
- Transition detection
- Mix point identification
- Blend type classification
- Energy flow analysis
- Structure boundary detection
- Empty audio detection and graceful recovery
- Sample rate validation and default fallbacks
- Signal integrity verification
- Automatic recovery mechanisms
- Streaming processing for large files
- Resource optimization and monitoring
- Garbage collection optimization
- Chunked processing of large audio files
- Proper window functions to eliminate spectral leakage
- Normalized processing paths
- Adaptive parameters based on content
- Fault-tolerant alignment algorithms
Heihachi provides a comprehensive REST API for integrating audio analysis capabilities into web applications, mobile apps, and other systems. The API supports both synchronous and asynchronous processing, making it suitable for both real-time and batch processing scenarios.
# Install API dependencies
pip install flask flask-cors flask-limiter
# Start the API server
python api_server.py --host 0.0.0.0 --port 5000
# Or with custom configuration
python api_server.py --production --config-path configs/production.yaml| Endpoint | Method | Description | Rate Limit |
|---|---|---|---|
/health |
GET | Health check | None |
/api |
GET | API information and endpoints | None |
/api/v1/analyze |
POST | Full audio analysis | 10/min |
/api/v1/features |
POST | Extract audio features | 20/min |
/api/v1/beats |
POST | Detect beats and tempo | 20/min |
/api/v1/drums |
POST | Analyze drum patterns | 10/min |
/api/v1/stems |
POST | Separate audio stems | 5/min |
/api/v1/semantic/analyze |
POST | Semantic analysis with emotion mapping | 10/min |
/api/v1/semantic/search |
POST | Search indexed tracks semantically | 20/min |
/api/v1/semantic/emotions |
POST | Extract emotional features only | 20/min |
/api/v1/semantic/text-analysis |
POST | Analyze text descriptions | 30/min |
/api/v1/semantic/stats |
GET | Get semantic search statistics | None |
/api/v1/batch-analyze |
POST | Batch process multiple files | 2/min |
/api/v1/jobs/{id} |
GET | Get job status and results | None |
/api/v1/jobs |
GET | List all jobs (paginated) | None |
Synchronous Processing:
curl -X POST http://localhost:5000/api/v1/analyze \
-F "[email protected]" \
-F "config=configs/default.yaml"Asynchronous Processing:
curl -X POST http://localhost:5000/api/v1/analyze \
-F "[email protected]" \
-F "async=true"curl -X POST http://localhost:5000/api/v1/features \
-F "[email protected]" \
-F "model=microsoft/BEATs-base"curl -X POST http://localhost:5000/api/v1/beats \
-F "[email protected]"curl -X POST http://localhost:5000/api/v1/drums \
-F "[email protected]" \
-F "visualize=true"curl -X POST http://localhost:5000/api/v1/stems \
-F "[email protected]" \
-F "save_stems=true" \
-F "format=wav"curl -X POST http://localhost:5000/api/v1/batch-analyze \
-F "[email protected]" \
-F "[email protected]" \
-F "[email protected]"curl -X POST http://localhost:5000/api/v1/semantic/analyze \
-F "[email protected]" \
-F "include_emotions=true" \
-F "index_for_search=true" \
-F "title=Track Title" \
-F "artist=Artist Name"curl -X POST http://localhost:5000/api/v1/semantic/emotions \
-F "[email protected]"curl -X POST http://localhost:5000/api/v1/semantic/search \
-H "Content-Type: application/json" \
-d '{"query": "dark aggressive neurofunk with heavy bass", "top_k": 5}'curl -X POST http://localhost:5000/api/v1/semantic/text-analysis \
-H "Content-Type: application/json" \
-d '{"text": "This track has an amazing dark atmosphere with aggressive drums"}'curl http://localhost:5000/api/v1/jobs/550e8400-e29b-41d4-a716-446655440000import requests
import json
# API base URL
base_url = "http://localhost:5000/api/v1"
# Upload and analyze audio file
def analyze_audio(file_path, async_processing=False):
url = f"{base_url}/analyze"
with open(file_path, 'rb') as f:
files = {'file': f}
data = {'async': str(async_processing).lower()}
response = requests.post(url, files=files, data=data)
return response.json()
# Extract features
def extract_features(file_path, model='microsoft/BEATs-base'):
url = f"{base_url}/features"
with open(file_path, 'rb') as f:
files = {'file': f}
data = {'model': model}
response = requests.post(url, files=files, data=data)
return response.json()
# Check job status
def get_job_status(job_id):
url = f"{base_url}/jobs/{job_id}"
response = requests.get(url)
return response.json()
# Semantic analysis with emotions
def semantic_analyze(file_path, include_emotions=True, index_for_search=False, title=None, artist=None):
url = f"{base_url}/semantic/analyze"
with open(file_path, 'rb') as f:
files = {'file': f}
data = {
'include_emotions': str(include_emotions).lower(),
'index_for_search': str(index_for_search).lower()
}
if title:
data['title'] = title
if artist:
data['artist'] = artist
response = requests.post(url, files=files, data=data)
return response.json()
# Semantic search
def semantic_search(query, top_k=5, enhance_query=True):
url = f"{base_url}/semantic/search"
data = {
'query': query,
'top_k': top_k,
'enhance_query': enhance_query
}
response = requests.post(url, json=data)
return response.json()
# Extract emotions only
def extract_emotions(file_path):
url = f"{base_url}/semantic/emotions"
with open(file_path, 'rb') as f:
files = {'file': f}
response = requests.post(url, files=files)
return response.json()
# Example usage
if __name__ == "__main__":
# Synchronous analysis
result = analyze_audio("track.wav", async_processing=False)
print("Analysis result:", json.dumps(result, indent=2))
# Semantic analysis with emotion mapping
semantic_result = semantic_analyze("track.wav", include_emotions=True,
index_for_search=True, title="My Track", artist="My Artist")
print("Emotions:", semantic_result['semantic_analysis']['emotions'])
# Extract just emotions
emotions = extract_emotions("track.wav")
print("Emotional analysis:", emotions['emotions'])
print("Dominant emotion:", emotions['summary']['dominant_emotion'])
# Search for similar tracks
search_results = semantic_search("dark aggressive neurofunk with heavy bass")
print("Search results:", search_results['results'])
# Asynchronous analysis
job = analyze_audio("long_track.wav", async_processing=True)
job_id = job['job_id']
print(f"Job created: {job_id}")
# Poll job status
import time
while True:
status = get_job_status(job_id)
print(f"Job status: {status['status']}")
if status['status'] in ['completed', 'failed']:
break
time.sleep(5) # Wait 5 seconds before checking againconst FormData = require('form-data');
const fetch = require('node-fetch');
const fs = require('fs');
const API_BASE = 'http://localhost:5000/api/v1';
// Analyze audio file
async function analyzeAudio(filePath, asyncProcessing = false) {
const form = new FormData();
form.append('file', fs.createReadStream(filePath));
form.append('async', asyncProcessing.toString());
const response = await fetch(`${API_BASE}/analyze`, {
method: 'POST',
body: form
});
return await response.json();
}
// Extract features
async function extractFeatures(filePath, model = 'microsoft/BEATs-base') {
const form = new FormData();
form.append('file', fs.createReadStream(filePath));
form.append('model', model);
const response = await fetch(`${API_BASE}/features`, {
method: 'POST',
body: form
});
return await response.json();
}
// Check job status
async function getJobStatus(jobId) {
const response = await fetch(`${API_BASE}/jobs/${jobId}`);
return await response.json();
}
// Example usage
(async () => {
try {
// Extract features
const features = await extractFeatures('track.mp3');
console.log('Features:', JSON.stringify(features, null, 2));
// Start async analysis
const job = await analyzeAudio('track.wav', true);
console.log('Job started:', job.job_id);
// Poll job status
let status;
do {
await new Promise(resolve => setTimeout(resolve, 5000)); // Wait 5 seconds
status = await getJobStatus(job.job_id);
console.log('Job status:', status.status);
} while (!['completed', 'failed'].includes(status.status));
if (status.status === 'completed') {
console.log('Results:', JSON.stringify(status.results, null, 2));
}
} catch (error) {
console.error('Error:', error);
}
})();All API endpoints return JSON responses with the following structure:
Success Response:
{
"status": "completed",
"results": {
// Analysis results vary by endpoint
},
"processing_time": 45.2
}Async Job Response:
{
"job_id": "550e8400-e29b-41d4-a716-446655440000",
"status": "processing",
"message": "Analysis started. Use /api/v1/jobs/{job_id} to check status."
}Error Response:
{
"error": "File too large",
"message": "Maximum file size is 500MB"
}Configure the API using environment variables or command-line arguments:
| Variable | Default | Description |
|---|---|---|
PORT |
5000 | Server port |
MAX_FILE_SIZE |
500MB | Maximum upload file size |
PROCESSING_TIMEOUT |
1800 | Processing timeout in seconds |
MAX_CONCURRENT_JOBS |
5 | Maximum concurrent processing jobs |
HUGGINGFACE_API_KEY |
"" | HuggingFace API key for gated models |
UPLOAD_FOLDER |
uploads | Directory for uploaded files |
RESULTS_FOLDER |
results | Directory for results |
FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
EXPOSE 5000
CMD ["python", "api_server.py", "--production", "--host", "0.0.0.0"]# Using gunicorn for production
pip install gunicorn
# Start with gunicorn
gunicorn -w 4 -b 0.0.0.0:5000 "src.api.app:create_app()"
# Or with custom configuration
gunicorn -w 4 -b 0.0.0.0:5000 --timeout 1800 "src.api.app:create_app()"Heihachi integrates specialized AI models from Hugging Face, enabling advanced neural processing of audio using state-of-the-art models. This integration follows a structured implementation approach with models carefully selected for electronic music analysis tasks.
The following specialized audio analysis models are available:
| Category | Model Type | Default Model | Description | Priority |
|---|---|---|---|---|
| Core Feature Extraction | Generic spectral + temporal embeddings | microsoft/BEATs | Bidirectional ViT-style encoder trained with acoustic tokenisers; provides 768-d latent at ~20 ms hop | High |
| Robust speech & non-speech features | openai/whisper-large-v3 | Trained on >5M hours; encoder provides 1280-d features tracking energy, voicing & language | High | |
| Audio Source Separation | Stem isolation | Demucs v4 | Returns 4-stem or 6-stem tensors for component-level analysis | High |
| Rhythm Analysis | Beat / down-beat tracking | Beat-Transformer | Dilated self-attention encoder with F-measure ~0.86 | High |
| Low-latency beat-tracking | BEAST | 50 ms latency, causal attention; ideal for real-time DJ analysis | Medium | |
| Drum-onset / kit piece ID | DunnBC22/wav2vec2-base-Drum_Kit_Sounds | Fine-tuned on kick/snare/tom/overhead labels | Medium | |
| Multimodal & Similarity | Multimodal similarity / tagging | laion/clap-htsat-fused | Query with free-text and compute cosine similarity on 512-d embeddings | Medium |
| Zero-shot tag & prompt embedding | UniMus/OpenJMLA | Score arbitrary tag strings for effect-chain heuristics | Medium | |
| Future Extensions | Audio captioning | slseanwu/beats-conformer-bart-audio-captioner | Produces textual descriptions per segment | Low |
| Similarity retrieval UI | CLAP embeddings + FAISS | Index embeddings and expose nearest-neighbor search | Low |
Configure HuggingFace models in configs/huggingface.yaml:
# Enable/disable HuggingFace integration
enabled: true
# API key for accessing HuggingFace models (leave empty to use public models only)
api_key: ""
# Specialized model settings
feature_extraction:
enabled: true
model: "microsoft/BEATs-base"
beat_detection:
enabled: true
model: "nicolaus625/cmi"
# Additional models (disabled by default to save resources)
drum_sound_analysis:
enabled: false
model: "DunnBC22/wav2vec2-base-Drum_Kit_Sounds"
similarity:
enabled: false
model: "laion/clap-htsat-fused"
# See configs/huggingface.yaml for all available options# Extract features
python -m src.main hf extract path/to/audio.mp3 --output features.json
# Separate stems
python -m src.main hf separate path/to/audio.mp3 --output-dir ./stems --save-stems
# Detect beats
python -m src.main hf beats path/to/audio.mp3 --output beats.json
# Analyze drums
python -m src.main hf analyze-drums audio.wav --visualize
# Other available commands
python -m src.main hf drum-patterns audio.wav --mode pattern
python -m src.main hf tag audio.wav --categories "genre:techno,house,ambient"
python -m src.main hf caption audio.wav --mix-notes
python -m src.main hf similarity audio.wav --mode timestamps --query "bass drop"
python -m src.main hf realtime-beats --file --input audio.wavfrom heihachi.huggingface import FeatureExtractor, StemSeparator, BeatDetector
# Extract features
extractor = FeatureExtractor(model="microsoft/BEATs-base")
features = extractor.extract(audio_path="track.mp3")
# Separate stems
separator = StemSeparator()
stems = separator.separate(audio_path="track.mp3")
drums = stems["drums"]
bass = stems["bass"]
# Detect beats
detector = BeatDetector()
beats = detector.detect(audio_path="track.mp3", visualize=True, output_path="beats.png")
print(f"Tempo: {beats['tempo']} BPM")This section presents visualization results from audio analysis examples processed through the Heihachi framework, demonstrating the capabilities of the system in extracting meaningful insights from audio data.
The following visualizations showcase the results from analyzing drum hits within a 33-minute electronic music mix. The analysis employs a multi-stage approach:
- Onset Detection: Using adaptive thresholding with spectral flux and phase deviation to identify percussion events
- Drum Classification: Neural network classification to categorize each detected hit
- Confidence Scoring: Model-based confidence estimation for each classification
- Temporal Analysis: Pattern recognition across the timeline of detected hits
The analysis identified 91,179 drum hits spanning approximately 33 minutes (1999.5 seconds) of audio. The percussion events were classified into five primary categories with the following distribution:
- Hi-hat: 26,530 hits (29.1%)
- Snare: 16,699 hits (18.3%)
- Tom: 16,635 hits (18.2%)
- Kick: 16,002 hits (17.6%)
- Cymbal: 15,313 hits (16.8%)
These classifications were derived using a specialized audio recognition model that separates and identifies percussion components based on their spectral and temporal characteristics.
The density plot reveals the distribution of drum hits over time, providing insight into the rhythmic structure and intensity variations throughout the mix. Notable observations include:
- Clear sections of varying percussion density, indicating track transitions and arrangement changes
- Consistent underlying beat patterns maintained throughout the mix
- Periodic intensity peaks corresponding to build-ups and drops in the arrangement
The heatmap visualization represents normalized hit density across time for each drum type, revealing:
- Structured patterns in kick and snare placement, typical of electronic dance music
- Variations in hi-hat and cymbal usage that correspond to energy shifts
- Clearly defined segments with distinct drum programming approaches
The timeline visualization provides a comprehensive view of all drum events plotted against time, allowing for detailed analysis of the rhythmic structure. Key observations from this temporal analysis include:
- Microtiming Variations: Subtle deviations from the quantized grid, particularly evident in hi-hats and snares, contribute to the human feel of the percussion
- Structural Markers: Clear delineation of musical sections visible through changes in drum event density and type distribution
- Layering Techniques: Overlapping drum hits at key points (e.g., stacked kick and cymbal events) to create impact moments
- Rhythmic Motifs: Recurring patterns of specific drum combinations that serve as stylistic identifiers throughout the mix
The temporal analysis employed statistical methods to identify:
- Event Clustering: Hierarchical clustering based on temporal proximity, velocity, and drum type
- Pattern Detection: N-gram analysis of drum sequences to identify common motifs
- Grid Alignment: Adaptive grid inference to determine underlying tempo and quantization
- Transition Detection: Change-point analysis to identify structural boundaries
These analytical methods reveal the sophisticated rhythmic programming underlying the seemingly straightforward electronic beat patterns, with calculated variation applied to create both consistency and interest.
The confidence metrics for the drum classification model demonstrate varying levels of certainty depending on the drum type:
| Drum Type | Avg. Confidence | Avg. Velocity |
|---|---|---|
| Tom | 0.385 | 1.816 |
| Snare | 0.381 | 1.337 |
| Kick | 0.370 | 0.589 |
| Cymbal | 0.284 | 1.962 |
| Hi-hat | 0.223 | 1.646 |
The confidence scores reflect the model's certainty in classification, with higher values for toms and snares suggesting these sounds have more distinctive spectral signatures. Meanwhile, velocity measurements indicate the relative energy of each hit, with cymbals and toms showing the highest average values.
The scatter plot visualization reveals the relationship between classification confidence and hit velocity across all percussion events. This analysis provides critical insights into the performance of the neural classification model:
-
Velocity-Confidence Correlation: The plot demonstrates a positive correlation between hit velocity and classification confidence for most drum types, particularly evident in the upper-right quadrant where high-velocity hits receive more confident classifications.
-
Type-Specific Clusters: Each percussion type forms distinct clusters in the confidence-velocity space, with:
- Kicks (blue): Concentrated in the low-velocity, medium-confidence region
- Snares (orange): Forming a broad distribution across medium velocities with varying confidence
- Toms (green): Creating a distinctive cluster in the high-velocity, high-confidence region
- Hi-hats (red): Showing the widest distribution, indicating greater variability in classification performance
- Cymbals (purple): Forming a more diffuse pattern at higher velocities with moderate confidence
-
Classification Challenges: The lower confidence regions (bottom half of the plot) indicate areas where the model experiences greater uncertainty, particularly:
- Low-velocity hits across all percussion types
- Overlapping spectral characteristics between similar percussion sounds (e.g., certain hi-hats and cymbals)
- Boundary cases where multiple drum types may be present simultaneously
-
Performance Insights: The density of points in different regions provides a robust evaluation metric for the classification model, revealing both strengths in distinctive percussion identification and challenges in boundary cases.
This visualization serves as a valuable tool for evaluating classification performance and identifying specific areas for model improvement in future iterations of the framework.
The drum hit analysis also generated an interactive HTML timeline that allows for detailed exploration of the percussion events. This visualization maps each drum hit across time with interactive tooltips displaying precise timing, confidence scores, and velocity information.
The interactive timeline is available at:
visualizations/drum_feature_analysis/interactive_timeline.html
To view the interactive timeline alongside the music:
- Open the interactive timeline HTML file in a browser
- In a separate browser tab, play the corresponding audio mix
- Synchronize playback position to explore the relationship between audio and detected drum events
The drum hit analysis pipeline employs several advanced techniques:
-
Onset Detection Algorithm: Utilizes a combination of spectral flux, high-frequency content (HFC), and complex domain methods to detect percussion events with high temporal precision (±5ms).
-
Neural Classification: Implements a specialized convolutional neural network trained on isolated drum samples to classify detected onsets into specific percussion categories.
-
Confidence Estimation: Employs softmax probability outputs from the classification model to assess classification reliability, with additional weighting based on signal-to-noise ratio and onset clarity.
-
Pattern Recognition: Applies a sliding-window approach with dynamic time warping (DTW) to identify recurring rhythmic patterns and variations.
-
Memory-Optimized Processing: Implements chunked processing with a sliding window approach to handle large audio files while maintaining consistent analysis quality.
The complete analysis was performed using the following command:
python -m src.main hf analyze-drums /path/to/mix.mp3 --visualizeCurrent limitations of the drum analysis include:
- Occasional misclassification between similar drum types (e.g., toms vs. snares)
- Limited ability to detect layered drum hits occurring simultaneously
- Reduced accuracy during segments with heavy processing effects
Future improvements will focus on:
- Enhanced separation of overlapping drum sounds
- Tempo-aware pattern recognition
- Integration with musical structure analysis
- Improved classification of electronic drum sounds and synthesized percussion
- Streaming processing for large files
- Efficient cache utilization
- GPU memory optimization
- Automatic garbage collection optimization
- Chunked loading for very large files
- Audio validation at each processing stage
- Multi-threaded feature extraction
- Batch processing capabilities
- Distributed analysis support
- Adaptive resource allocation
- Scalable parallel execution
- Compressed result storage
- Metadata indexing
- Version control for analysis results
- Simple, consistent path handling
- Track boundary detection
- Transition type classification
- Mix structure analysis
- Energy flow visualization
- Sound design deconstruction
- Arrangement analysis
- Effect chain detection
- Reference track comparison
- Similar track identification
- Style classification
- Groove pattern matching
- VIP/Dubplate detection
-
Enhanced Neural Processing
- Integration of deep learning models
- Real-time processing capabilities
- Adaptive threshold optimization
-
Extended Analysis Capabilities
- Additional genre support
- Extended effect detection
- Advanced pattern recognition
- Further error resilience improvements
-
Improved Visualization
- Interactive dashboards
- 3D visualization options
- Real-time visualization
- Error diagnostics visualization
This project is licensed under the MIT License - see the LICENSE file for details.
If you use this framework in your research, please cite:
@software{heihachi2024,
title = {Heihachi: Neural Processing of Electronic Music},
author = {Kundai Farai Sachikonye},
year = {2024},
url = {https://github.com/fullscreen-triangle/heihachi}
}