Merge pull request #187 from NotJoeMartinez/handle-bot-blocking-maybe-use-api

This pull request introduces several improvements and updates to the `yt-fts` project, focusing on troubleshooting documentation, type hinting, dependency updates, and Python compatibility. The most significant changes include adding a comprehensive troubleshooting guide for 403 errors, updating type hints across multiple functions for better code clarity, upgrading dependencies, and increasing the minimum required Python version.

### Randomizing User agent, retry method
added yt-dlp config options to retry downloads and randomize user agent

### Documentation Enhancements:
* Added a new file `docs/TROUBLESHOOTING_403.md` with detailed explanations of 403 errors, diagnosis tools, common solutions, advanced troubleshooting steps, and prevention tips. This includes example workflows and error message references for user guidance.

### Type Hinting Improvements:
* Updated type hints across functions in `src/yt_fts/config.py`, `src/yt_fts/db_utils.py`, and `src/yt_fts/export.py` to improve code readability and enforce stricter type checking. Examples include specifying return types (`str | None`, `list[tuple[int, str, str, str]]`) and parameter types (`channel_id: str`, `limit: int | None`). [[1]](diffhunk://#diff-6113cfecef0a78f531444ed85e58c8783626a2f7610b4e062a3ca6571989ff97L1-R8) [[2]](diffhunk://#diff-dc3d4af5ca8205a5ad2fecf0f00f98cf853640e148149333af296439c1a6cde2L13-R13) [[3]](diffhunk://#diff-03b0323cdf4466eba537f666efacf13a39ed4b56b0efdbb3da03625ce133964dL22-R22)

### Dependency Updates:
* Upgraded dependencies in `pyproject.toml`:
  - `openai` updated from `1.35.3` to `1.93.0`.
  - `chromadb` updated from `0.5.2` to `1.0.15`.

### Python Compatibility:
* Increased the minimum required Python version from `>=3.8` to `>=3.10` in `pyproject.toml` to leverage newer language features and maintain compatibility with updated dependencies.

### File Renaming:
* Renamed `yt_fts/config.py`, `yt_fts/db_utils.py`, and `yt_fts/export.py` to `src/yt_fts/config.py`, `src/yt_fts/db_utils.py`, and `src/yt_fts/export.py` respectively, reflecting a change in project structure. [[1]](diffhunk://#diff-6113cfecef0a78f531444ed85e58c8783626a2f7610b4e062a3ca6571989ff97L1-R8) [[2]](diffhunk://#diff-dc3d4af5ca8205a5ad2fecf0f00f98cf853640e148149333af296439c1a6cde2L13-R13) [[3]](diffhunk://#diff-03b0323cdf4466eba537f666efacf13a39ed4b56b0efdbb3da03625ce133964dL22-R22)

Author: NotJoeMartinez

.gitignore

@@ -174,4 +174,9 @@ UCYO_jab_esuFRV4b17AJtAw
 .ignore/
 tests/test_data/
 .idea
-*.sh
+*.sh
+
+# custom
+
+scratch/
+.env

docs/TROUBLESHOOTING_403.md

@@ -0,0 +1,157 @@
+# Troubleshooting 403 Errors
+
+## What are 403 Errors?
+
+403 Forbidden errors occur when YouTube blocks requests from your application. This typically happens due to:
+
+- **Rate limiting**: Too many requests in a short time period
+- **Missing authentication**: No cookies or session data
+- **Bot detection**: YouTube identifying automated requests
+- **IP blocking**: Your IP address has been temporarily blocked
+- **Geographic restrictions**: Content not available in your region
+
+## Quick Diagnosis
+
+Run the built-in diagnosis tool to identify the issue:
+
+```bash
+# Basic diagnosis
+yt-fts diagnose
+
+# Diagnosis with browser cookies
+yt-fts diagnose --cookies-from-browser chrome
+
+# Diagnosis with specific job count
+yt-fts diagnose -j 4
+```
+
+## Common Solutions
+
+### 1. Use Browser Cookies
+
+The most effective solution is to use cookies from your browser:
+
+```bash
+# Use Chrome cookies
+yt-fts download --cookies-from-browser chrome <channel_url>
+
+# Use Firefox cookies
+yt-fts download --cookies-from-browser firefox <channel_url>
+```
+
+**How to set up cookies:**
+1. Log into YouTube in your browser (Chrome or Firefox)
+2. Make sure you're logged in and can access the channel
+3. Run the download command with `--cookies-from-browser`
+
+### 2. Reduce Parallel Jobs
+
+High parallel job counts can trigger rate limiting:
+
+```bash
+# Use fewer parallel jobs
+yt-fts download -j 2 <channel_url>
+yt-fts download -j 4 <channel_url>
+
+# For very problematic channels, use just 1 job
+yt-fts download -j 1 <channel_url>
+```
+
+### 3. Wait Between Attempts
+
+If you're getting rate limited, wait a few minutes before trying again:
+
+```bash
+# Wait 5-10 minutes between attempts
+# Then try again with reduced jobs
+yt-fts download -j 2 --cookies-from-browser chrome <channel_url>
+```
+
+### 4. Check Channel Accessibility
+
+Some channels may be:
+- **Private**: Only accessible to subscribers
+- **Age-restricted**: Requires login and age verification
+- **Region-blocked**: Not available in your country
+
+Try accessing the channel in your browser first to verify it's publicly accessible.
+
+## Advanced Troubleshooting
+
+### Test Network Connectivity
+
+```bash
+# Test basic connectivity
+curl -I https://www.youtube.com
+
+# Test with custom user agent
+curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" https://www.youtube.com
+```
+
+### Update yt-dlp
+
+Ensure you have the latest version of yt-dlp:
+
+```bash
+pip install --upgrade yt-dlp
+```
+
+### Check for VPN/Proxy Issues
+
+If you're using a VPN or proxy:
+1. Try disabling it temporarily
+2. Switch to a different server/location
+3. Use a residential IP if possible
+
+### Monitor Rate Limits
+
+Watch for these error patterns:
+- **429 Too Many Requests**: Immediate rate limit
+- **403 Forbidden**: General blocking
+- **503 Service Unavailable**: Temporary server issues
+
+## Error Message Reference
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| `403 Forbidden` | General blocking | Use cookies, reduce jobs |
+| `429 Too Many Requests` | Rate limiting | Wait, reduce jobs |
+| `Video unavailable` | Private/restricted | Check channel access |
+| `Sign in to confirm your age` | Age restriction | Use logged-in cookies |
+
+## Prevention Tips
+
+1. **Always use browser cookies** for consistent access
+2. **Start with low job counts** (2-4) and increase gradually
+3. **Monitor for errors** and adjust accordingly
+4. **Don't run multiple instances** simultaneously
+5. **Respect rate limits** - wait between large downloads
+
+## Getting Help
+
+If you're still experiencing issues:
+
+1. Run the diagnosis tool: `yt-fts diagnose`
+2. Check the error messages for specific details
+3. Try the test script: `python test_403_diagnosis.py`
+4. Report issues with:
+   - Error messages
+   - Channel URL (if public)
+   - Your configuration (jobs, cookies, etc.)
+   - Diagnosis output
+
+## Example Workflow
+
+```bash
+# 1. Diagnose the issue
+yt-fts diagnose --cookies-from-browser chrome
+
+# 2. Try with cookies and low job count
+yt-fts download --cookies-from-browser chrome -j 2 <channel_url>
+
+# 3. If successful, gradually increase jobs
+yt-fts download --cookies-from-browser chrome -j 4 <channel_url>
+
+# 4. For large channels, consider breaking into smaller batches
+# Download in chunks with breaks between them
+``` 

pyproject.toml

@@ -7,22 +7,22 @@ name = "yt-fts"
 version = "0.1.60"
 description = "Search all of a YouTube channel from the command line"
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.10"
 license = { file = "LICENSE" }
 authors = [
     { name = "NotJoeMartinez", email = "notjoemartinez@protonmail.com" }
 ]
 keywords = ["youtube", "subtitles", "search"]
 classifiers = [
-    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
     "License :: OSI Approved :: The Unlicense (Unlicense)", 
     "Operating System :: OS Independent",
 ]
 
 dependencies = [
     "click==8.1.7",
-    "openai==1.35.3",
-    "chromadb==0.5.2",
+    "openai==1.93.0",
+    "chromadb==1.0.15",
     "requests>=2.32.2,<3",
     "rich==13.7.1",
     "sqlite-utils==3.36",

yt_fts/__init__.py src/yt_fts/__init__.pyyt_fts/__init__.py renamed to src/yt_fts/__init__.py

@@ -1,12 +1,11 @@
-
 import sys 
 import os
 
 import chromadb
 from chromadb.config import Settings
 
 
-def get_config_path():
+def get_config_path() -> str | None:
 
     platform = sys.platform
 
@@ -27,7 +26,7 @@ def get_config_path():
     return None
 
 
-def make_config_dir():
+def make_config_dir() -> str | None:
     platform = sys.platform
 
     try:
@@ -49,7 +48,7 @@ def make_config_dir():
         return None
 
 
-def get_db_path():
+def get_db_path() -> str:
     from .db_utils import make_db
     # make sure config path exists
     # if config path is none, make config path
@@ -92,7 +91,7 @@ def get_db_path():
     return "subtitles.db" 
 
 
-def get_or_make_chroma_path():
+def get_or_make_chroma_path() -> str:
 
     config_path = get_config_path()
 
@@ -112,7 +111,7 @@ def get_or_make_chroma_path():
         return chroma_path
 
 
-def get_chroma_client():
+def get_chroma_client() -> chromadb.PersistentClient:
     chroma_path = get_or_make_chroma_path()
     return chromadb.PersistentClient(path=chroma_path, 
                                      settings=Settings(anonymized_telemetry=False))