Add + Discover Sites: Full-page discovery feature with multi-source search#2027
Open
samuelclay wants to merge 83 commits intomainfrom
Open
Add + Discover Sites: Full-page discovery feature with multi-source search#2027samuelclay wants to merge 83 commits intomainfrom
samuelclay wants to merge 83 commits intomainfrom
Conversation
- Add full-page Add Site view with tabbed interface for discovering content - Support YouTube channel search via YouTube Data API - Support Reddit subreddit search via Reddit API - Add Google News feed builder with topics and custom keywords - Add Newsletters tab for Substack/newsletter discovery - Add Podcasts tab for podcast search via Podcast Index API - Add Trending tab showing time-windowed trending feeds - Add Categories tab with algorithmically clustered feed categories - Add sidebar header item "Add Site" above Reading Archive - Add URL routing for /add-site route - Include grid/list view toggle for search results - Fix Subscribe button CSS to prevent shrinking in flex layout - Add dark mode support for all Add Site components Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Rename sidebar header from "Add Site" to "Add + Discover" - Add discover section to quick add popover with source shortcuts - Quick links to YouTube, Reddit, Newsletters, Podcasts tabs - "Browse all sources" button opens full Add + Discover page - Clicking source buttons opens Add + Discover with that tab selected - Add CSS for discover section in popover (light and dark modes) - Support initial_tab option in AddSiteView for direct tab navigation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Change route from /add-site to /add - Consolidate add modal and full page routes (modal opens with ?url= param) - Fix sidebar header to match other folders (remove tall blue gradient) - Use subtle blue text and icon color instead of prominent button style - Add z-index: 100 to add-site-view to properly cover story chrome - Add dark mode styles for sidebar header Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Move Google News from Search tab to its own dedicated tab - Restructure header: tabs and view toggle now share same line - Move search bar below tabs into Search tab content - Each source tab has its own search bar - Remove unused render_google_news_section method - Add CSS for new tab search bar and tabs row layout - Update dark mode styling for new layout Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* archive-extension: Remove archive layout animations and add dismissible extension promo
- Hide north/south taskbars using jQuery Layout engine (like Reading Archive) - Pass true parameter for instant hide/show without animation - Load intelligence slider on entry so correct filter is selected - Wrap make_feed_title_in_stories in try-catch to handle fake folders - Update close/open story titles pane to check for add_site_view flag - Add dark mode styling for Google News topics and result cards Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…e channels - Add curated lists of popular podcasts, newsletters, and YouTube channels - Add category filtering for YouTube, podcasts, Reddit, and newsletters - Add discovery container with source pills and filter pills UI - Add section header styling with icons and links - Enable filtering by category (tech, news, gaming, education, etc.) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ggle - Remove Trending and Categories tabs from tab bar, keeping only Search, YouTube, Reddit, Newsletters, Podcasts, and Google News - Integrate trending feeds into Search tab empty state with grid/list view toggle support using TrendingFeeds collection - Replace "Browse by Source" with "Browse by Category" section featuring 6 category pills (Technology, News, Business, Science, Entertainment, Sports) that navigate to category feeds inline - Add CSS for trending feed grid cards and category pill hover states - Update newsletter cards to use actual icons when available Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
samuelclay
changed the title
Move all feed discovery, search, and add site related endpoints from rss_feeds app into new apps/discover/ app for better organization. Changes: - Create apps/discover/ with views.py, urls.py, tests.py - Move 11 view functions: feed_autocomplete, search_feed, trending_sites, youtube_search, reddit_search, reddit_popular, podcast_search, newsletter_convert, google_news_feed, discover_feeds, discover_stories - Update all frontend JS files to use new /discover/ endpoints - Add HTTP 308 redirects for pre-existing URLs to maintain backward compatibility (preserves POST method) New URL structure under /discover/: - /autocomplete, /search_feed, /trending - /youtube/search, /reddit/search, /reddit/popular - /podcast/search, /newsletter/convert, /google-news/feed - /similar/<feed_id>, /similar/feeds, /similar/stories/<hash> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* main: (45 commits) Separate one-time grandfathering from daily email task Add hide button to trial module and style all module hide buttons Show growth prompts only once per user, ever Add /commit-update-pr slash command for Claude Code Simplify grandfathering to single tier: 1 year grace period for all users over 1024 feeds Fix grandfathering: only apply to users over 1024 feeds Simplify grandfathering: two-tier system for existing premium users Fix worktree-close to reliably stop containers and sync permissions Add printable invoice feature for payment history Add /worktree slash command for Claude Code Fix PaymentHistory.MultipleObjectsReturned error in setup_premium_history Make relative image URLs absolute by modifying soup elements directly Improve feed merge and unmute handling Remove orphan migrator, add iOS unmute support Add delete shared stories feature with inline story counts Add safe single-feed mute/unmute endpoint Fix feed merge bugs causing muted feeds and data loss Main on staging Route Mercury requests to staging domain on staging server Switch to @jocmp/mercury-parser fork ...
* main: Fix add_feed_limit rendering Python None as string instead of JS null Fix NameError in save_feed_chooser by adding missing approve_all parameter
Resolved conflict in darkmode.css by keeping both Add Site view styles and Manage Training Tab styles. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* main: (23 commits) Update blog post title to indicate Premium Archive feature Fix N+1 query in get_auto_mark_read_cutoff for river stories Add retry logic to CI workflow for transient network failures Fix Grafana consul alert timing to require continuous 15min downtime Ignore Stripe/PayPal webhooks for non-NewsBlur products (Crabigator) Fix PayPal webhook KeyError when custom_id/custom field is missing Batch analytics cleanup deletes to prevent MongoDB memory exhaustion Fix TypeError when YouTube feeds receive RFC 5005 archive_page option Add Premium Archive and Pro user charts to top of NewsBlur dashboard Add request deduplication for Android river_stories to prevent duplicate concurrent requests Add replication lag panels to main NewsBlur dashboard Add database replication lag monitoring for Redis, Postgres, and Mongo Add Postgres replication slot to prevent WAL gaps on secondary Add rename and delete functionality for saved story tags Redesign days of unread preference with segmented control and slider Add blog post for per-feed auto-mark-as-read feature Fix Grafana alerts: noDataState and email subject format Redesign feed settings dialog auto-mark-read to match popover style Fix per-feed auto mark read filtering for cached stories and subfolders Fix auto mark read cutoff and folder inheritance issues ... # Conflicts: # media/css/reader/darkmode.css
- Preserve search input focus during Add Site search by only updating results section instead of re-rendering entire tab - Track search version to ignore stale responses from previous searches - Add database fallback when Elasticsearch returns no results, searching feed_title and feed_address fields Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use hybrid search (text + semantic) for feed autocomplete instead of text-only with fallback - finds "smitten kitchen" when searching "cooking" - Increase autocomplete limit to 20 to include more semantic matches - Add loading spinner in search input that shows during search requests - Include dark mode styling for the spinner Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use render_feed_card() for trending grid view to match search results - Remove fake trending feeds fallback, use real popular feeds from DB - Add tagline/description to feeds in trending and discover APIs (full=True) - Style improvements for trending feed cards in grid and list views - Update FeedBadge to support grid-style folder selector in add site view Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…oning - Fix tabs overflow: remove horizontal scroll, allow tabs to shrink with ellipsis truncation, reduce padding/gap for compact fit - Add subscribed feed detection: show checkmark indicator with Stats and Open buttons instead of folder selector and Add button - Redesign subscribed indicator: simple green checkmark text instead of button-like pill styling - Add Stats button to both grid and list views for subscribed feeds - Fix style popover arrow position: align with Style button using right offset instead of percentage-based left position - Prevent controls container from shrinking with flex-shrink: 0 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add AddSiteStylePopover view for customizing feed display preferences (preview count, images, content, sort order, card density) - Fix reader.js guard for active_folder methods in Add Site view context - Add in_add_site_view option to story title/titles views for custom content preview behavior with CSS line clamping - Refine style popover CSS: fixed width, better segmented control sizing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add left green border accent to grid view cards to match list view. Update list view FeedBadge to use grid-style header layout with 40px favicon, title, and subscriber/stories meta. Style Stats button consistently with border in both views. Add Stats and Open buttons to list view subscribed state. Fix padding and layout consistency between views. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix dark mode for Grid/List toggle buttons, search input, trending feeds title, and section borders - Make search input background transparent to match wrapper - Override global input background-color for search input - Add infinite scroll for trending feeds with append mode - Add scroll event handler with dynamic binding for results container - Improve feed badge layout with folder/add button grouping - Add trending feed badge dark mode styles Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Move Try button to the left of folder selector in grid view cards (now: Try | Folder | Add instead of Folder | Try | Add) - Add responsive overflow menu for tabs when window is narrow - Tabs that don't fit collapse into "More" dropdown menu - Active tab always remains visible regardless of overflow - Add dark mode styles for overflow button and menu Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Separate the style popover options into clearly labeled sections so users can see which settings apply to which view mode. Grid View section (with new auto/1-4 column selector) appears first, followed by List View section (Stories, Image Preview, Story Text). Sort By remains shared at the top. Remove unused Card Density setting. Use subtle background grouping instead of divider lines for cleaner visual hierarchy. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The relevance sort option was a pass-through for server order which isn't meaningful for the Add Site view. Default sort order is now subscribers, and the staging branch is pointed to add-site. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Show a colored dot + label on feed cards indicating how recently a feed was updated: active (green, <1 year), stale (amber, >1 year with date), or no stories yet (blue-gray). Extracted into shared helper make_freshness_indicator() used by grid view cards, list view FeedBadge, and source cards. Dates formatted with user's browser locale. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… tabs Refactor all source tab card renderers to use make_source_card with Try/Add buttons, freshness indicator, and subscription detection matching the Search tab's render_feed_card pattern. Replaces custom DOM building in render_podcast_card, render_newsletter_card, and render_popular_newsletter_card with consistent make_source_card calls. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
render_newsletter_card and render_podcast_card were not passing popular_feed_id to make_source_card, so unlinked cards had no data-popular-feed-id attribute. This caused link_popular_feed to silently return, making the Try button a no-op for most entries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* main: Fix briefing folder filter to include subfolders and prevent silent fallback Add call counts, millisecond precision, and dim styling to DB times log Inline briefing icons as data URIs, add model/stats debug footer Embed briefing favicons and section icons server-side for email support Redesign welcome page hero and platform sections Fix OpenAI reasoning model token limits, add gray briefing icons Revert SVG source colors, use CSS grayscale filter for briefing icons Normalize briefing section icon colors to consistent #95968E gray Fix briefing folder selection, remove Pro gating from model selector Add configurable AI model selection for daily briefing
- Build newsletter platform pills dynamically from backend data instead of hardcoding (removes broken Buttondown/Ghost pills with zero feeds) - Filter category counts by selected platform so numbers reflect only that platform's newsletters (e.g. Substack shows 185, not 475) - Return platform_counts from /discover/popular_feeds API endpoint - Reclassify generic/direct newsletters as "independent" in fixture - Add count badges on platform pills matching category pill style - Add dark mode styles for platform pill counts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…om APIs Discovers newsletters from Substack category API (3,625), Ghost Explore sitemap (595), and Medium search (106) - replacing AI-generated fixture data with 4,326 real newsletters with verified titles, descriptions, and subscriber counts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* main: Refine welcome page sub-features: cover-fit images, remove redundant app entries, swap order Fix trainer folder dropdown to include folders with folder-scoped classifiers Update welcome page features heading and add 6 new sub-features
Scrapes TopPub.xyz leaderboard across 99 tags (3 pages each) to discover Medium publications with titles, descriptions, and follower counts. Combined with existing search, total newsletters: 3,625 Substack + 2,526 Medium + 600 Ghost. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace 452 AI-generated subreddit entries with real data from Reddit API. Add discover_subreddits management command (OAuth2 auth, popular pagination, taxonomy keyword search, Claude categorization). Fix Try/Stats buttons by passing popular_feed_id in render_reddit_card. Add debounced search-as-you-type for Reddit tab matching newsletter pattern. Add count header to Reddit section. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…trending section The Trending tab duplicated the trending sites already shown in the Search tab's empty state. This consolidates them: the time period dropdown (Today/ This Week/This Month) now lives in the Search tab's trending section header, trending results expand from 10 to 20 per page, and the standalone Trending tab is removed. Also adds semantic search to popular feeds, podcast discovery command, and real podcast fixture data. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ories Expanded TAXONOMY from 27 to 47 categories by reusing the extended taxonomy from discover_real_feeds.py (adding diy & crafts, automotive, pets & animals, anime & manga, space & astronomy, economics, etc.) and added a new categorize_subreddits management command that uses Claude Haiku to batch-assign subcategories. All 5,637 Reddit subreddits now have proper category and subcategory assignments with zero gaps. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* main: Add clickable story links, favicon tooltips, and fix briefing email icon Add descriptive labels to LLM Costs stat panels in Grafana Move 3 misplaced table panels into Trending Feeds section in Grafana dashboard Fix LLM costs Prometheus scrape timeout by eliminating scan_iter on 150K keys Fix LLM Costs panels overlapping Discover Usage in Grafana dashboard Fix daily briefing JS errors from missing secure_image_urls and uninitialized views Replace abstract thinking icon with recognizable Lucide brain SVG Fix Discover panel overlap with LLM Costs and refactor taskbar info to flexbox Fix bottom toolbar layout by converting taskbar-info to flex child Add discover usage tracking to Prometheus/Grafana Redesign daily briefing with inline styles for email-compatible rendering Move try feed Subscribe/Follow/Sign up buttons to story titles list Fix model dropdown scroll and undefined model on re-ask click Add Ask AI thinking mode toggle and fix briefing lock TTL Fix triple daily briefing by adding Redis distributed locks Fix Manage Training crash when folder-scoped classifiers exist Fix daily briefing to reload on re-click, matching feed behavior Fix sub-feature layout with flexbox and add mobile viewport meta tag # Conflicts: # apps/rss_feeds/views.py
Move background gradient to outer popover container, removing the inner NB-add-form border that created a busy double-border effect. Relocate folder chooser to the header row (replacing RSS/Web feeds badges), widen the popover, increase padding and spacing throughout, add rounded corners, and offset 2px from left edge. Dynamic placeholder on new folder input shows parent folder name. Clamp negative subscriber counts to 0 in autocomplete results. Remove hover transition delay and expand autocomplete max-height to viewport-based calculation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Discover popover feed headers were unstyled after the Add Site modal redesign scoped CSS too specifically. This adds the same feed badge styling (flex layout, 40px favicon, title/meta/tagline fonts, freshness indicator, action buttons) to the Discover popover context. Also fixes negative subscriber counts displaying as "-1 subscribers" and makes the Add button green to match the Try button style. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…r feeds Semantic search results were bypassing category and subcategory filters, causing inline search to return results from all categories instead of the selected one. Also skip extra semantic Feed objects when filtering by category since they lack category metadata. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Already covered by the *_cache.json gitignore pattern. This intermediate cache file was force-added but isn't needed in the repo — only popular_feeds.json (the final consolidated fixture) needs to be tracked. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Validated all 435 YouTube entries against the YouTube Data API and found 86% (363) had fake channel IDs. Removed those and rediscovered real channels by searching the YouTube API across 47 categories. Each new entry has a verified channel ID and real subscriber count. YouTube coverage: 72 → 1,537 entries across 52 categories. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Thread max_stories option through get_feed_from_url → Feed.update → ProcessFeed.process so bootstrap_popular_feeds saves only 1 story per feed instead of up to 100, preventing millions of unnecessary stories from polluting the database when bootstrapping 46K+ feeds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…xecutor Split the command into phases: sequential PopularFeed creation, then parallel Feed.get_feed_from_url calls (default 10 workers). Each worker thread closes its DB connection on completion. Adds --workers flag to control concurrency. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prints a status line every 100 feeds showing completion percentage, linked count, and failed count during the parallel fetch phase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When trying a feed from Discover, captures the current tab, category, subcategory, and search query as origin context. Renders a clickable breadcrumb below the subscribe banner (e.g. "Go back to Discover > Newsletters > Science") that navigates back to the exact Discover state. Includes dark mode styles. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows restarting from a specific position in the fixture list after a killed process, e.g. --offset 4263 --limit 5000 to process a batch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…r input Invert mono tab icons (Popular, Search, etc.) in dark mode so they're visible against dark backgrounds. Brighten source header title/tagline colors for better contrast. Match filter input background/border to main search input. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each progress line now shows the exact --offset to use for restarting, e.g. "Fetching feeds: 100/43000 (0.2%) - 87 linked, 3 failed [restart with --offset 4365]" Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* main: Fix Daily Briefing stories replacing dashboard river story titles Fix deploy downtime by waiting for HAProxy health checks and tuning check thresholds Add idle detection to archive extension for accurate reading time Show "Untitled" for stories with no title, content, or permalink Bump daily briefing email font size from 16px to 18px for better readability Fix briefing crash by calling secure_image_urls on Feed instead of MStory Fix three Daily Briefing bugs: nested folder matching, disabled sections, and custom prompts
Phase 1 was silent while processing 46k entries, making it look stuck. Now logs progress every 500 entries with rate, ETA, and restart offset. Added utils/bootstrap_loop.sh that parses [restart with --offset N] from output, saves to /tmp, and auto-restarts on OOM kills. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1 progress was writing [restart with --offset N] which overwrote Phase 2's offset, causing the loop to skip ahead to the end on restart. Now Phase 1 logs progress without the parseable offset tag. Loop script processes in batches (BATCH=2000) with fewer workers (WORKERS=3) to avoid OOM kills. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the minimal single-icon empty state with four illustrated cards in a 2x2 grid that explain how web feeds work: creating feeds from any website, AI-powered story pattern detection, refining with hints, and automatic updates. Each card has a two-color SVG illustration (gray + blue accent), accessible description, and technical detail line. Includes staggered fade-in animation and full dark mode support. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add right padding to the URL input when the autocomplete spinner is active so text doesn't render underneath the loading indicator. Also redesign the web-feed-refine SVG icon to show a magnifying glass over a highlighted story row instead of the previous input bar with refresh arrow. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* main: Enable all briefing sections by default and fix custom section names in sidebar Add favicon-as-bullet indentation for editorial and headlines briefing styles Apply black and isort formatting fixes Add ScrapingBee fallback when adding feeds blocked by Cloudflare Add comprehensive test suite for Daily Briefing feature (145 tests) Fix flaky CI test by mocking MStory.objects in archive sync tests Add production health check tooling and downtime investigation playbook Fix disabled sections still appearing in Daily Briefing sidebar Fix folder-scoped classifiers showing from wrong folders in Train dialog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a comprehensive redesign of the Add Site experience, transforming it into a full-page discovery hub with multiple content sources and an improved user interface.
Key Features
Add + Discover Sites Page
Popular Sites Tab (NEW)
--discover-rssmode to query well-read feeds from production DB, Claude API taxonomy generator for RSS categorizationSearch Tab Enhancements
Source-Specific Discovery
UI/UX Improvements
Archive Extension Integration
Test plan
🤖 Generated with Claude Code