Feature/telemetry feedback rating clean#177
Draft
gblanc-1a wants to merge 37 commits intoAmadeusITGroup:mainfrom
Draft
Feature/telemetry feedback rating clean#177gblanc-1a wants to merge 37 commits intoAmadeusITGroup:mainfrom
gblanc-1a wants to merge 37 commits intoAmadeusITGroup:mainfrom
Conversation
- Add telemetry types (TelemetryEvent, TelemetryFilter, TelemetryEventType) - Add rating types (Rating, RatingScore, RatingStats) - Add feedback types (Feedback, EngagementResourceType) - Add backend configuration types (FileBackendConfig, GitHubDiscussionsBackendConfig) - Add privacy settings types (EngagementPrivacySettings) - Add hub engagement configuration types (HubEngagementConfig) These types form the foundation for the engagement system, supporting telemetry tracking, user ratings, feedback collection, and multiple backend implementations.
- Add Wilson score lower bound for binary ratings (up/down votes) - Add Bayesian smoothing for positive-only ratings - Add score-to-stars and stars-to-score conversion utilities - Add aggregate score calculation for collections - Add comprehensive tests for all algorithms These algorithms provide robust ranking that handles small sample sizes well, preventing items with few votes from dominating rankings.
- Add EngagementStorage class for persistent storage of engagement data - Support telemetry events, ratings, and feedback storage - Implement JSON file-based persistence with atomic writes - Add query methods with filtering support - Add comprehensive tests for all storage operations This provides the persistence layer for the engagement system, storing data in the extension's global storage directory.
- Add IEngagementBackend interface defining backend contract - Add BaseEngagementBackend abstract class with common functionality - Implement FileBackend for local file-based storage - Support telemetry, ratings, and feedback operations - Add comprehensive tests for FileBackend The backend abstraction allows multiple implementations (file, GitHub Discussions, API) while maintaining a consistent interface.
- Implement GitHubDiscussionsBackend for voting via GitHub reactions - Support collection-level voting on discussions - Support resource-level voting on comments - Load collection mappings from collections.yaml URL - Post feedback as discussion comments - Use FileBackend for local telemetry/feedback storage - Add comprehensive tests with nock HTTP mocking This backend enables community engagement through GitHub Discussions, using reactions for voting and comments for feedback.
- Implement singleton EngagementService as unified facade - Support hub-specific backend routing - Initialize default file backend on startup - Register hub backends dynamically based on configuration - Emit events for rating and feedback submissions - Support privacy settings enforcement - Add comprehensive tests for service operations The EngagementService provides a single entry point for all engagement operations, routing requests to the appropriate backend based on hub.
- Add VoteService for voting via GitHub Discussion reactions - Support upvote (+1) and downvote (-1) on collections and resources - Implement vote toggle and removal - Add VoteCommands for VS Code command registration - Use VS Code GitHub authentication provider - Add comprehensive tests for voting operations Users can now vote on collections directly from VS Code, with votes synced to GitHub Discussions as reactions.
- Add RatingService for fetching ratings from hub's ratings.json - Add RatingCache for in-memory caching with TTL - Support rating lookup by bundle ID with source prefix matching - Parse and validate ratings.json format - Add comprehensive tests for rating operations Ratings are pre-computed by GitHub Actions and served as static JSON, enabling fast rating display without runtime computation.
- Add FeedbackService for feedback operations - Add FeedbackCache for in-memory feedback caching - Add FeedbackCommands with unified feedback dialog - Implement star rating (1-5) with optional comment - Support issue redirect to source repository - Extract GitHub org/repo for proper issue URL construction - Add comprehensive tests for feedback operations Users can submit feedback with star ratings and optional comments, with the option to open a GitHub issue for detailed bug reports.
- Add engagement configuration schema to hub-config.schema.json - Add registerHubEngagement() to HubManager for backend initialization - Add initializeEngagementBackends() for activation-time initialization - Support GitHub Discussions backend configuration - Add validation tests for engagement configuration Hubs can now configure engagement backends, enabling automatic registration of the appropriate backend when a hub is imported.
- Add rating display to marketplace bundle tiles - Add rating display to tree view items - Integrate RatingCache for fetching cached ratings - Show star rating, vote count, and confidence level - Update registry types for rating display support Users can now see ratings directly in the marketplace and tree view, helping them discover high-quality bundles.
- Add marketplace webview assets (HTML, CSS, JS) as separate files - Add bundleDetails webview assets (HTML, CSS, JS) as separate files - Use IIFE pattern for JavaScript to comply with CSP - Use data-action attributes instead of inline event handlers - Add CopyWebpackPlugin to webpack config for asset copying Externalizing webview content improves maintainability, enables CSP compliance, and simplifies debugging of webview code.
- Initialize EngagementService on extension activation - Initialize engagement backends for all existing hubs - Register FeedbackCommands and VoteCommands - Add copy-webpack-plugin dependency for webview assets - Add feedback and vote commands to package.json - Update context menu entries for engagement commands The extension now fully integrates the engagement system, initializing all services and registering commands on activation.
- Add compute-ratings.ts for rating computation from GitHub Discussions - Add setup-discussions.ts for creating discussions from collections - Implement Wilson score and Bayesian smoothing algorithms - Support star rating parsing from feedback comments - Add user deduplication (last rating wins) - Add CLI wrappers for direct script execution - Add comprehensive tests for rating computation - Configure as standalone npm package (@prompt-registry/lib) This package runs in GitHub Actions to compute ratings from Discussion reactions and feedback comments, outputting ratings.json for static hosting.
- Update compute-ratings description to mention star rating parsing from comments - Update setup-discussions description to mention star ratings instead of reactions - Add npx --package syntax to all engagement tool examples - Note backward compatibility with legacy thumbs up/down reactions
- Add comprehensive user guide for engagement features - Document feedback submission flow with star ratings - Add hub maintainer setup guide for GitHub Discussions - Add compute-ratings.yml workflow as downloadable asset - Update hub schema documentation for engagement config - Update commands reference for feedback and vote commands - Document webview architecture and CSP compliance - Fix ESLint errors in FeedbackCommands.ts regex patterns
- Update license from MIT to Apache-2.0 to match main repository - Bump version to 1.0.3
- Remove VoteCommands.ts and VoteService.ts (not used in UI) - Remove all voting command registrations from package.json - Remove voting command imports from extension.ts - Update documentation to focus on feedback-only engagement - Update .gitignore to exclude working/analysis files The voting functionality was implemented but never integrated into the UI or used anywhere in the codebase. This cleanup removes the unused code while keeping the core feedback and rating features.
- Remove legacy reaction-based fallback when no star ratings exist - Collections with no ratings now show star_rating: 0 instead of 3.5 - Convert bin/compute-ratings.js to wrapper script calling dist/ code - Fix wilson_score calculation to properly normalize 5-star ratings Previously, when no star ratings were found, the system fell back to the legacy reaction-based system which used bayesianSmoothing() with a prior mean of 3.5. This caused collections with 0 ratings to show a star_rating of 3.5, which is incorrect. Now, collections with no ratings show 0, and only actual star ratings from feedback comments are used for the rating computation.
- Add rating_count field to CollectionRating interface - Track actual number of star ratings separately from reactions - Update RatingService to use rating_count instead of up+down - Fix issue where ratings with 0 reactions but >0 star ratings were hidden Previously, totalVotes was calculated as up+down (reactions), which caused bundles with star ratings but no reactions to show voteCount=0 and be hidden from the UI. Now we properly track the actual number of star ratings in the rating_count field.
The RatingCache was storing ratings using source_id from ratings.json (e.g., 'awesome-copilot-official'), but the UI was looking them up using the full extension source ID (e.g., 'hub-awesome-copilot-hub-099578-awesome-copilot-official'). This caused all cache lookups to return undefined, resulting in 'No ratings yet' being displayed even when ratings existed. Fix by: - Adding sourceIdMap parameter to RatingCache.refreshFromHub() - Building source ID mapping in extension.ts when loading hub ratings - Using mapped source IDs when populating the cache This ensures ratings are stored with the correct source IDs that match what the UI uses for lookups.
Covers network resilience, optimistic rating updates, interactive stars, sort fix, and other UX improvements from issue AmadeusITGroup#98. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
12-task TDD implementation plan covering network resilience, optimistic rating updates, interactive stars, sort fix, confidence removal, zero-rating consistency, report/request links, and retry mechanism. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use starRating as primary sort key with voteCount as tiebreaker. Previously used wilsonScore (0-1 range) which produced unintuitive ordering. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Keep vote count but remove confidence text (low/medium/high/very_high) from both marketplace badges and TreeView tooltips. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hide rating badge entirely when voteCount is 0 or starRating is 0. Previously could show empty stars or '0.0' for unrated bundles. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add /^installed_bundle/ to VALID_CONTEXT_PATTERNS_FOR_MENUS to match the package.json when clause for feedback menu items. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add PendingFeedback type and CRUD operations to EngagementStorage for tracking feedback that may not yet be synced to GitHub. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Save feedback locally regardless of GitHub API success. Show warning on network failure with instructions to retry. Feedback is marked as synced/unsynced for future retry support. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After user submits a rating, immediately update the cache with a recalculated average. Silently overwritten on next ratings.json fetch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After submitting feedback, immediately update RatingCache so the user sees their rating reflected in the UI without waiting for workflow. Add sourceId field to FeedbackableItem for cache key matching. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace display-only rating badge with hover-preview + click-confirm stars. Shows inline comment form after clicking a star. Submits feedback through existing engagement pipeline. Pre-filled ratings from webview bypass the QuickPick dialogs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the command-based feedback flow with direct EngagementService submission in MarketplaceViewProvider. The entire star rating and comment flow now stays inside the webview. Add feedbackSubmitted message to show confirmation inline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the "Rate & Feedback" button in the bundle detail panel with interactive hover/click stars and inline comment form. Feedback is submitted directly to EngagementService without VS Code dialogs. Both the marketplace card and bundle detail views now use the same inline star rating pattern. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align interactive stars to the left like the rest of the content in the bundle detail panel. Remove confidence text from rating display. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add Report Issue and Request Feature links in marketplace bundle cards and TreeView context menu. Opens source repository issues page with pre-filled bug or feature template. Add Retry Feedback Submission to context menu for re-submitting locally saved feedback that failed due to network issues. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
gblanc-1a
force-pushed
the
feature/telemetry-feedback-rating-clean
branch
from
22f075e to
a146b6f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Type of Change
Related Issues
Closes #
Fixes #
Relates to #
Changes Made
Testing
Test Coverage
Manual Testing Steps
Tested On
macOS
Windows
Linux
VS Code Stable
VS Code Insiders
Screenshots
Checklist
Documentation
Additional Notes
Reviewer Guidelines
Please pay special attention to:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.