Feature/telemetry feedback rating clean#153
Merged
wherka-ama merged 21 commits intoAmadeusITGroup:feature/telemetry-feedback-rating-cleanfrom Feb 4, 2026
Conversation
added 21 commits
February 4, 2026 10:31
- Add telemetry types (TelemetryEvent, TelemetryFilter, TelemetryEventType) - Add rating types (Rating, RatingScore, RatingStats) - Add feedback types (Feedback, EngagementResourceType) - Add backend configuration types (FileBackendConfig, GitHubDiscussionsBackendConfig) - Add privacy settings types (EngagementPrivacySettings) - Add hub engagement configuration types (HubEngagementConfig) These types form the foundation for the engagement system, supporting telemetry tracking, user ratings, feedback collection, and multiple backend implementations.
- Add Wilson score lower bound for binary ratings (up/down votes) - Add Bayesian smoothing for positive-only ratings - Add score-to-stars and stars-to-score conversion utilities - Add aggregate score calculation for collections - Add comprehensive tests for all algorithms These algorithms provide robust ranking that handles small sample sizes well, preventing items with few votes from dominating rankings.
- Add EngagementStorage class for persistent storage of engagement data - Support telemetry events, ratings, and feedback storage - Implement JSON file-based persistence with atomic writes - Add query methods with filtering support - Add comprehensive tests for all storage operations This provides the persistence layer for the engagement system, storing data in the extension's global storage directory.
- Add IEngagementBackend interface defining backend contract - Add BaseEngagementBackend abstract class with common functionality - Implement FileBackend for local file-based storage - Support telemetry, ratings, and feedback operations - Add comprehensive tests for FileBackend The backend abstraction allows multiple implementations (file, GitHub Discussions, API) while maintaining a consistent interface.
- Implement GitHubDiscussionsBackend for voting via GitHub reactions - Support collection-level voting on discussions - Support resource-level voting on comments - Load collection mappings from collections.yaml URL - Post feedback as discussion comments - Use FileBackend for local telemetry/feedback storage - Add comprehensive tests with nock HTTP mocking This backend enables community engagement through GitHub Discussions, using reactions for voting and comments for feedback.
- Implement singleton EngagementService as unified facade - Support hub-specific backend routing - Initialize default file backend on startup - Register hub backends dynamically based on configuration - Emit events for rating and feedback submissions - Support privacy settings enforcement - Add comprehensive tests for service operations The EngagementService provides a single entry point for all engagement operations, routing requests to the appropriate backend based on hub.
- Add VoteService for voting via GitHub Discussion reactions - Support upvote (+1) and downvote (-1) on collections and resources - Implement vote toggle and removal - Add VoteCommands for VS Code command registration - Use VS Code GitHub authentication provider - Add comprehensive tests for voting operations Users can now vote on collections directly from VS Code, with votes synced to GitHub Discussions as reactions.
- Add RatingService for fetching ratings from hub's ratings.json - Add RatingCache for in-memory caching with TTL - Support rating lookup by bundle ID with source prefix matching - Parse and validate ratings.json format - Add comprehensive tests for rating operations Ratings are pre-computed by GitHub Actions and served as static JSON, enabling fast rating display without runtime computation.
- Add FeedbackService for feedback operations - Add FeedbackCache for in-memory feedback caching - Add FeedbackCommands with unified feedback dialog - Implement star rating (1-5) with optional comment - Support issue redirect to source repository - Extract GitHub org/repo for proper issue URL construction - Add comprehensive tests for feedback operations Users can submit feedback with star ratings and optional comments, with the option to open a GitHub issue for detailed bug reports.
- Add engagement configuration schema to hub-config.schema.json - Add registerHubEngagement() to HubManager for backend initialization - Add initializeEngagementBackends() for activation-time initialization - Support GitHub Discussions backend configuration - Add validation tests for engagement configuration Hubs can now configure engagement backends, enabling automatic registration of the appropriate backend when a hub is imported.
- Add rating display to marketplace bundle tiles - Add rating display to tree view items - Integrate RatingCache for fetching cached ratings - Show star rating, vote count, and confidence level - Update registry types for rating display support Users can now see ratings directly in the marketplace and tree view, helping them discover high-quality bundles.
- Add marketplace webview assets (HTML, CSS, JS) as separate files - Add bundleDetails webview assets (HTML, CSS, JS) as separate files - Use IIFE pattern for JavaScript to comply with CSP - Use data-action attributes instead of inline event handlers - Add CopyWebpackPlugin to webpack config for asset copying Externalizing webview content improves maintainability, enables CSP compliance, and simplifies debugging of webview code.
- Initialize EngagementService on extension activation - Initialize engagement backends for all existing hubs - Register FeedbackCommands and VoteCommands - Add copy-webpack-plugin dependency for webview assets - Add feedback and vote commands to package.json - Update context menu entries for engagement commands The extension now fully integrates the engagement system, initializing all services and registering commands on activation.
- Add compute-ratings.ts for rating computation from GitHub Discussions - Add setup-discussions.ts for creating discussions from collections - Implement Wilson score and Bayesian smoothing algorithms - Support star rating parsing from feedback comments - Add user deduplication (last rating wins) - Add CLI wrappers for direct script execution - Add comprehensive tests for rating computation - Configure as standalone npm package (@prompt-registry/lib) This package runs in GitHub Actions to compute ratings from Discussion reactions and feedback comments, outputting ratings.json for static hosting.
- Update compute-ratings description to mention star rating parsing from comments - Update setup-discussions description to mention star ratings instead of reactions - Add npx --package syntax to all engagement tool examples - Note backward compatibility with legacy thumbs up/down reactions
- Add comprehensive user guide for engagement features - Document feedback submission flow with star ratings - Add hub maintainer setup guide for GitHub Discussions - Add compute-ratings.yml workflow as downloadable asset - Update hub schema documentation for engagement config - Update commands reference for feedback and vote commands - Document webview architecture and CSP compliance - Fix ESLint errors in FeedbackCommands.ts regex patterns
- Update license from MIT to Apache-2.0 to match main repository - Bump version to 1.0.3
- Remove VoteCommands.ts and VoteService.ts (not used in UI) - Remove all voting command registrations from package.json - Remove voting command imports from extension.ts - Update documentation to focus on feedback-only engagement - Update .gitignore to exclude working/analysis files The voting functionality was implemented but never integrated into the UI or used anywhere in the codebase. This cleanup removes the unused code while keeping the core feedback and rating features.
- Remove legacy reaction-based fallback when no star ratings exist - Collections with no ratings now show star_rating: 0 instead of 3.5 - Convert bin/compute-ratings.js to wrapper script calling dist/ code - Fix wilson_score calculation to properly normalize 5-star ratings Previously, when no star ratings were found, the system fell back to the legacy reaction-based system which used bayesianSmoothing() with a prior mean of 3.5. This caused collections with 0 ratings to show a star_rating of 3.5, which is incorrect. Now, collections with no ratings show 0, and only actual star ratings from feedback comments are used for the rating computation.
- Add rating_count field to CollectionRating interface - Track actual number of star ratings separately from reactions - Update RatingService to use rating_count instead of up+down - Fix issue where ratings with 0 reactions but >0 star ratings were hidden Previously, totalVotes was calculated as up+down (reactions), which caused bundles with star ratings but no reactions to show voteCount=0 and be hidden from the UI. Now we properly track the actual number of star ratings in the rating_count field.
The RatingCache was storing ratings using source_id from ratings.json (e.g., 'awesome-copilot-official'), but the UI was looking them up using the full extension source ID (e.g., 'hub-awesome-copilot-hub-099578-awesome-copilot-official'). This caused all cache lookups to return undefined, resulting in 'No ratings yet' being displayed even when ratings existed. Fix by: - Adding sourceIdMap parameter to RatingCache.refreshFromHub() - Building source ID mapping in extension.ts when loading hub ratings - Using mapped source IDs when populating the cache This ensures ratings are stored with the correct source IDs that match what the UI uses for lookups.
gblanc-1a
reviewed
Feb 4, 2026
| lib/ratings.json | ||
|
|
||
| # Spec working directories | ||
| .kiro/specs/*/IMPLEMENTATION_PLAN.md |
Contributor
There was a problem hiding this comment.
Should we actually commit our spec ? Was thinking a new way of working could be open the PR already after generating the specs so that we can review. (You already did that by presenting it to us beforehand which is great !)
WDYT ?
gblanc-1a
reviewed
Feb 4, 2026
| |-------|------|-------------| | ||
| | `enabled` | boolean | Enable engagement features for this hub | | ||
| | `backend` | object | Backend configuration for engagement data storage | | ||
| | `telemetry` | object | Telemetry configuration | |
Contributor
There was a problem hiding this comment.
Not sure if we should include the telemetry as part of the engagement (it would be more technical)
wherka-ama
changed the base branch from
main
to
feature/telemetry-feedback-rating-clean
wherka-ama
merged commit 9f90fb6
into
AmadeusITGroup:feature/telemetry-feedback-rating-clean
14 checks passed
Contributor
|
Notes taken while discussing about remaining point to adress with @wherka-ama Reactive rating with single workflow inside engagement repository
Telemetry
Bug
UX Improvements
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces a comprehensive engagement system for the Prompt Registry, enabling users to rate bundles and submit feedback. The system provides a complete UI integration in both the marketplace and registry explorer.
The engagement system includes:
Type of Change
Related Issues
Relates to feedback system refactoring and community engagement features.
#98 (#20, #25, #80 )
Changes Made
Core Engagement System
Type Definitions (
src/types/engagement.ts)Storage Layer (
src/storage/EngagementStorage.ts)Rating Algorithms (
src/utils/ratingAlgorithms.ts)Backend Implementations
Backend Interface (
src/services/engagement/IEngagementBackend.ts)File Backend (
src/services/engagement/backends/FileBackend.ts)GitHub Discussions Backend (
src/services/engagement/backends/GitHubDiscussionsBackend.ts)Services
EngagementService (
src/services/engagement/EngagementService.ts)RatingService (
src/services/engagement/RatingService.ts)RatingCache (
src/services/engagement/RatingCache.ts)FeedbackService (
src/services/engagement/FeedbackService.ts)FeedbackCache (
src/services/engagement/FeedbackCache.ts)Commands
FeedbackCommands (
src/commands/FeedbackCommands.ts)UI Integration
Marketplace (
src/ui/MarketplaceViewProvider.ts,src/ui/webview/marketplace/*)Registry Tree View (
src/ui/RegistryTreeProvider.ts)Bundle Details (
src/ui/webview/bundleDetails/*)Hub Integration
Hub Manager (
src/services/HubManager.ts)Hub Schema (
schemas/hub-config.schema.json,src/types/hub.ts)Extension Integration
Extension Activation (
src/extension.ts)Package Configuration (
package.json)Rating Computation Package
@prompt-registry/collection-scripts (
lib/)@prompt-registry/collection-scripts@1.0.3Documentation
User Guide (
docs/user-guide/engagement.md)Reference Documentation
docs/reference/commands.md)docs/reference/hub-schema.md)docs/contributor-guide/architecture/ui-components.md)Workflow Asset (
docs/assets/compute-ratings.yml)Package Documentation (
lib/README.md)Build Configuration
Webpack (
webpack.config.js)Testing
Test Coverage
Test Files Added
Engagement Services (1,900+ lines of tests)
test/services/engagement/EngagementService.test.ts- Service facade teststest/services/engagement/RatingService.test.ts- Rating service teststest/services/engagement/RatingCache.test.ts- Cache teststest/services/engagement/FeedbackCache.test.ts- Feedback cache testsBackend Tests (700+ lines)
test/services/engagement/backends/FileBackend.test.ts- File backend teststest/services/engagement/backends/GitHubDiscussionsBackend.test.ts- GitHub backend tests with nock HTTP mockingCommand Tests (200+ lines)
test/commands/FeedbackCommands.test.ts- Feedback command testsStorage Tests (670+ lines)
test/storage/EngagementStorage.test.ts- Storage layer testsUtility Tests (700+ lines)
test/utils/ratingAlgorithms.test.ts- Rating algorithm testslib/test/compute-ratings.test.ts- Rating computation testsType Validation Tests (420+ lines)
test/types/hub-engagement-validation.test.ts- Hub engagement config validationTotal Test Coverage: 4,800+ lines of comprehensive tests
Manual Testing Steps
Install and activate a bundle from marketplace
Submit feedback with star rating
Hub engagement configuration
Rating computation (for hub maintainers)
npx --package @prompt-registry/collection-scripts setup-discussionsnpx --package @prompt-registry/collection-scripts compute-ratingsTested On
Screenshots
Checklist
Documentation
Additional Notes
Published Package
The rating computation tools have been published to npm:
@prompt-registry/collection-scripts@1.0.3compute-ratings,setup-discussionsCLI toolsBreaking Changes
None. This is a purely additive feature that does not affect existing functionality.
Migration Notes
For hub maintainers who want to enable community engagement:
setup-discussionsto create discussions for bundlesdocs/assets/compute-ratings.ymlKey Implementation Fixes
During development, several critical issues were identified and resolved:
Rating Count Field: Added
rating_countfield to track actual star ratings separately from legacy reaction-based votes (up/down). This ensures the 5-star rating system correctly reflects the number of star ratings rather than reaction counts.Source ID Mapping: Fixed sourceId mismatch where
RatingCachestored ratings usingsource_idfromratings.json(e.g.,"awesome-copilot-official"), but the UI looked them up using hub-prefixed source IDs (e.g.,"hub-awesome-copilot-hub-099578-awesome-copilot-official"). Implemented source ID mapping inRatingCache.refreshFromHub()to ensure ratings are stored with the correct source IDs that match UI lookups.Rating Computation Algorithm: Corrected the star rating computation to properly parse 1-5 star ratings from feedback comments and calculate accurate averages with proper confidence levels.
Architecture Decisions
Reviewer Guidelines
Please pay special attention to:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.