diff --git a/README.md b/README.md index d657d8a8..4f26ae82 100644 --- a/README.md +++ b/README.md @@ -7,9 +7,47 @@ _NOTE: For information on support and assistance, click [here](https://github.co This application displays a set of charts with various metrics related to GitHub Copilot for your GitHub Organization or Enterprise Account. These visualizations are designed to provide clear representations of the data, making it easy to understand and analyze the impact and adoption of GitHub Copilot. This app utilizes the [GitHub Copilot Metrics API](https://docs.github.com/en/enterprise-cloud@latest/rest/copilot/copilot-usage?apiVersion=2022-11-28). -## Video +## Application Overview -https://github.com/github-copilot-resources/copilot-metrics-viewer/assets/3329307/bc7e2a16-cc73-43c4-887a-b50809c08533 +The GitHub Copilot Metrics Viewer provides comprehensive analytics through an intuitive dashboard interface: + +

+ Main Dashboard Overview +

+ +## New Features + +### Date Range Filtering (up to 100 days) +Users can now filter metrics for custom date ranges up to 100 days, with an intuitive calendar picker interface. The system also supports excluding weekends and holidays from calculations. + +

+ Date Range Filter +

+ +### Teams Comparison +Compare Copilot metrics across multiple teams within your organization to understand adoption patterns and identify high-performing teams. + +

+ Teams Comparison +

+ +### GitHub.com Integration & Model Analytics +View comprehensive statistics for GitHub.com features including Chat, PR Summaries, and detailed model usage analytics. Each section provides expandable details showing model types, editors, and usage patterns. + +

+ GitHub.com Tab +

+ +

+ Model Usage Details +

+ +### CSV Export Functionality +Export your metrics data in multiple formats for further analysis or reporting. Options include summary reports, full detailed exports, and direct clipboard copying. + +

+ CSV Export Options +

## Charts @@ -19,7 +57,7 @@ https://github.com/github-copilot-resources/copilot-metrics-viewer/assets/332930 Here are the key metrics visualized in these charts:

- image + Key Metrics Overview

1. **Acceptance Rate:** This metric represents the ratio of accepted lines and suggestions to the total suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot's suggestions. However, as with any metric, it should be used with caution as developers use Copilot in many different ways (research, confirm, verify, etc., not always "inject"). @@ -50,31 +88,28 @@ Here are the key metrics visualized in these charts: Pie charts with the top 5 languages by accepted prompts and acceptance rate (by count/by lines) are displayed at the top.

- image + Updated Language breakdown with charts and data table

-The language breakdown analysis tab also displays a table showing the Accepted Prompts, Accepted Lines of Code, and Acceptance Rate (%) for each language over the past 28 days. The entries are sorted by the number of _accepted lines of code descending_. -

- image -

+The language breakdown analysis tab also displays a table showing the Accepted Prompts, Accepted Lines of Code, and Acceptance Rate (%) for each language over the selected time period. The entries are sorted by the number of _accepted lines of code descending_. ## Copilot Chat Metrics

- image + Copilot Chat Metrics Dashboard

-1. **Cumulative Number of Turns:** This metric represents the total number of turns (interactions) with the Copilot over the past 28 days. A 'turn' includes both user inputs and Copilot's responses. +1. **Cumulative Number of Turns:** This metric represents the total number of turns (interactions) with the Copilot over the selected time period. A 'turn' includes both user inputs and Copilot's responses. -2. **Cumulative Number of Acceptances:** This metric shows the total number of lines of code suggested by Copilot that have been accepted by users over the past 28 days. +2. **Cumulative Number of Acceptances:** This metric shows the total number of lines of code suggested by Copilot that have been accepted by users over the selected time period. 3. **Total Turns | Total Acceptances Count:** This is a chart that displays the total number of turns and acceptances. -4. **Total Active Copilot Chat Users:** A bar chart that illustrates the total number of users who have actively interacted with Copilot over the past 28 days. +4. **Total Active Copilot Chat Users:** A bar chart that illustrates the total number of users who have actively interacted with Copilot over the selected time period. ## Seat Analysis

- image + Seat Analysis Dashboard

1. **Total Assigned:** This metric represents the total number of Copilot seats assigned within the current organization/enterprise. @@ -85,6 +120,33 @@ The language breakdown analysis tab also displays a table showing the Accepted P 4. **No Activity in the Last 7 Days (including never used seats):** A table to display seats that have had no activity in the past 7 days, ordered by the date of last activity. Seats that were used earlier are displayed at the top. +## Advanced Features + +### Flexible Date Range Selection +The application supports flexible date range selection allowing users to analyze metrics for any period up to 100 days. The date picker provides an intuitive calendar interface with options to exclude weekends and holidays from the analysis. + +### Data Export Capabilities +Multiple export options are available in the API Response tab: +- **Download CSV (Summary)**: Exports key metrics in a condensed format +- **Download CSV (Full)**: Exports comprehensive detailed data +- **Copy Metrics to Clipboard**: Quick copy functionality for immediate use +- **Check Metric Data Quality**: Validates data integrity and completeness + +### Team Analytics +Organizations can compare metrics across different teams to: +- Identify high-performing teams +- Understand adoption patterns +- Share best practices across teams +- Monitor team-specific engagement levels + +### Model Usage Analytics +Detailed insights into AI model usage including: +- IDE Code Completions by editor and model type +- IDE Chat interactions and model preferences +- GitHub.com Chat usage patterns +- PR Summary generation statistics +- Custom vs. default model adoption rates + ## Setup Instructions In the `.env` file, you can configure several environment variables that control the behavior of the application. diff --git a/e2e-tests/pages/DashboardPage.ts b/e2e-tests/pages/DashboardPage.ts index c1de80de..8b5f88ad 100644 --- a/e2e-tests/pages/DashboardPage.ts +++ b/e2e-tests/pages/DashboardPage.ts @@ -16,6 +16,7 @@ export class DashboardPage { readonly toolbarTitle: Locator; readonly teamTabLink: Locator; + readonly teamsTabLink: Locator; readonly orgTabLink: Locator; readonly enterpriseTabLink: Locator; @@ -43,12 +44,13 @@ export class DashboardPage { this.githubTabLink = page.getByRole('tab', { name: 'github.com' }) this.teamTabLink = page.getByRole('tab', { name: 'team' }) + this.teamsTabLink = page.getByRole('tab', { name: 'teams' }) this.orgTabLink = page.getByRole('tab', { name: 'organization' }) this.enterpriseTabLink = page.getByRole('tab', { name: 'enterprise' }) } async expectTeamsTabVisible() { - await expect(this.teamTabLink).toBeVisible(); + await expect(this.teamsTabLink).toBeVisible(); } async expectOrgTabVisible() { @@ -106,6 +108,12 @@ export class DashboardPage { return new GitHubTab(this.page); } + async gotoTeamsTab() { + await this.teamsTabLink.click(); + // No specific page object for teams tab, just return this for fluent interface + return this; + } + async close() { await this.page.close(); } diff --git a/e2e-tests/teams-comparison.spec.ts b/e2e-tests/teams-comparison.spec.ts new file mode 100644 index 00000000..5c9fc9b6 --- /dev/null +++ b/e2e-tests/teams-comparison.spec.ts @@ -0,0 +1,95 @@ +import { expect, test } from '@playwright/test' +import { DashboardPage } from './pages/DashboardPage'; + +const tag = { tag: ['@teams-comparison'] }; + +test.describe('Teams Comparison tests', () => { + + let dashboard: DashboardPage; + + test.beforeAll(async ({ browser }) => { + const page = await browser.newPage(); + await page.goto('/orgs/mocked-org?mock=true'); + + dashboard = new DashboardPage(page); + + // wait for the data + await dashboard.expectMetricLabelsVisible(); + }); + + test.afterAll(async () => { + await dashboard?.close(); + }); + + test('has teams tab available', tag, async () => { + // Verify that teams tab is visible for org scope + await dashboard.expectTeamsTabVisible(); + }); + + test('teams comparison functionality with team selection', tag, async () => { + // Click on teams tab + await dashboard.gotoTeamsTab(); + + // Wait for the team selection combobox to be visible + const teamsDropdown = dashboard.page.locator('[role="combobox"]').first(); + await expect(teamsDropdown).toBeVisible(); + + // Click on the dropdown to open it + await teamsDropdown.click(); + + // Wait for the dropdown to expand and team options to appear + await expect(dashboard.page.locator('[role="listbox"]')).toBeVisible(); + + // Select teams using more specific selectors within the listbox + const theATeamOption = dashboard.page.locator('[role="listbox"]').getByText('The A Team').first(); + const devTeamOption = dashboard.page.locator('[role="listbox"]').getByText('Development Team').first(); + + await expect(theATeamOption).toBeVisible(); + await theATeamOption.click(); + + await expect(devTeamOption).toBeVisible(); + await devTeamOption.click(); + + // Click outside the dropdown to close it + await dashboard.page.locator('body').click(); + + // Wait a moment for the UI to settle + await dashboard.page.waitForTimeout(1000); + + // Verify that teams are selected + const selectedTeamsSection = dashboard.page.getByText('Selected Teams'); + await expect(selectedTeamsSection).toBeVisible(); + + // Verify that team metrics cards are visible + const teamsSelectedCard = dashboard.page.getByText('Teams Selected'); + await expect(teamsSelectedCard).toBeVisible(); + + const totalActiveUsersCard = dashboard.page.getByText('Total Active Users'); + await expect(totalActiveUsersCard).toBeVisible(); + + // Verify that charts are displayed + const languageUsageChart = dashboard.page.getByText('Language Usage by Team'); + await expect(languageUsageChart).toBeVisible(); + + const editorUsageChart = dashboard.page.getByText('Editor Usage by Team'); + await expect(editorUsageChart).toBeVisible(); + + // Take a screenshot for documentation purposes + await dashboard.page.screenshot({ + path: 'images/teams-comparison-test.png', + fullPage: true + }); + }); + + test('teams comparison empty state', tag, async () => { + // Click on teams tab + await dashboard.gotoTeamsTab(); + + // Verify empty state message when no teams are selected + const emptyStateMessage = dashboard.page.getByText('No Teams Selected'); + await expect(emptyStateMessage).toBeVisible(); + + const emptyStateDescription = dashboard.page.getByText('Select one or more teams from the dropdown above to view and compare their metrics.'); + await expect(emptyStateDescription).toBeVisible(); + }); +}); \ No newline at end of file diff --git a/images/100-day-date-range.png b/images/100-day-date-range.png new file mode 100644 index 00000000..bf71b3b7 Binary files /dev/null and b/images/100-day-date-range.png differ diff --git a/images/copilot-chat-metrics.png b/images/copilot-chat-metrics.png new file mode 100644 index 00000000..d54b6e4c Binary files /dev/null and b/images/copilot-chat-metrics.png differ diff --git a/images/csv-export-functionality.png b/images/csv-export-functionality.png new file mode 100644 index 00000000..d1e13030 Binary files /dev/null and b/images/csv-export-functionality.png differ diff --git a/images/date-range-filter.png b/images/date-range-filter.png new file mode 100644 index 00000000..b7781d24 Binary files /dev/null and b/images/date-range-filter.png differ diff --git a/images/github-com-models-expanded.png b/images/github-com-models-expanded.png new file mode 100644 index 00000000..9466df6d Binary files /dev/null and b/images/github-com-models-expanded.png differ diff --git a/images/github-com-tab.png b/images/github-com-tab.png new file mode 100644 index 00000000..8f53101b Binary files /dev/null and b/images/github-com-tab.png differ diff --git a/images/languages-breakdown.png b/images/languages-breakdown.png new file mode 100644 index 00000000..d938e60d Binary files /dev/null and b/images/languages-breakdown.png differ diff --git a/images/main-metrics-dashboard.png b/images/main-metrics-dashboard.png new file mode 100644 index 00000000..209a474f Binary files /dev/null and b/images/main-metrics-dashboard.png differ diff --git a/images/seat-analysis.png b/images/seat-analysis.png new file mode 100644 index 00000000..4fa51c5c Binary files /dev/null and b/images/seat-analysis.png differ diff --git a/images/teams-comparison.png b/images/teams-comparison.png new file mode 100644 index 00000000..d3867509 Binary files /dev/null and b/images/teams-comparison.png differ