[Feature][Plugin] Add Python Plugin for GitHub Copilot Enterprise & User Usage Metrics

[ansri-abodey]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Use case

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

I would like to contribute a new plugin to DevLake that ingests GitHub Copilot Usage Metrics.

To fully understand the impact of AI on our development lifecycle, we need both high-level adoption trends and granular user-level data. I plan to use the Copilot Usage Metrics API (2022-11-28) to collect this data.

Strategic Value: Why Measure This?

Integrating Copilot metrics allows us to answer critical questions about the ROI of AI coding assistants by correlating them with existing DORA and engineering metrics:

1. Velocity & Lead Time:
   • Hypothesis: High Copilot usage (acceptance rates) should reduce Lead Time for Changes by speeding up the "coding" phase.
   • Metric: Correlate total_lines_accepted with PR Cycle Time.
2. Quality & Stability:
   • Hypothesis: We need to monitor if increased AI speed impacts code quality.
   • Metric: Correlate copilot_active_users with Change Failure Rate (CFR) and Bug Count.
3. Adoption vs. Performance:
   • Hypothesis: Teams with higher Copilot adoption perform better.
   • Metric: Compare Deployment Frequency between high-adoption and low-adoption teams (using the User-level breakdown).

Data Sources

I will target the following two report endpoints:

1. Enterprise Metrics (Aggregated)

   • Endpoint: GET /enterprises/{enterprise}/copilot/metrics/reports/enterprise-1-day
   • Purpose: Provides high-level daily summaries of active users, acceptance rates, and chat interactions across the entire enterprise.
   • Docs: Get Copilot enterprise usage metrics

2. User Metrics (Granular)

   • Endpoint: GET /enterprises/{enterprise}/copilot/metrics/reports/users-1-day
   • Purpose: Provides detailed usage stats per user (e.g., lines suggested vs. accepted, editor used, last activity). This allows for correlation with individual developer output.
   • Docs: Get Copilot users usage metrics

Note: I will use the *-1-day endpoints to allow DevLake to incrementally collect and store daily historical data.

Implementation Plan

I propose implementing this as a Python Plugin using the pydevlake framework.

Proposed Scope:

1. Create a new Python plugin copilot.
2. Stream 1: CopilotEnterpriseMetricsStream:
   • Fetches the daily enterprise report URL.
   • Downloads and parses the JSON to store daily aggregated stats.
3. Stream 2: CopilotUserMetricsStream:
   • Fetches the daily user report URL.
   • Downloads and parses the JSON to store individual user activity.

Questions for Maintainers

1. Domain Modeling:
   • Should the Enterprise Metrics map to a new table (e.g., copilot_enterprise_daily)?
   • Should the User Metrics map to a new table (e.g., copilot_user_daily)?### Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

I would like to contribute a new plugin to DevLake that ingests GitHub Copilot Usage Metrics.

To fully understand the impact of AI on our development lifecycle, we need both high-level adoption trends and granular user-level data. I plan to use the Copilot Usage Metrics API (2022-11-28) to collect this data.

Strategic Value: Why Measure This?

Integrating Copilot metrics allows us to answer critical questions about the ROI of AI coding assistants by correlating them with existing DORA and engineering metrics:

1. Velocity & Lead Time:
   • Hypothesis: High Copilot usage (acceptance rates) should reduce Lead Time for Changes by speeding up the "coding" phase.
   • Metric: Correlate total_lines_accepted with PR Cycle Time.
2. Quality & Stability:
   • Hypothesis: We need to monitor if increased AI speed impacts code quality.
   • Metric: Correlate copilot_active_users with Change Failure Rate (CFR) and Bug Count.
3. Adoption vs. Performance:
   • Hypothesis: Teams with higher Copilot adoption perform better.
   • Metric: Compare Deployment Frequency between high-adoption and low-adoption teams (using the User-level breakdown).

Data Sources

I will target the following two report endpoints:

1. Enterprise Metrics (Aggregated)

   • Endpoint: GET /enterprises/{enterprise}/copilot/metrics/reports/enterprise-1-day
   • Purpose: Provides high-level daily summaries of active users, acceptance rates, and chat interactions across the entire enterprise.
   • Docs: Get Copilot enterprise usage metrics

2. User Metrics (Granular)

   • Endpoint: GET /enterprises/{enterprise}/copilot/metrics/reports/users-1-day
   • Purpose: Provides detailed usage stats per user (e.g., lines suggested vs. accepted, editor used, last activity). This allows for correlation with individual developer output.
   • Docs: Get Copilot users usage metrics

Note: I will use the *-1-day endpoints to allow DevLake to incrementally collect and store daily historical data.

Implementation Plan

I propose implementing this as a Python Plugin using the pydevlake framework.

Proposed Scope:

1. Create a new Python plugin copilot.
2. Stream 1: CopilotEnterpriseMetricsStream:
   • Fetches the daily enterprise report URL.
   • Downloads and parses the JSON to store daily aggregated stats.
3. Stream 2: CopilotUserMetricsStream:
   • Fetches the daily user report URL.
   • Downloads and parses the JSON to store individual user activity.

Questions for Maintainers

1. Domain Modeling:
   • Should the Enterprise Metrics map to a new table (e.g., copilot_enterprise_daily)?
   • Should the User Metrics map to a new table (e.g., copilot_user_daily)?

Description

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hi @ansri-abodey! I'm Dosu and I’m helping the incubator-devlake team.

Yes, the established convention in DevLake plugins—especially with pydevlake—is to map each distinct data stream to its own dedicated table for clarity and normalization. For your Copilot plugin, it's recommended to create a new table for enterprise metrics (e.g., copilot_enterprise_daily) and another for user metrics (e.g., copilot_user_daily). This approach is used in plugins like q_dev, which maps user metrics to a table named _tool_q_dev_user_data with fields for user ID, date, display name, and various metric counts, supporting granular analysis and efficient queries [example].

In pydevlake, table naming typically follows the pattern _tool_{plugin_name}_{plural_entity} [reference]. For best results, include primary keys and indexes on fields like connection_id, user_id, and date to facilitate joins and cross-metric analysis with DORA metrics and other DevLake data [reference].

No conflicting guidance or alternative conventions were found in recent issues or threads. This modeling approach will make your plugin consistent with current DevLake practices and support robust analytics.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[klesh]

Sounds good to me! I'm excited to see someone start measuring AI—it's an important metric, in my opinion. Feel free to reach out if you need any help!

[klesh]

I’d recommend building the plugin in Go instead of Python—Python support is limited here, and I’m less familiar with it.

[ansri-abodey]

I’d recommend building the plugin in Go instead of Python—Python support is limited here, and I’m less familiar with it.

Hmmm.. I am more of a python developer. But I guess I can use copilot to help me based on the other GO plugins. Might give Go a Go :D

[klesh]

I’d recommend building the plugin in Go instead of Python—Python support is limited here, and I’m less familiar with it.

Hmmm.. I am more of a python developer. But I guess I can use copilot to help me based on the other GO plugins. Might give Go a Go :D

Nice move!

[ewega]

Would love to support contributing to this, I started working on it in a fork. @ansri-abodey Do you have a public fork yet? Otherwise I'm chugging along at it here: ewega#9

Branch: #8674

[mieliespoor]

@ewega I see you named your plugin copilot. I would suggest adding github in some shape or form to the name as copilot is too generic and could possibly clash with other services. For example - Microsoft names more than one service as copilot, or rather, attaches the copilot name to it, and all are distinctly different to the other.

[ansri-abodey]

Would love to support contributing to this, I started working on it in a fork. @ansri-abodey Do you have a public fork yet? Otherwise I'm chugging along at it here: ewega#9

Thanks @ewega. Haven’t started on this yet as I was away. Will look at your fork. Thanks.

[ewega]

@mieliespoor Noted and good call. Will rename it

[ewega]

@ansri-abodey Please refer to the above PR instead, one I previously shared I discarded: #8674