Skip to content

Commit 41fd917

Browse files
PavelLinearBPavelLinearB
authored
Merge pull request #689 from linear-b/measure-ai
Measure ai
2 parents dad4d12 + 1892857 commit 41fd917

File tree

9 files changed

+274
-6
lines changed

9 files changed

+274
-6
lines changed

‎docs/automations/integrations/ai-assistance/flag-ai-pr/README.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
title: Automation - Label AI-Assisted PRs
3+
description: Automatically apply labels to PRs based on AI assistance.
4+
category: [quality, genai, ai, quickstart]
5+
starter_kits: [genai]
6+
---
7+
# Automatically Label AI-Assisted PRs
8+
<!-- --8<-- [start:example]-->
9+
Automatically apply labels to PRs based on whether they were assisted by AI tools. Developers can indicate the specific AI tools or models used, or pre-add an AI-related label to skip the prompt.
10+
11+
=== "Ask Developers About AI Assistance"
12+
### Ask Developers About AI Assistance
13+
Prompt PR authors with a convenient checkbox survey to indicate which AI tools they used for the PR. Developers can pre-add any `🤖 ai-*` label to skip the question. The automation checks for existing labels before posting the prompt.
14+
15+
![Ask About AI Assistance](/automations/integrations/ai-assistance/flag-ai-pr/ask-ai-assistance.png)
16+
17+
!!! info "Configuration Description"
18+
Conditions:
19+
20+
* A PR is created, and no `🤖 ai-*` label is pre-applied.
21+
* The question hasn't been asked before.
22+
23+
Automation Actions:
24+
25+
* Post a comment with a checkbox survey about AI tools used.
26+
* Apply labels based on the checkboxes selected.
27+
28+
!!! example "Ask the PR author about AI assistance."
29+
```yaml+jinja
30+
--8<-- "docs/downloads/automation-library/integrations/ai-assistance/comment_ai_prompt.cm"
31+
```
32+
<div class="result" markdown>
33+
<span>
34+
[:octicons-download-24: Download this example as a CM file.](/downloads/automation-library/integrations/ai-assistance/comment_ai_prompt.cm){ .md-button }
35+
</span>
36+
</div>
37+
38+
39+
### Track AI Tools
40+
Add labels to the PR according to the tools that the developer checked in the micro-survey above
41+
![Label AI Assistance](/automations/integrations/ai-assistance/flag-ai-pr/label-ai-assistance.png)
42+
43+
!!! info "Configuration Description"
44+
Conditions:
45+
46+
* A PR with selected checkboxes for AI tools used.
47+
* Optional: Details about AI Service and Model specified.
48+
49+
Automation Actions:
50+
51+
* Apply labels for specific AI tools (e.g., `🤖 ai-copilot`, `🤖 ai-cursor`).
52+
* Apply labels for AI services and models if provided.
53+
54+
!!! example "Track AI tools, models and services."
55+
```yaml+jinja
56+
--8<-- "docs/downloads/automation-library/integrations/ai-assistance/track_ai_models.cm"
57+
```
58+
<div class="result" markdown>
59+
<span>
60+
[:octicons-download-24: Download this example as a CM file.](/downloads/automation-library/integrations/ai-assistance/track_ai_models.cm){ .md-button }
61+
</span>
62+
</div>
63+
64+
=== "Skip the survey for known users"
65+
### Ask Developers About AI Assistance
66+
The automation will not prompt known users for AI assistance. This allows a smoother experience for users who were predefined as AI users, to avoid answering the same questions again.
67+
!!! example "Skip the survey for known users."
68+
```yaml+jinja
69+
--8<-- "docs/downloads/automation-library/integrations/ai-assistance/comment_ai_prompt_with_known_users.cm"
70+
```
71+
<div class="result" markdown>
72+
<span>
73+
[:octicons-download-24: Download this example as a CM file.](/downloads/automation-library/integrations/ai-assistance/comment_ai_prompt_with_known_users.cm){ .md-button }
74+
</span>
75+
</div>
76+
77+
### Track AI Tools
78+
Add labels to the PR according to the tools that the developer checked in the micro-survey above
79+
![Label AI Assistance](/automations/integrations/ai-assistance/flag-ai-pr/label-ai-assistance.png)
80+
81+
!!! example "Track AI tools, models and services."
82+
```yaml+jinja
83+
--8<-- "docs/downloads/automation-library/integrations/ai-assistance/track_ai_models.cm"
84+
```
85+
<div class="result" markdown>
86+
<span>
87+
[:octicons-download-24: Download this example as a CM file.](/downloads/automation-library/integrations/ai-assistance/track_ai_models.cm){ .md-button }
88+
</span>
89+
</div>
90+
91+
<!-- --8<-- [end:example]-->

‎docs/automations/integrations/ai-assistance/flag-ai-pr/ask-ai-assistance.png‎

168 KB
Loading

‎docs/automations/integrations/ai-assistance/flag-ai-pr/label-ai-assistance.png‎

14.6 KB
Loading

‎docs/downloads/automation-library/integrations/ai-assistance/comment_ai_prompt.cm‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# -*- mode: yaml -*-
2+
3+
manifest:
4+
version: 1.0
5+
6+
triggers:
7+
on:
8+
- pr_created
9+
10+
automations:
11+
comment_ai_assistance_prompt:
12+
# Post a comment for all PRs to prompt the PR author to indicate whether they used AI to assist coding in this PR
13+
# Only do this if there's no existing AI-related label and we haven't asked before
14+
if:
15+
- {{ pr.labels | match(regex=r/🤖 ai-*/) | nope }}
16+
- {{ pr.comments | filter(attr='commenter', term='gitstream-cm') | filter(attr='content', regex=r/Please mark which AI tools you used/) | nope }}
17+
run:
18+
- action: add-comment@v1
19+
args:
20+
comment: |
21+
Please mark which AI tools you used for this PR by checking the appropriate boxes:
22+
23+
- [ ] GitHub Copilot
24+
- [ ] Cursor
25+
- [ ] Tabnine
26+
- [ ] JetBrains AI Assistant
27+
- [ ] VSCode IntelliCode
28+
- [ ] ChatGPT
29+
- [ ] Claude
30+
- [ ] Gemini
31+
- [ ] Other AI tool
32+
- [ ] No AI tools were used
33+
34+
Tip: If you want to avoid this comment in the future, you can add a label of the format `🤖 ai-*` when creating your PR.
35+

‎docs/downloads/automation-library/integrations/ai-assistance/comment_ai_prompt_with_known_users.cm‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# -*- mode: yaml -*-
2+
3+
manifest:
4+
version: 1.0
5+
triggers:
6+
on:
7+
- pr_created
8+
9+
automations:
10+
comment_ai_assistance_prompt:
11+
# Post a comment for all PRs to prompt the PR author to indicate whether they used AI to assist coding in this PR
12+
# Only do this if there's no existing AI-related label and we haven't asked before
13+
if:
14+
- {{ pr.labels | match(regex=r/🤖 ai-*/) | nope }}
15+
- {{ pr.author | match(list=copilot_contributors) | nope }}
16+
- {{ pr.author | match(list=cursor_contributors) | nope }}
17+
run:
18+
- action: add-comment@v1
19+
args:
20+
comment: |
21+
Please mark which AI tools you used for this PR by checking the appropriate boxes:
22+
23+
- [ ] GitHub Copilot
24+
- [ ] Cursor
25+
- [ ] Tabnine
26+
- [ ] JetBrains AI Assistant
27+
- [ ] VSCode IntelliCode
28+
- [ ] ChatGPT
29+
- [ ] Claude
30+
- [ ] Gemini
31+
- [ ] Other AI tool
32+
- [ ] No AI tools were used
33+
34+
**Tip**: If you want to avoid this comment in the future, add a label of the format `🤖 ai-*` when creating your PR, or ask your admin to add you to the pre-defined lists of known users
35+
36+
label_copilot_known_users:
37+
# For all PRs authored by someone who is specified in the genai_contributors list
38+
if:
39+
- {{ pr.author | match(list=copilot_contributors) | some }}
40+
# Apply a label indicating the user has adopted Copilot
41+
run:
42+
- action: add-label@v1
43+
args:
44+
label: '🤖 ai-copilot'
45+
46+
label_cursor_known_users:
47+
# For all PRs authored by someone who is specified in the genai_contributors list
48+
if:
49+
- {{ pr.author | match(list=cursor_contributors) | some }}
50+
# Apply a label indicating the user has adopted Copilot
51+
run:
52+
- action: add-label@v1
53+
args:
54+
label: '🤖 ai-cursor'
55+
56+
copilot_contributors:
57+
- username1
58+
- username2
59+
- usernameN
60+
61+
cursor_contributors:
62+
- username1
63+
- username2
64+
- usernameN
65+

‎docs/downloads/automation-library/integrations/ai-assistance/track_ai_models.cm‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# -*- mode: yaml -*-
2+
3+
manifest:
4+
version: 1.0
5+
6+
automations:
7+
label_ai_tools_by_checkbox:
8+
# Apply labels based on the checkboxes selected in the PR comment
9+
if:
10+
- {{ pr.comments | filter(attr='content', regex=r/\- \[x\] GitHub Copilot/) | some }}
11+
run:
12+
- action: add-label@v1
13+
args:
14+
label: "🤖 ai-copilot"
15+
16+
label_ai_cursor:
17+
if:
18+
- {{ pr.comments | filter(attr='content', regex=r/\- \[x\] Cursor/) | some }}
19+
run:
20+
- action: add-label@v1
21+
args:
22+
label: "🤖 ai-cursor"
23+
24+
label_ai_tabnine:
25+
if:
26+
- {{ pr.comments | filter(attr='content', regex=r/\- \[x\] Tabnine/) | some }}
27+
run:
28+
- action: add-label@v1
29+
args:
30+
label: "🤖 ai-tabnine"
31+
32+
label_ai_jetbrains:
33+
if:
34+
- {{ pr.comments | filter(attr='content', regex=r/\- \[x\] JetBrains AI Assistant/) | some }}
35+
run:
36+
- action: add-label@v1
37+
args:
38+
label: "🤖 ai-jetbrains"
39+
40+
label_ai_intellicode:
41+
if:
42+
- {{ pr.comments | filter(attr='content', regex=r/\- \[x\] VSCode IntelliCode/) | some }}
43+
run:
44+
- action: add-label@v1
45+
args:
46+
label: "🤖 ai-intellicode"
47+
48+
label_ai_chatgpt:
49+
if:
50+
- {{ pr.comments | filter(attr='content', regex=r/\- \[x\] ChatGPT/) | some }}
51+
run:
52+
- action: add-label@v1
53+
args:
54+
label: "🤖 ai-chatgpt"
55+
56+
label_ai_other:
57+
if:
58+
- {{ pr.comments | filter(attr='content', regex=r/\- \[x\] Other AI tool/) | some }}
59+
run:
60+
- action: add-label@v1
61+
args:
62+
label: "🤖 ai-other"
63+
64+
label_ai_none:
65+
if:
66+
- {{ pr.comments | filter(attr='content', regex=r/\- \[x\] No AI tools were used/) | some }}
67+
run:
68+
- action: add-label@v1
69+
args:
70+
label: "🤖 ai-none"
71+

‎docs/downloads/automation-library/integrations/copilot/comment_copilot_prompt.cm‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
1-
-*- mode: yaml -*-
1+
# -*- mode: yaml -*-
22

33
manifest:
44
version: 1.0
55

66
on:
77
- pr_created
8-
8+
99
automations:
1010
comment_copilot_prompt:
1111
# Post a comment for all PRs to prompt the PR author to indicate whether they used Copilot to assist coding in this PR

‎docs/downloads/automation-library/integrations/copilot/label_copilot_by_contributors.cm‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,17 @@ manifest:
44
version: 1.0
55

66
automations:
7-
label_genai:
7+
label_copilot_by_contributors:
88
# For all PRs authored by someone who is specified in the genai_contributors list
99
if:
10-
- {{ pr.author | match(list=genai_contributors) | some }}
10+
- {{ pr.author | match(list=genai_contributors) | some }}
1111
# Apply a label indicating the user has adopted Copilot
1212
run:
1313
- action: add-label@v1
1414
args:
15-
label: '🤖 Copilot'
15+
label: '🤖 ai-copilot'
1616

1717
genai_contributors:
1818
- username1
1919
- username2
20-
- etc
20+
- usernameN

‎docs/integrations/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,12 @@ visible: false
77
<!-- --8<-- [start:integrations]-->
88
<div class="integrations-list" markdown="1">
99

10+
<div class="integrations-card" markdown="1">
11+
<div class="integrations-card-title" markdown="1">
12+
[:material-assistant: AI Assistance](/automations/integrations/ai-assistance/flag-ai-pr)
13+
</div>
14+
</div>
15+
1016
<div class="integrations-card" markdown="1">
1117
<div class="integrations-card-title" markdown="1">
1218
<a href=/integrations/linearb>![LinearB](/downloads/images/linearb-symbol-dark.png#only-light) ![LinearB](/downloads/images/linearb-symbol-white.png#only-dark) LinearB</a>

0 commit comments

Comments
 (0)