Skip to content

Commit 3f8957d

Browse files
leigaolkaranA-aws
leigaol
authored and
karanA-aws
committed
telemetry(amazonq): calculate % of non-generated (user-written) code aws#5991
## Problem With the release of many Q features(Inline Suggestion, chat, inline chat, /dev, /test, /doc, /review, /transform), we need to know the % code written by all Q features. This requires calculating and reporting the user written code. The reporting of the code contribution of each Q features was already implemented. ## Solution Calculate and report the user written code for each language by listening to document change events while Q is not making changes to the editor. We add flags to know whether Q is making temporary changes for suggestion rendering or Q suggestion is accepted, by doing so, the document change events are coming from the user. We ignore certain document changes when their length of new characters exceeds 50. Previous data driven research has shown that user tend to copy a huge file from one place to another, making the user written code count skyrocketing but that is actually some existing code not written by the user. We plan to first collect data from IDEs and let it run in the background in shadow mode before we finish the service side aggregation, fix possible bugs and eventually present the AI code written % to the customers. Note: The JB PR aws/aws-toolkit-jetbrains#5215. The JB implementation depends on a reliable JB internal message bus to pass information. Using VSC event listener might mess up the boolean state of Q editing or not.
1 parent 70b533d commit 3f8957d

File tree

18 files changed

+431
-3
lines changed

18 files changed

+431
-3
lines changed

‎packages/amazonq/src/inlineChat/controller/inlineChatController.ts

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import { computeDecorations } from '../decorations/computeDecorations'
1313
import { CodelensProvider } from '../codeLenses/codeLenseProvider'
1414
import { PromptMessage, ReferenceLogController } from 'aws-core-vscode/codewhispererChat'
1515
import { CodeWhispererSettings } from 'aws-core-vscode/codewhisperer'
16+
import { UserWrittenCodeTracker } from 'aws-core-vscode/codewhisperer'
1617
import {
1718
codicon,
1819
getIcon,
@@ -84,6 +85,7 @@ export class InlineChatController {
8485
await this.updateTaskAndLenses(task)
8586
this.referenceLogController.addReferenceLog(task.codeReferences, task.replacement ? task.replacement : '')
8687
await this.reset()
88+
UserWrittenCodeTracker.instance.onQFinishesEdits()
8789
}
8890

8991
public async rejectAllChanges(task = this.task, userInvoked: boolean): Promise<void> {
@@ -199,7 +201,7 @@ export class InlineChatController {
199201
getLogger().info('inlineQuickPick query is empty')
200202
return
201203
}
202-
204+
UserWrittenCodeTracker.instance.onQStartsMakingEdits()
203205
this.userQuery = query
204206
await textDocumentUtil.addEofNewline(editor)
205207
this.task = await this.createTask(query, editor.document, editor.selection)

‎packages/amazonq/test/unit/codewhisperer/tracker/userWrittenCodeTracker.test.ts

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
/*!
2+
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
3+
* SPDX-License-Identifier: Apache-2.0
4+
*/
5+
6+
import assert from 'assert'
7+
import * as sinon from 'sinon'
8+
import * as vscode from 'vscode'
9+
import { UserWrittenCodeTracker, TelemetryHelper, AuthUtil } from 'aws-core-vscode/codewhisperer'
10+
import { createMockDocument, resetCodeWhispererGlobalVariables } from 'aws-core-vscode/test'
11+
12+
describe('userWrittenCodeTracker', function () {
13+
describe('isActive()', function () {
14+
afterEach(async function () {
15+
await resetCodeWhispererGlobalVariables()
16+
UserWrittenCodeTracker.instance.reset()
17+
sinon.restore()
18+
})
19+
20+
it('inactive case: telemetryEnable = true, isConnected = false', function () {
21+
sinon.stub(TelemetryHelper.instance, 'isTelemetryEnabled').returns(true)
22+
sinon.stub(AuthUtil.instance, 'isConnected').returns(false)
23+
assert.strictEqual(UserWrittenCodeTracker.instance.isActive(), false)
24+
})
25+
26+
it('inactive case: telemetryEnabled = false, isConnected = false', function () {
27+
sinon.stub(TelemetryHelper.instance, 'isTelemetryEnabled').returns(false)
28+
sinon.stub(AuthUtil.instance, 'isConnected').returns(false)
29+
assert.strictEqual(UserWrittenCodeTracker.instance.isActive(), false)
30+
})
31+
32+
it('active case: telemetryEnabled = true, isConnected = true', function () {
33+
sinon.stub(TelemetryHelper.instance, 'isTelemetryEnabled').returns(true)
34+
sinon.stub(AuthUtil.instance, 'isConnected').returns(true)
35+
assert.strictEqual(UserWrittenCodeTracker.instance.isActive(), true)
36+
})
37+
})
38+
39+
describe('onDocumentChange', function () {
40+
let tracker: UserWrittenCodeTracker | undefined
41+
42+
beforeEach(async function () {
43+
await resetCodeWhispererGlobalVariables()
44+
tracker = UserWrittenCodeTracker.instance
45+
if (tracker) {
46+
sinon.stub(tracker, 'isActive').returns(true)
47+
}
48+
})
49+
50+
afterEach(function () {
51+
sinon.restore()
52+
UserWrittenCodeTracker.instance.reset()
53+
})
54+
55+
it('Should skip when content change size is more than 50', function () {
56+
if (!tracker) {
57+
assert.fail()
58+
}
59+
tracker.onQFeatureInvoked()
60+
tracker.onTextDocumentChange({
61+
reason: undefined,
62+
document: createMockDocument(),
63+
contentChanges: [
64+
{
65+
range: new vscode.Range(0, 0, 0, 600),
66+
rangeOffset: 0,
67+
rangeLength: 600,
68+
text: 'def twoSum(nums, target):\nfor '.repeat(20),
69+
},
70+
],
71+
})
72+
assert.strictEqual(tracker.getUserWrittenCharacters('python'), 0)
73+
assert.strictEqual(tracker.getUserWrittenLines('python'), 0)
74+
})
75+
76+
it('Should not skip when content change size is less than 50', function () {
77+
if (!tracker) {
78+
assert.fail()
79+
}
80+
tracker.onQFeatureInvoked()
81+
tracker.onTextDocumentChange({
82+
reason: undefined,
83+
document: createMockDocument(),
84+
contentChanges: [
85+
{
86+
range: new vscode.Range(0, 0, 0, 49),
87+
rangeOffset: 0,
88+
rangeLength: 49,
89+
text: 'a = 123'.repeat(7),
90+
},
91+
],
92+
})
93+
tracker.onTextDocumentChange({
94+
reason: undefined,
95+
document: createMockDocument('', 'test.java', 'java'),
96+
contentChanges: [
97+
{
98+
range: new vscode.Range(0, 0, 1, 3),
99+
rangeOffset: 0,
100+
rangeLength: 11,
101+
text: 'a = 123\nbcd',
102+
},
103+
],
104+
})
105+
assert.strictEqual(tracker.getUserWrittenCharacters('python'), 49)
106+
assert.strictEqual(tracker.getUserWrittenLines('python'), 0)
107+
assert.strictEqual(tracker.getUserWrittenCharacters('java'), 11)
108+
assert.strictEqual(tracker.getUserWrittenLines('java'), 1)
109+
assert.strictEqual(tracker.getUserWrittenLines('cpp'), 0)
110+
})
111+
112+
it('Should skip when Q is editing', function () {
113+
if (!tracker) {
114+
assert.fail()
115+
}
116+
tracker.onQFeatureInvoked()
117+
tracker.onQStartsMakingEdits()
118+
tracker.onTextDocumentChange({
119+
reason: undefined,
120+
document: createMockDocument(),
121+
contentChanges: [
122+
{
123+
range: new vscode.Range(0, 0, 0, 30),
124+
rangeOffset: 0,
125+
rangeLength: 30,
126+
text: 'def twoSum(nums, target):\nfor',
127+
},
128+
],
129+
})
130+
tracker.onQFinishesEdits()
131+
tracker.onTextDocumentChange({
132+
reason: undefined,
133+
document: createMockDocument(),
134+
contentChanges: [
135+
{
136+
range: new vscode.Range(0, 0, 0, 2),
137+
rangeOffset: 0,
138+
rangeLength: 2,
139+
text: '\na',
140+
},
141+
],
142+
})
143+
assert.strictEqual(tracker.getUserWrittenCharacters('python'), 2)
144+
assert.strictEqual(tracker.getUserWrittenLines('python'), 1)
145+
})
146+
147+
it('Should not reduce tokens when delete', function () {
148+
if (!tracker) {
149+
assert.fail()
150+
}
151+
const doc = createMockDocument('import math', 'test.py', 'python')
152+
153+
tracker.onQFeatureInvoked()
154+
tracker.onTextDocumentChange({
155+
reason: undefined,
156+
document: doc,
157+
contentChanges: [
158+
{
159+
range: new vscode.Range(0, 0, 0, 1),
160+
rangeOffset: 0,
161+
rangeLength: 0,
162+
text: 'a',
163+
},
164+
],
165+
})
166+
tracker.onTextDocumentChange({
167+
reason: undefined,
168+
document: doc,
169+
contentChanges: [
170+
{
171+
range: new vscode.Range(0, 0, 0, 1),
172+
rangeOffset: 0,
173+
rangeLength: 0,
174+
text: 'b',
175+
},
176+
],
177+
})
178+
assert.strictEqual(tracker.getUserWrittenCharacters('python'), 2)
179+
tracker.onTextDocumentChange({
180+
reason: undefined,
181+
document: doc,
182+
contentChanges: [
183+
{
184+
range: new vscode.Range(0, 0, 0, 1),
185+
rangeOffset: 1,
186+
rangeLength: 1,
187+
text: '',
188+
},
189+
],
190+
})
191+
assert.strictEqual(tracker.getUserWrittenCharacters('python'), 2)
192+
})
193+
})
194+
})

‎packages/core/src/amazonq/commons/controllers/contentController.ts

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import {
1616
getSelectionFromRange,
1717
} from '../../../shared/utilities/textDocumentUtilities'
1818
import { extractFileAndCodeSelectionFromMessage, fs, getErrorMsg, ToolkitError } from '../../../shared'
19+
import { UserWrittenCodeTracker } from '../../../codewhisperer/tracker/userWrittenCodeTracker'
1920

2021
export class ContentProvider implements vscode.TextDocumentContentProvider {
2122
constructor(private uri: vscode.Uri) {}
@@ -41,6 +42,7 @@ export class EditorContentController {
4142
) {
4243
const editor = window.activeTextEditor
4344
if (editor) {
45+
UserWrittenCodeTracker.instance.onQStartsMakingEdits()
4446
const cursorStart = editor.selection.active
4547
const indentRange = new vscode.Range(new vscode.Position(cursorStart.line, 0), cursorStart)
4648
// use the user editor intent if the position to the left of cursor is just space or tab
@@ -66,9 +68,11 @@ export class EditorContentController {
6668
if (appliedEdits) {
6769
trackCodeEdit(editor, cursorStart)
6870
}
71+
UserWrittenCodeTracker.instance.onQFinishesEdits()
6972
},
7073
(e) => {
7174
getLogger().error('TextEditor.edit failed: %s', (e as Error).message)
75+
UserWrittenCodeTracker.instance.onQFinishesEdits()
7276
}
7377
)
7478
}
@@ -97,6 +101,7 @@ export class EditorContentController {
97101

98102
if (filePath && message?.code?.trim().length > 0 && selection) {
99103
try {
104+
UserWrittenCodeTracker.instance.onQStartsMakingEdits()
100105
const doc = await vscode.workspace.openTextDocument(filePath)
101106

102107
const code = getIndentedCode(message, doc, selection)
@@ -130,6 +135,8 @@ export class EditorContentController {
130135
const wrappedError = ChatDiffError.chain(error, `Failed to Accept Diff`, { code: chatDiffCode })
131136
getLogger().error('%s: Failed to open diff view %s', chatDiffCode, getErrorMsg(wrappedError, true))
132137
throw wrappedError
138+
} finally {
139+
UserWrittenCodeTracker.instance.onQFinishesEdits()
133140
}
134141
}
135142
}

‎packages/core/src/amazonqFeatureDev/client/featureDev.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ import { createCodeWhispererChatStreamingClient } from '../../shared/clients/cod
2525
import { getClientId, getOptOutPreference, getOperatingSystem } from '../../shared/telemetry/util'
2626
import { extensionVersion } from '../../shared/vscode/env'
2727
import apiConfig = require('./codewhispererruntime-2022-11-11.json')
28+
import { UserWrittenCodeTracker } from '../../codewhisperer'
2829
import {
2930
FeatureDevCodeAcceptanceEvent,
3031
FeatureDevCodeGenerationEvent,
@@ -260,6 +261,7 @@ export class FeatureDevClient {
260261
references?: CodeReference[]
261262
}
262263
}
264+
UserWrittenCodeTracker.instance.onQFeatureInvoked()
263265

264266
const newFileContents: { zipFilePath: string; fileContent: string }[] = []
265267
for (const [filePath, fileContent] of Object.entries(newFiles)) {

‎packages/core/src/amazonqTest/chat/controller/controller.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import {
2121
TestGenerationBuildStep,
2222
testGenState,
2323
unitTestGenerationCancelMessage,
24+
UserWrittenCodeTracker,
2425
} from '../../../codewhisperer'
2526
import {
2627
fs,
@@ -664,12 +665,14 @@ export class TestController {
664665
acceptedLines = acceptedLines < 0 ? 0 : acceptedLines
665666
acceptedChars -= originalContent.length
666667
acceptedChars = acceptedChars < 0 ? 0 : acceptedChars
668+
UserWrittenCodeTracker.instance.onQStartsMakingEdits()
667669
const document = await vscode.workspace.openTextDocument(absolutePath)
668670
await applyChanges(
669671
document,
670672
new vscode.Range(document.lineAt(0).range.start, document.lineAt(document.lineCount - 1).range.end),
671673
updatedContent
672674
)
675+
UserWrittenCodeTracker.instance.onQFinishesEdits()
673676
} else {
674677
await fs.writeFile(absolutePath, updatedContent)
675678
}
@@ -831,6 +834,7 @@ export class TestController {
831834
const chatRequest = triggerPayloadToChatRequest(triggerPayload)
832835
const client = await createCodeWhispererChatStreamingClient()
833836
const response = await client.generateAssistantResponse(chatRequest)
837+
UserWrittenCodeTracker.instance.onQFeatureInvoked()
834838
await this.messenger.sendAIResponse(
835839
response,
836840
session,

‎packages/core/src/codewhisperer/activation.ts

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ import { SecurityIssueTreeViewProvider } from './service/securityIssueTreeViewPr
9999
import { setContext } from '../shared/vscode/setContext'
100100
import { syncSecurityIssueWebview } from './views/securityIssue/securityIssueWebview'
101101
import { detectCommentAboveLine } from '../shared/utilities/commentUtils'
102+
import { UserWrittenCodeTracker } from './tracker/userWrittenCodeTracker'
102103

103104
let localize: nls.LocalizeFunc
104105

@@ -552,7 +553,7 @@ export async function activate(context: ExtContext): Promise<void> {
552553
}
553554

554555
CodeWhispererCodeCoverageTracker.getTracker(e.document.languageId)?.countTotalTokens(e)
555-
556+
UserWrittenCodeTracker.instance.onTextDocumentChange(e)
556557
/**
557558
* Handle this keystroke event only when
558559
* 1. It is not a backspace

‎packages/core/src/codewhisperer/client/user-service-2.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -626,7 +626,9 @@
626626
"timestamp": { "shape": "Timestamp" },
627627
"unmodifiedAcceptedCharacterCount": { "shape": "PrimitiveInteger" },
628628
"totalNewCodeCharacterCount": { "shape": "PrimitiveInteger" },
629-
"totalNewCodeLineCount": { "shape": "PrimitiveInteger" }
629+
"totalNewCodeLineCount": { "shape": "PrimitiveInteger" },
630+
"userWrittenCodeCharacterCount": { "shape": "PrimitiveInteger" },
631+
"userWrittenCodeLineCount": { "shape": "PrimitiveInteger" }
630632
}
631633
},
632634
"CodeFixAcceptanceEvent": {

‎packages/core/src/codewhisperer/commands/basicCommands.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ import { cancel, confirm } from '../../shared'
6666
import { startCodeFixGeneration } from './startCodeFixGeneration'
6767
import { DefaultAmazonQAppInitContext } from '../../amazonq/apps/initContext'
6868
import path from 'path'
69+
import { UserWrittenCodeTracker } from '../tracker/userWrittenCodeTracker'
6970
import { parsePatch } from 'diff'
7071

7172
const MessageTimeOut = 5_000
@@ -451,6 +452,7 @@ export const applySecurityFix = Commands.declare(
451452
}
452453
let languageId = undefined
453454
try {
455+
UserWrittenCodeTracker.instance.onQStartsMakingEdits()
454456
const document = await vscode.workspace.openTextDocument(targetFilePath)
455457
languageId = document.languageId
456458
const updatedContent = await getPatchedCode(targetFilePath, suggestedFix.code)
@@ -565,6 +567,7 @@ export const applySecurityFix = Commands.declare(
565567
applyFixTelemetryEntry.result,
566568
!!targetIssue.suggestedFixes.length
567569
)
570+
UserWrittenCodeTracker.instance.onQFinishesEdits()
568571
}
569572
}
570573
)

‎packages/core/src/codewhisperer/commands/onInlineAcceptance.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ import { RecommendationService } from '../service/recommendationService'
3232
import { Container } from '../service/serviceContainer'
3333
import { telemetry } from '../../shared/telemetry'
3434
import { TelemetryHelper } from '../util/telemetryHelper'
35+
import { UserWrittenCodeTracker } from '../tracker/userWrittenCodeTracker'
3536

3637
export const acceptSuggestion = Commands.declare(
3738
'aws.amazonq.accept',
@@ -126,6 +127,7 @@ export async function onInlineAcceptance(acceptanceEntry: OnRecommendationAccept
126127
acceptanceEntry.editor.document.getText(insertedCoderange),
127128
acceptanceEntry.editor.document.fileName
128129
)
130+
UserWrittenCodeTracker.instance.onQFinishesEdits()
129131
if (acceptanceEntry.references !== undefined) {
130132
const referenceLog = ReferenceLogViewProvider.getReferenceLog(
131133
acceptanceEntry.recommendation,

‎packages/core/src/codewhisperer/index.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,3 +102,4 @@ export * as CodeWhispererConstants from '../codewhisperer/models/constants'
102102
export { getSelectedCustomization, setSelectedCustomization, baseCustomization } from './util/customizationUtil'
103103
export { Container } from './service/serviceContainer'
104104
export * from './util/gitUtil'
105+
export { UserWrittenCodeTracker } from './tracker/userWrittenCodeTracker'

0 commit comments

Comments
 (0)