Skip to content

telemetry(amazonq): add telemetry for falcon features #6691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jpinkney-aws merged 4 commits into from
Mar 3, 2025
Merged

telemetry(amazonq): add telemetry for falcon features #6691

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8c3db34
telemetry(amazonq): add telemetry for additional context
avi-alpert Feb 27, 2025
e978cfd
telemetry: rename descriptions
avi-alpert Feb 28, 2025
1fa0415
fix: circular dependencies
avi-alpert Feb 28, 2025
68de758
telemetry: switch createSavedPrompt to ui_click
avi-alpert Feb 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion packages/core/src/codewhispererChat/controllers/chat/controller.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ const aditionalContentNameLimit = 1024
// temporary limit for @workspace and @file combined context length
const contextMaxLength = 40_000

const getUserPromptsDirectory = () => {
export const getUserPromptsDirectory = () => {
return path.join(fs.getUserHomeDir(), '.aws', 'amazonq', 'prompts')
}

Expand Down Expand Up @@ -497,6 +497,7 @@ export class ChatController {
.map(([name]) => ({
command: path.basename(name, promptFileExtension),
icon: 'magic' as MynahIconsType,
id: 'prompt',
route: [userPromptsDirectory, name],
}))
)
Expand Down Expand Up @@ -525,13 +526,15 @@ export class ChatController {
command: path.basename(contextCommandItem.relativePath),
description: path.join(wsFolderName, contextCommandItem.relativePath),
route: [contextCommandItem.workspaceFolder, contextCommandItem.relativePath],
id: 'file',
icon: 'file' as MynahIconsType,
})
} else {
folderCmd.children?.[0].commands.push({
command: path.basename(contextCommandItem.relativePath),
description: path.join(wsFolderName, contextCommandItem.relativePath),
route: [contextCommandItem.workspaceFolder, contextCommandItem.relativePath],
id: 'folder',
icon: 'folder' as MynahIconsType,
})
}
Expand Down Expand Up @@ -581,6 +584,7 @@ export class ChatController {
await fs.writeFile(newFilePath, newFileContent)
const newFileDoc = await vscode.workspace.openTextDocument(newFilePath)
await vscode.window.showTextDocument(newFileDoc)
telemetry.amazonq_createSavedPrompt.emit()
}
}

Expand Down Expand Up @@ -906,6 +910,7 @@ export class ChatController {
})
)
}
triggerPayload.workspaceRulesCount = workspaceRules.length

// Add context commands added by user to context
if (triggerPayload.context !== undefined && triggerPayload.context.length > 0) {
Expand All @@ -931,6 +936,12 @@ export class ChatController {

let currentContextLength = 0
triggerPayload.additionalContents = []
triggerPayload.additionalContextLengths = this.telemetryHelper.getContextLengths(prompts)
triggerPayload.truncatedAdditionalContextLengths = {
fileContextLength: 0,
promptContextLength: 0,
ruleContextLength: 0,
}
for (const prompt of prompts.slice(0, 20)) {
// Todo: add mechanism for sorting/prioritization of additional context
const entry = {
Expand All @@ -946,6 +957,16 @@ export class ChatController {
getLogger().warn(`Selected context exceeds context size limit: ${entry.description} `)
break
}

const contextType = this.telemetryHelper.getContextType(prompt)
if (contextType === 'rule') {
triggerPayload.truncatedAdditionalContextLengths.ruleContextLength += entry.innerContext.length
} else if (contextType === 'prompt') {
triggerPayload.truncatedAdditionalContextLengths.promptContextLength += entry.innerContext.length
} else if (contextType === 'file') {
triggerPayload.truncatedAdditionalContextLengths.fileContextLength += entry.innerContext.length
}

triggerPayload.additionalContents.push(entry)
currentContextLength += entry.innerContext.length
let relativePath = path.relative(workspaceFolder, prompt.filePath)
Expand Down
9 changes: 9 additions & 0 deletions packages/core/src/codewhispererChat/controllers/chat/model.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,15 @@ export interface TriggerPayload {
documentReferences?: DocumentReference[]
useRelevantDocuments?: boolean
traceId?: string
additionalContextLengths?: AdditionalContextLengths
truncatedAdditionalContextLengths?: AdditionalContextLengths
workspaceRulesCount?: number
}

export type AdditionalContextLengths = {
fileContextLength: number
promptContextLength: number
ruleContextLength: number
}

// TODO move this to API definition (or just use this across the codebase)
Expand Down
77 changes: 77 additions & 0 deletions packages/core/src/codewhispererChat/controllers/chat/telemetryHelper.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import {
ResponseBodyLinkClickMessage,
SourceLinkClickMessage,
TriggerPayload,
AdditionalContextLengths,
} from './model'
import { TriggerEvent, TriggerEventsStorage } from '../../storages/triggerEvents'
import globals from '../../../shared/extensionGlobals'
Expand All @@ -38,6 +39,8 @@ import { supportedLanguagesList } from '../chat/chatRequest/converter'
import { AuthUtil } from '../../../codewhisperer/util/authUtil'
import { getSelectedCustomization } from '../../../codewhisperer/util/customizationUtil'
import { undefinedIfEmpty } from '../../../shared/utilities/textUtilities'
import { AdditionalContextPrompt } from '../../../amazonq/lsp/types'
import { getUserPromptsDirectory } from './controller'

export function logSendTelemetryEventFailure(error: any) {
let requestId: string | undefined
Expand Down Expand Up @@ -142,6 +145,39 @@ export class CWCTelemetryHelper {
telemetry.amazonq_exitFocusChat.emit({ result: 'Succeeded', passive: true })
}

public getContextType(prompt: AdditionalContextPrompt): string {
if (prompt.relativePath.startsWith('.amazonq/rules')) {
return 'rule'
} else if (prompt.filePath.startsWith(getUserPromptsDirectory())) {
return 'prompt'
} else {
return 'file'
}
}

public getContextLengths(prompts: AdditionalContextPrompt[]): AdditionalContextLengths {
let fileContextLength = 0
let promptContextLength = 0
let ruleContextLength = 0

for (const prompt of prompts) {
const type = this.getContextType(prompt)
switch (type) {
case 'rule':
ruleContextLength += prompt.content.length
break
case 'file':
fileContextLength += prompt.content.length
break
case 'prompt':
promptContextLength += prompt.content.length
break
}
}

return { fileContextLength, promptContextLength, ruleContextLength }
}

public async recordFeedback(message: ChatItemFeedbackMessage) {
const logger = getLogger()
try {
Expand Down Expand Up @@ -420,6 +456,30 @@ export class CWCTelemetryHelper {
})
}

private getAdditionalContextCounts(triggerPayload: TriggerPayload) {
const counts = {
fileContextCount: 0,
folderContextCount: 0,
promptContextCount: 0,
}

if (triggerPayload.context) {
for (const context of triggerPayload.context) {
if (typeof context !== 'string') {
if (context.id === 'file') {
counts.fileContextCount++
} else if (context.id === 'folder') {
counts.folderContextCount++
} else if (context.id === 'prompt') {
counts.promptContextCount++
}
}
}
}

return counts
}

public emitAddMessage(tabID: string, fullDisplayLatency: number, traceId: string, startTime?: number) {
const payload = this.messageStorage.get(tabID)
if (!payload) {
Expand All @@ -433,6 +493,9 @@ export class CWCTelemetryHelper {
triggerPayload.relevantTextDocuments &&
triggerPayload.relevantTextDocuments.length > 0 &&
triggerPayload.useRelevantDocuments === true

const contextCounts = this.getAdditionalContextCounts(triggerPayload)

const event: AmazonqAddMessage = {
result: 'Succeeded',
cwsprChatConversationId: this.getConversationId(message.tabID) ?? '',
Expand Down Expand Up @@ -464,6 +527,20 @@ export class CWCTelemetryHelper {
credentialStartUrl: AuthUtil.instance.startUrl,
codewhispererCustomizationArn: triggerPayload.customization.arn,
cwsprChatHasProjectContext: hasProjectLevelContext,
cwsprChatHasContextList: triggerPayload.documentReferences && triggerPayload.documentReferences?.length > 0,
cwsprChatFolderContextCount: contextCounts.folderContextCount,
cwsprChatFileContextCount: contextCounts.fileContextCount,
cwsprChatFileContextLength: triggerPayload.additionalContextLengths?.fileContextLength ?? 0,
cwsprChatFileContextTruncatedLength:
triggerPayload.truncatedAdditionalContextLengths?.fileContextLength ?? 0,
cwsprChatRuleContextCount: triggerPayload.workspaceRulesCount,
cwsprChatRuleContextLength: triggerPayload.additionalContextLengths?.ruleContextLength ?? 0,
cwsprChatRuleContextTruncatedLength:
triggerPayload.truncatedAdditionalContextLengths?.ruleContextLength ?? 0,
cwsprChatPromptContextCount: contextCounts.promptContextCount,
cwsprChatPromptContextLength: triggerPayload.additionalContextLengths?.promptContextLength ?? 0,
cwsprChatPromptContextTruncatedLength:
triggerPayload.truncatedAdditionalContextLengths?.promptContextLength ?? 0,
traceId,
}

Expand Down
103 changes: 103 additions & 0 deletions packages/core/src/shared/telemetry/vscodeTelemetry.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,61 @@
"type": "int",
"description": "Query latency in ms for local project context"
},
{
"name": "cwsprChatFolderContextCount",
"type": "int",
"description": "# of folders manually added to context"
},
{
"name": "cwsprChatFileContextCount",
"type": "int",
"description": "# of files manually added to context"
},
{
"name": "cwsprChatFileContextLength",
"type": "int",
"description": "Total length of files added to context"
},
{
"name": "cwsprChatFileContextTruncatedLength",
"type": "int",
"description": "Truncated length of files added to context"
},
{
"name": "cwsprChatPromptContextCount",
"type": "int",
"description": "# of saved prompts manually added to context"
},
{
"name": "cwsprChatPromptContextLength",
"type": "int",
"description": "Total length of saved prompts added to context"
},
{
"name": "cwsprChatPromptContextTruncatedLength",
"type": "int",
"description": "Truncated length of saved prompts added to context"
},
{
"name": "cwsprChatRuleContextCount",
"type": "int",
"description": "# of workspace rules automatically added to context"
},
{
"name": "cwsprChatRuleContextLength",
"type": "int",
"description": "Total length of workspace rules added to context"
},
{
"name": "cwsprChatRuleContextTruncatedLength",
"type": "int",
"description": "Truncated length of workspace rules added to context"
},
{
"name": "cwsprChatHasContextList",
"type": "boolean",
"description": "true if context list is displayed to user"
},
{
"name": "amazonqIndexFileSizeInMB",
"type": "int",
Expand Down Expand Up @@ -638,6 +693,10 @@
"name": "amazonq_closeChat",
"description": "When chat panel is closed"
},
{
"name": "amazonq_createSavedPrompt",
Copy link
Contributor

@jpinkney-aws jpinkney-aws Feb 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is also in the commons PR, is theres telemetry changes expected to live here or there?

Copy link
Contributor Author

@avi-alpert avi-alpert Feb 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there. but i can get rid of it from both PRs now that im switching it over to ui_click

"description": "When a saved user prompt is created"
},
{
"name": "amazonq_startConversation",
"description": "When user starts a new conversation",
Expand Down Expand Up @@ -804,6 +863,50 @@
{
"type": "codewhispererCustomizationArn",
"required": false
},
{
"type": "cwsprChatHasContextList",
"required": false
},
{
"type": "cwsprChatFolderContextCount",
"required": false
},
{
"type": "cwsprChatFileContextCount",
"required": false
},
{
"type": "cwsprChatFileContextLength",
"required": false
},
{
"type": "cwsprChatFileContextTruncatedLength",
"required": false
},
{
"type": "cwsprChatPromptContextCount",
"required": false
},
{
"type": "cwsprChatPromptContextLength",
"required": false
},
{
"type": "cwsprChatRuleContextTruncatedLength",
"required": false
},
{
"type": "cwsprChatRuleContextCount",
"required": false
},
{
"type": "cwsprChatRuleContextLength",
"required": false
},
{
"type": "cwsprChatPromptContextTruncatedLength",
"required": false
}
]
},
Expand Down
Loading