Skip to content

telemetry: Metrics for amazonq unit test generation #905

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 14 commits into from
Closed

telemetry: Metrics for amazonq unit test generation #905

Changes from 11 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
9c158f3
Metrics for amazonq Unit test generation
laileni-aws Nov 10, 2024
e0452fc
lint errors
laileni-aws Nov 10, 2024
dddb602
lint errors
laileni-aws Nov 10, 2024
23f8653
lint errors
laileni-aws Nov 10, 2024
027ad72
lint errors
laileni-aws Nov 10, 2024
44dfd00
lint errors
laileni-aws Nov 10, 2024
ccd1f70
lint errors
laileni-aws Nov 10, 2024
e592ced
lint errors
laileni-aws Nov 10, 2024
1f192b4
lint errors
laileni-aws Nov 10, 2024
6c71a1a
lint errors
laileni-aws Nov 10, 2024
f514400
Addressing comments
laileni-aws Nov 11, 2024
050008d
Addressing comments
laileni-aws Nov 12, 2024
758dd96
Addressing commenrs
laileni-aws Nov 12, 2024
21c835e
Addressing commenrs
laileni-aws Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
253 changes: 253 additions & 0 deletions telemetry/definitions/commonDefinitions.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
{
"types": [
{
"name": "acceptedCount",
"type": "int",
"description": "The number of accepted cases"
},
{
"name": "action",
"type": "string",
Expand Down Expand Up @@ -199,6 +204,16 @@
"type": "boolean",
"description": "Boolean value of whether or not a Cfn parameter file is provided."
},
{
"name": "charsOfCodeAccepted",
"type": "int",
"description": "Number of characters of code accepted"
},
{
"name": "charsOfCodeGenerated",
"type": "int",
"description": "Number of characters of code generated"
},
{
"name": "checkType",
"type": "string",
Expand Down Expand Up @@ -1207,6 +1222,11 @@
"type": "string",
"description": "The name of the EventBridge Schema used in the operation"
},
{
"name": "executedCount",
"type": "int",
"description": "The number of executed operations"
},
{
"name": "experimentId",
"type": "string",
Expand Down Expand Up @@ -1273,6 +1293,11 @@
"type": "string",
"description": "Application framework being used"
},
{
"name": "generatedCount",
"type": "int",
"description": "The number of generated cases"
},
{
"name": "generateFailure",
"type": "string",
Expand Down Expand Up @@ -1417,6 +1442,26 @@
"type": "string",
"description": "Language used for the project."
},
{
"name": "latencyOfClientSideBuild",
"type": "int",
"description": "Time from start to end of Client-side build."
},
{
"name": "latencyOfGeneratingTests",
"type": "int",
"description": "Time from start to end of Generating unit tests"
},
{
"name": "linesOfCodeAccepted",
"type": "int",
"description": "Number of lines of code accepted"
},
{
"name": "linesOfCodeGenerated",
Copy link
Contributor

@justinmk3 justinmk3 Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you really need to track chars and lines? how is that the existing codewhisperer metrics don't already have fields for this?

Suggested change
"name": "linesOfCodeAccepted",
"type": "int",
"description": "Number of lines of code accepted"
},
{
"name": "linesOfCodeGenerated",
"name": "acceptedLinesCount",
"type": "int",
"description": "Number of lines of code accepted"
},
{
"name": "generatedLinesCount",

Copy link
Contributor Author

@laileni-aws laileni-aws Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are using cwsprChatAcceptedNumberOfLines which is specific name not generic but trying to use the same.

"type": "int",
"description": "Number of lines of code generated"
},
{
"name": "loadFileTime",
"type": "int",
Expand Down Expand Up @@ -1670,11 +1715,21 @@
"type": "string",
"description": "Date/time that an SSO client registration expires."
},
{
"name": "step",
"type": "string",
"description": "Indicates the stage at which a user interface click action was performed."
},
{
"name": "successCount",
"type": "int",
"description": "The number of successful operations"
},
{
"name": "supportedLanguage",
"type": "boolean",
"description": "supported language"
},
Copy link
Contributor

@justinmk3 justinmk3 Nov 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are multiple such fields already. do not add redundant fields.

Copy link
Contributor Author

@laileni-aws laileni-aws Nov 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't see a field which checks the supported language(boolean).
LMK If I am missing something.

Copy link

@chandankmishra chandankmishra Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This field should be isSupportedLanguage to indicate it is boolean field. SupportedLanguage seems like we are going to pass the actual value of language which could be string/enum type.

Also the comment should be "Indicate if the language is supported"

Copy link
Contributor Author

@laileni-aws laileni-aws Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not see convention naming isXXXX for boolean values in the json file so for keeping consistent went with supportedLanguage
Let me change this accordingly

{
"name": "syncedResources",
"type": "string",
Expand All @@ -1694,6 +1749,16 @@
"type": "string",
"description": "Generic name of a template"
},
{
"name": "testGenerationJobGroupName",
"type": "string",
"description": "Unit test generation job group name"
},
{
"name": "testGenerationJobId",
Copy link
Contributor

@justinmk3 justinmk3 Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a group name and job id could be used in many metrics. please do not add hyper-specific fields for every metric.

I already see fields like codeTransformJobId, and that's another case where we're paying the price for those fields being over-specific.

Suggested change
"name": "testGenerationJobGroupName",
"type": "string",
"description": "Unit test generation job group name"
},
{
"name": "testGenerationJobId",
"name": "jobGroup",
"type": "string",
"description": "Job group name associated with the metric"
},
{
"name": "jobId",

Copy link
Contributor Author

@laileni-aws laileni-aws Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will update with this

"type": "string",
"description": "Unit test generation job id"
},
{
"name": "toolId",
"type": "string",
Expand All @@ -1717,6 +1782,16 @@
"type": "string",
"description": "Unique identifier for the trace (a set of events) this metric belongs to"
},
{
"name": "unitTestGenerationBuildPayloadBytes",
"type": "int",
"description": "The uncompressed payload size in bytes of the source files in customer project context sent for unit test generation"
},
{
"name": "unitTestGenerationBuildZipFileBytes",
"type": "int",
"description": "The compressed payload size of source files in bytes of customer project context sent for unit test generation"
},
{
"name": "update",
"type": "boolean",
Expand All @@ -1732,6 +1807,11 @@
"type": "string",
"description": "User selection from a predefined menu (not user-provided input). See also `action`."
},
{
"name": "userEnteredPromptMessage",
"type": "boolean",
"description": "True if user enter prompt message as input else false"
},
{
"name": "userId",
"type": "string",
Expand Down Expand Up @@ -2214,6 +2294,175 @@
}
]
},
{
"name": "amazonq_unitTestGeneration",
"description": "Client side metrics of AmazonQ Unit Test Generation",
"metadata": [
{
"type": "acceptedCount",
"required": false
},
{
"type": "charsOfCodeAccepted",
"required": false
},
{
"type": "charsOfCodeGenerated",
"required": false
},
{
"type": "credentialStartUrl",
"required": false
},
{
"type": "cwsprChatProgrammingLanguage"
},
{
"type": "executedCount",
"required": false
},
{
"type": "failedCount",
"required": false
},
{
"type": "generatedCount",
"required": false
},
{
"type": "linesOfCodeAccepted",
"required": false
},
{
"type": "linesOfCodeGenerated",
"required": false
},
{
"type": "successCount",
"required": false
},
{
"type": "supportedLanguage"
},
{
"type": "testGenerationJobGroupName",
"required": false
},
{
"type": "userEnteredPromptMessage"
}
]
},
{
"name": "amazonq_utg_buildLoop",
"description": "Client side invocation of the AmazonQ Unit Test Generation build loop",
"metadata": [
{
"type": "credentialStartUrl",
"required": false
},
{
"type": "cwsprChatProgrammingLanguage"
},
{
"type": "latencyOfClientSideBuild",
"required": false
},
{
"type": "result",
"required": false
},
{
"type": "source",
"required": false
},
{
"type": "supportedLanguage"
},
{
"type": "testGenerationJobGroupName",
"required": false
},
{
"type": "testGenerationJobId",
"required": false
},
{
"type": "userEnteredPromptMessage"
}
]
},
{
"name": "amazonq_utg_generateTests",
"description": "Client side invocation of the AmazonQ Unit Test Generation",
"metadata": [
{
"type": "acceptedCount",
"required": false
},
{
"type": "artifactsUploadDuration",
"required": false
},
{
"type": "charsOfCodeAccepted",
"required": false
},
{
"type": "charsOfCodeGenerated",
"required": false
},
{
"type": "credentialStartUrl",
"required": false
},
{
"type": "cwsprChatProgrammingLanguage"
},
{
"type": "generatedCount",
"required": false
},
{
"type": "latencyOfGeneratingTests",
"required": false
},
{
"type": "linesOfCodeAccepted",
"required": false
},
{
"type": "linesOfCodeGenerated",
"required": false
},
{
"type": "source",
"required": false
},
{
"type": "supportedLanguage"
},
{
"type": "testGenerationJobGroupName",
"required": false
},
{
"type": "testGenerationJobId",
"required": false
},
{
"type": "unitTestGenerationBuildPayloadBytes",
"required": false
},
{
"type": "unitTestGenerationBuildZipFileBytes",
"required": false
},
{
"type": "userEnteredPromptMessage"
}
]
},
{
"name": "amazonq_viewChatPanel",
"description": "Captures if Q chat panel is successfully viewed or not",
Expand Down Expand Up @@ -6997,6 +7246,10 @@
"metadata": [
{
"type": "elementId"
},
{
"type": "step",
"required": false
}
]
},
Expand Down