From 9f51ec7e183ca49cc5a5110b5c77cc5a1921568a Mon Sep 17 00:00:00 2001 From: Joshua Pinkney Date: Tue, 8 Oct 2024 09:46:14 -0400 Subject: [PATCH 1/5] docs(amazonq): Add visual representation for performance telemetry metrics --- docs/telemetry-perf.md | 263 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 263 insertions(+) create mode 100644 docs/telemetry-perf.md diff --git a/docs/telemetry-perf.md b/docs/telemetry-perf.md new file mode 100644 index 00000000000..24ccaa04332 --- /dev/null +++ b/docs/telemetry-perf.md @@ -0,0 +1,263 @@ +# Telemetry Performance Metrics + +Visual representations of performance telemetry metrics + +## codewhispererFirstCompletionLatency + +How long it took to receive the first suggestion after we started calling the getRecommendations API + +```mermaid + sequenceDiagram + participant User + participant invoke as Inline invoked + participant rService as Recommendation Service + participant rHandler as Recommendation Handler + participant backend as CWSPR backend + participant sdk as Create CWSPR SDK + participant token as Toolkit auth + + + User->>invoke: Finished typing + invoke->>rService: calls + rService->>rHandler: calls + rHandler->>sdk: calls + + sdk->>token: Start getting bearer token + token->>sdk: Finished getting bearer token + + sdk->>rHandler: Return client + note over rHandler, backend: codewhispererFirstCompletionLatency + rect rgb(75, 75, 0) + loop Get paginated recommendations + rHandler->>backend: calls + end + backend->>rHandler: first response received + end + rHandler->>User: show results + backend->>rHandler: all the other responses + rHandler->>User: add to already shown results +``` + +## codewhispererEndToEndLatency + +How long it took from when we started calling the getRecommendations API to when the first suggestion was shown + +```mermaid + sequenceDiagram + participant User + participant invoke as Inline invoked + participant rService as Recommendation Service + participant rHandler as Recommendation Handler + participant backend as CWSPR backend + participant sdk as Create CWSPR SDK + participant token as Toolkit auth + + + User->>invoke: Finished typing + invoke->>rService: calls + rService->>rHandler: calls + rHandler->>sdk: calls + + sdk->>token: Start getting bearer token + token->>sdk: Finished getting bearer token + + sdk->>rHandler: Return client + note over User, backend: codewhispererEndToEndLatency + rect rgb(75, 75, 0) + loop Get paginated recommendations + rHandler->>backend: calls + end + backend->>rHandler: first response received + rHandler->>User: show results + end + + backend->>rHandler: all the other responses + rHandler->>User: add to already shown results +``` + +## codewhispererAllCompletionsLatency + +How long it took to complete all paginated calls + +```mermaid + sequenceDiagram + participant User + participant invoke as Inline invoked + participant rService as Recommendation Service + participant rHandler as Recommendation Handler + participant backend as CWSPR backend + participant sdk as Create CWSPR SDK + participant token as Toolkit auth + + + User->>invoke: Finished typing + invoke->>rService: calls + rService->>rHandler: calls + rHandler->>sdk: calls + + sdk->>token: Start getting bearer token + token->>sdk: Finished getting bearer token + + sdk->>rHandler: Return client + note over User, backend: codewhispererAllCompletionsLatency + rect rgb(75, 75, 0) + loop Get paginated recommendations + rHandler->>backend: calls + end + backend->>rHandler: first response received + rHandler->>User: show results + backend->>rHandler: all the other responses + end + + + rHandler->>User: add to already shown results +``` + +## codewhispererPostprocessingLatency + +How long it took to display the first suggestion after it received the first response from the API + +```mermaid + sequenceDiagram + participant User + participant invoke as Inline invoked + participant rService as Recommendation Service + participant rHandler as Recommendation Handler + participant backend as CWSPR backend + participant sdk as Create CWSPR SDK + participant token as Toolkit auth + + + User->>invoke: Finished typing + invoke->>rService: calls + rService->>rHandler: calls + rHandler->>sdk: calls + + sdk->>token: Start getting bearer token + token->>sdk: Finished getting bearer token + + sdk->>rHandler: Return client + loop Get paginated recommendations + rHandler->>backend: calls + end + note over User, backend: codewhispererPostprocessingLatency + rect rgb(75, 75, 0) + backend->>rHandler: first response received + rHandler->>User: show results + + end + + backend->>rHandler: all the other responses + rHandler->>User: add to already shown results +``` + +## codewhispererCredentialFetchingLatency + +How long it took to get the bearer token + +```mermaid + sequenceDiagram + participant User + participant invoke as Inline invoked + participant rService as Recommendation Service + participant rHandler as Recommendation Handler + participant backend as CWSPR backend + participant sdk as Create CWSPR SDK + participant token as Toolkit auth + + User->>invoke: Finished typing + invoke->>rService: calls + rService->>rHandler: calls + rHandler->>sdk: calls + + note over sdk, token: codewhispererCredentialFetchingLatency + rect rgb(75, 75, 0) + sdk->>token: Start getting bearer token + token->>sdk: Finished getting bearer token + end + sdk->>rHandler: Return client + loop Get paginated recommendations + rHandler->>backend: calls + end + + backend->>rHandler: first response received + rHandler->>User: show results + + backend->>rHandler: all the other responses + rHandler->>User: add to already shown results +``` + +## codewhispererPreprocessingLatency + +How long it took to create the client and get ready to start sending getRecommendation API calls + +```mermaid + sequenceDiagram + participant User + participant invoke as Inline invoked + participant rService as Recommendation Service + participant rHandler as Recommendation Handler + participant backend as CWSPR backend + participant sdk as Create CWSPR SDK + participant token as Toolkit auth + + User->>invoke: Finished typing + invoke->>rService: calls + rService->>rHandler: calls + rHandler->>sdk: calls + + note over rHandler, token: codewhispererPreprocessingLatency + rect rgb(75, 75, 0) + sdk->>token: Start getting bearer token + token->>sdk: Finished getting bearer token + sdk->>rHandler: Return client + end + loop Get paginated recommendations + rHandler->>backend: calls + end + + backend->>rHandler: first response received + rHandler->>User: show results + + backend->>rHandler: all the other responses + rHandler->>User: add to already shown results +``` + +## codewhisperer_perceivedLatency duration + +How long it took from when the user stopped pressing a key to when they were shown a response + +```mermaid + sequenceDiagram + participant User + participant invoke as Inline invoked + participant rService as Recommendation Service + participant rHandler as Recommendation Handler + participant backend as CWSPR backend + participant sdk as Create CWSPR SDK + participant token as Toolkit auth + + User->>invoke: Finished typing + note over User, token: codewhisperer_perceivedLatency duration + rect rgb(75, 75, 0) + invoke->>rService: calls + rService->>rHandler: calls + rHandler->>sdk: calls + + + + sdk->>token: Start getting bearer token + token->>sdk: Finished getting bearer token + sdk->>rHandler: Return client + + loop Get paginated recommendations + rHandler->>backend: calls + end + + backend->>rHandler: first response received + rHandler->>User: show results + + backend->>rHandler: all the other responses + rHandler->>User: add to already shown results + end +``` From 91fd63e5a4d592af7a5db6d0fd2c6832108162e8 Mon Sep 17 00:00:00 2001 From: Joshua Pinkney Date: Tue, 8 Oct 2024 09:48:36 -0400 Subject: [PATCH 2/5] fixup --- docs/telemetry-perf.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/docs/telemetry-perf.md b/docs/telemetry-perf.md index 24ccaa04332..a580cc870cc 100644 --- a/docs/telemetry-perf.md +++ b/docs/telemetry-perf.md @@ -27,7 +27,7 @@ How long it took to receive the first suggestion after we started calling the ge sdk->>rHandler: Return client note over rHandler, backend: codewhispererFirstCompletionLatency - rect rgb(75, 75, 0) + rect rgb(75, 75, 75) loop Get paginated recommendations rHandler->>backend: calls end @@ -63,7 +63,7 @@ How long it took from when we started calling the getRecommendations API to when sdk->>rHandler: Return client note over User, backend: codewhispererEndToEndLatency - rect rgb(75, 75, 0) + rect rgb(75, 75, 75) loop Get paginated recommendations rHandler->>backend: calls end @@ -100,7 +100,7 @@ How long it took to complete all paginated calls sdk->>rHandler: Return client note over User, backend: codewhispererAllCompletionsLatency - rect rgb(75, 75, 0) + rect rgb(75, 75, 75) loop Get paginated recommendations rHandler->>backend: calls end @@ -141,7 +141,7 @@ How long it took to display the first suggestion after it received the first res rHandler->>backend: calls end note over User, backend: codewhispererPostprocessingLatency - rect rgb(75, 75, 0) + rect rgb(75, 75, 75) backend->>rHandler: first response received rHandler->>User: show results @@ -171,7 +171,7 @@ How long it took to get the bearer token rHandler->>sdk: calls note over sdk, token: codewhispererCredentialFetchingLatency - rect rgb(75, 75, 0) + rect rgb(75, 75, 75) sdk->>token: Start getting bearer token token->>sdk: Finished getting bearer token end @@ -207,7 +207,7 @@ How long it took to create the client and get ready to start sending getRecommen rHandler->>sdk: calls note over rHandler, token: codewhispererPreprocessingLatency - rect rgb(75, 75, 0) + rect rgb(75, 75, 75) sdk->>token: Start getting bearer token token->>sdk: Finished getting bearer token sdk->>rHandler: Return client @@ -239,13 +239,10 @@ How long it took from when the user stopped pressing a key to when they were sho User->>invoke: Finished typing note over User, token: codewhisperer_perceivedLatency duration - rect rgb(75, 75, 0) + rect rgb(75, 75, 75) invoke->>rService: calls rService->>rHandler: calls rHandler->>sdk: calls - - - sdk->>token: Start getting bearer token token->>sdk: Finished getting bearer token sdk->>rHandler: Return client From afeac969a88035b45ff3180c4460a6d727e13375 Mon Sep 17 00:00:00 2001 From: Joshua Pinkney Date: Tue, 8 Oct 2024 09:51:16 -0400 Subject: [PATCH 3/5] fixup --- docs/telemetry-perf.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/telemetry-perf.md b/docs/telemetry-perf.md index a580cc870cc..6cdfe167d93 100644 --- a/docs/telemetry-perf.md +++ b/docs/telemetry-perf.md @@ -27,7 +27,7 @@ How long it took to receive the first suggestion after we started calling the ge sdk->>rHandler: Return client note over rHandler, backend: codewhispererFirstCompletionLatency - rect rgb(75, 75, 75) + rect rgb(200, 255, 200) loop Get paginated recommendations rHandler->>backend: calls end @@ -63,7 +63,7 @@ How long it took from when we started calling the getRecommendations API to when sdk->>rHandler: Return client note over User, backend: codewhispererEndToEndLatency - rect rgb(75, 75, 75) + rect rgb(200, 255, 200) loop Get paginated recommendations rHandler->>backend: calls end @@ -100,7 +100,7 @@ How long it took to complete all paginated calls sdk->>rHandler: Return client note over User, backend: codewhispererAllCompletionsLatency - rect rgb(75, 75, 75) + rect rgb(200, 255, 200) loop Get paginated recommendations rHandler->>backend: calls end @@ -141,10 +141,9 @@ How long it took to display the first suggestion after it received the first res rHandler->>backend: calls end note over User, backend: codewhispererPostprocessingLatency - rect rgb(75, 75, 75) + rect rgb(200, 255, 200) backend->>rHandler: first response received rHandler->>User: show results - end backend->>rHandler: all the other responses @@ -171,7 +170,7 @@ How long it took to get the bearer token rHandler->>sdk: calls note over sdk, token: codewhispererCredentialFetchingLatency - rect rgb(75, 75, 75) + rect rgb(200, 255, 200) sdk->>token: Start getting bearer token token->>sdk: Finished getting bearer token end @@ -207,7 +206,7 @@ How long it took to create the client and get ready to start sending getRecommen rHandler->>sdk: calls note over rHandler, token: codewhispererPreprocessingLatency - rect rgb(75, 75, 75) + rect rgb(200, 255, 200) sdk->>token: Start getting bearer token token->>sdk: Finished getting bearer token sdk->>rHandler: Return client @@ -239,7 +238,7 @@ How long it took from when the user stopped pressing a key to when they were sho User->>invoke: Finished typing note over User, token: codewhisperer_perceivedLatency duration - rect rgb(75, 75, 75) + rect rgb(200, 255, 200) invoke->>rService: calls rService->>rHandler: calls rHandler->>sdk: calls From 48cd6125f2781a48192cd96364c51d5651daec6f Mon Sep 17 00:00:00 2001 From: Joshua Pinkney Date: Tue, 8 Oct 2024 09:55:45 -0400 Subject: [PATCH 4/5] fixup --- docs/telemetry-perf.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/telemetry-perf.md b/docs/telemetry-perf.md index 6cdfe167d93..be6c325537b 100644 --- a/docs/telemetry-perf.md +++ b/docs/telemetry-perf.md @@ -27,7 +27,7 @@ How long it took to receive the first suggestion after we started calling the ge sdk->>rHandler: Return client note over rHandler, backend: codewhispererFirstCompletionLatency - rect rgb(200, 255, 200) + rect rgb(230, 230, 230, 0.5) loop Get paginated recommendations rHandler->>backend: calls end @@ -52,7 +52,6 @@ How long it took from when we started calling the getRecommendations API to when participant sdk as Create CWSPR SDK participant token as Toolkit auth - User->>invoke: Finished typing invoke->>rService: calls rService->>rHandler: calls @@ -63,7 +62,7 @@ How long it took from when we started calling the getRecommendations API to when sdk->>rHandler: Return client note over User, backend: codewhispererEndToEndLatency - rect rgb(200, 255, 200) + rect rgb(230, 230, 230, 0.5) loop Get paginated recommendations rHandler->>backend: calls end @@ -100,7 +99,7 @@ How long it took to complete all paginated calls sdk->>rHandler: Return client note over User, backend: codewhispererAllCompletionsLatency - rect rgb(200, 255, 200) + rect rgb(230, 230, 230, 0.5) loop Get paginated recommendations rHandler->>backend: calls end @@ -141,7 +140,7 @@ How long it took to display the first suggestion after it received the first res rHandler->>backend: calls end note over User, backend: codewhispererPostprocessingLatency - rect rgb(200, 255, 200) + rect rgb(230, 230, 230, 0.5) backend->>rHandler: first response received rHandler->>User: show results end @@ -170,7 +169,7 @@ How long it took to get the bearer token rHandler->>sdk: calls note over sdk, token: codewhispererCredentialFetchingLatency - rect rgb(200, 255, 200) + rect rgb(230, 230, 230, 0.5) sdk->>token: Start getting bearer token token->>sdk: Finished getting bearer token end @@ -206,7 +205,7 @@ How long it took to create the client and get ready to start sending getRecommen rHandler->>sdk: calls note over rHandler, token: codewhispererPreprocessingLatency - rect rgb(200, 255, 200) + rect rgb(230, 230, 230, 0.5) sdk->>token: Start getting bearer token token->>sdk: Finished getting bearer token sdk->>rHandler: Return client @@ -238,7 +237,7 @@ How long it took from when the user stopped pressing a key to when they were sho User->>invoke: Finished typing note over User, token: codewhisperer_perceivedLatency duration - rect rgb(200, 255, 200) + rect rgb(230, 230, 230, 0.5) invoke->>rService: calls rService->>rHandler: calls rHandler->>sdk: calls From 69fdbc4ef9459ae068b79e038609d1bffdd3d67e Mon Sep 17 00:00:00 2001 From: Joshua Pinkney Date: Tue, 8 Oct 2024 10:02:09 -0400 Subject: [PATCH 5/5] fixup --- docs/telemetry-perf.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/telemetry-perf.md b/docs/telemetry-perf.md index be6c325537b..74334183ca4 100644 --- a/docs/telemetry-perf.md +++ b/docs/telemetry-perf.md @@ -2,7 +2,9 @@ Visual representations of performance telemetry metrics -## codewhispererFirstCompletionLatency +## Amazon Q Inline + +### codewhispererFirstCompletionLatency How long it took to receive the first suggestion after we started calling the getRecommendations API @@ -38,7 +40,7 @@ How long it took to receive the first suggestion after we started calling the ge rHandler->>User: add to already shown results ``` -## codewhispererEndToEndLatency +### codewhispererEndToEndLatency How long it took from when we started calling the getRecommendations API to when the first suggestion was shown @@ -74,7 +76,7 @@ How long it took from when we started calling the getRecommendations API to when rHandler->>User: add to already shown results ``` -## codewhispererAllCompletionsLatency +### codewhispererAllCompletionsLatency How long it took to complete all paginated calls @@ -112,7 +114,7 @@ How long it took to complete all paginated calls rHandler->>User: add to already shown results ``` -## codewhispererPostprocessingLatency +### codewhispererPostprocessingLatency How long it took to display the first suggestion after it received the first response from the API @@ -149,7 +151,7 @@ How long it took to display the first suggestion after it received the first res rHandler->>User: add to already shown results ``` -## codewhispererCredentialFetchingLatency +### codewhispererCredentialFetchingLatency How long it took to get the bearer token @@ -185,7 +187,7 @@ How long it took to get the bearer token rHandler->>User: add to already shown results ``` -## codewhispererPreprocessingLatency +### codewhispererPreprocessingLatency How long it took to create the client and get ready to start sending getRecommendation API calls @@ -221,7 +223,7 @@ How long it took to create the client and get ready to start sending getRecommen rHandler->>User: add to already shown results ``` -## codewhisperer_perceivedLatency duration +### codewhisperer_perceivedLatency duration How long it took from when the user stopped pressing a key to when they were shown a response