Skip to content

Commit cee0fa3

Browse files
jpinkney-awsjpinkney-aws
authored
docs(amazonq): Add latency metric diagrams for amazon q chat (#5865)
## Problem We have no documentation showing what latencies we are measuring for amazon q chat ## Solution Add documentation
1 parent 77bd8a7 commit cee0fa3

File tree

1 file changed

+286
-0
lines changed

1 file changed

+286
-0
lines changed

‎docs/telemetry-perf.md‎

Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,6 +259,292 @@ How long it took from when the user stopped pressing a key to when they were sho
259259
end
260260
```
261261

262+
## Amazon Q Chat
263+
264+
### amazonq_chatRoundTrip
265+
266+
Measures sequential response times in Q chat, from user input to message display. Tracks time intervals between key events: editor receiving the message, feature processing, and final message rendering
267+
268+
```mermaid
269+
sequenceDiagram
270+
participant User
271+
participant chat as Chat UI
272+
participant vscode as VSCode
273+
participant event as Event Recorder
274+
participant partner as Partner team code
275+
participant telemetry
276+
277+
User->>chat: Write chat message and press enter
278+
chat->>vscode: send message with timestamp
279+
vscode->>event: record chatMessageSent/editorReceivedMessage timestamps
280+
vscode->>partner: forward chat message
281+
partner->>event: record featureReceivedMessage timestamp
282+
partner->>partner: call backend/get response
283+
partner->>vscode: forward response contents
284+
vscode->>chat: display message
285+
chat->>vscode: send stop-chat-message-telemetry event
286+
vscode->>event: record messageDisplayed timestamp
287+
event->>vscode: get the telemetry timestamps
288+
vscode->>telemetry: emit amazonq_chatRoundTrip with telemetry timestamps
289+
```
290+
291+
### cwsprChatTimeToFirstChunk
292+
293+
The time between when the conversation stream is created and when we got back the first usable result
294+
295+
```mermaid
296+
sequenceDiagram
297+
participant user as User
298+
participant chat as Chat UI
299+
participant vscode as VSCode extension host
300+
participant generateResponse as Generate response
301+
participant backend as Q service backend
302+
303+
user->>chat: Presses enter with message
304+
chat->>vscode: Tell VSCode to generate a response
305+
vscode->>generateResponse: start generating
306+
generateResponse->>backend: start stream
307+
backend->>backend: create conversation id
308+
backend->>generateResponse: get conversation id
309+
note over backend, generateResponse: cwsprChatTimeToFirstChunk
310+
rect rgb(230, 230, 230, 0.5)
311+
backend->>backend: generate first chunk
312+
backend->>generateResponse: chunk received
313+
end
314+
generateResponse->>vscode: send chunk to display
315+
vscode->>chat: display chunk
316+
loop for each subsequent chunk
317+
backend->>backend: generate next chunk
318+
backend->>generateResponse: chunk received
319+
generateResponse->>vscode: send chunk to display
320+
vscode->>chat: display chunk
321+
end
322+
```
323+
324+
### cwsprChatTimeBetweenChunks
325+
326+
An array of time when successive pieces of data are received from the server
327+
328+
```mermaid
329+
sequenceDiagram
330+
participant user as User
331+
participant chat as Chat UI
332+
participant vscode as VSCode extension host
333+
participant generateResponse as Generate response
334+
participant backend as Q service backend
335+
336+
user->>chat: Presses enter with message
337+
chat->>vscode: Tell VSCode to generate a response
338+
vscode->>generateResponse: start generating
339+
generateResponse->>backend: start stream
340+
backend->>backend: create conversation id
341+
backend->>generateResponse: get conversation id
342+
343+
loop for each subsequent chunk
344+
note over backend, generateResponse: cwsprChatTimeBetweenChunks
345+
rect rgb(230, 230, 230, 0.5)
346+
backend->>backend: generate next chunk
347+
backend->>generateResponse: chunk received
348+
generateResponse->>generateResponse: record timestamp
349+
end
350+
351+
generateResponse->>vscode: send chunk to display
352+
vscode->>chat: display chunk
353+
end
354+
```
355+
356+
### cwsprChatFullResponseLatency
357+
358+
The time between when the conversation id was created and the final response from the server was received
359+
360+
```mermaid
361+
sequenceDiagram
362+
participant user as User
363+
participant chat as Chat UI
364+
participant vscode as VSCode extension host
365+
participant generateResponse as Generate response
366+
participant backend as Q service backend
367+
368+
user->>chat: Presses enter with message
369+
chat->>vscode: Tell VSCode to generate a response
370+
vscode->>generateResponse: start generating
371+
generateResponse->>backend: start stream
372+
backend->>backend: create conversation id
373+
backend->>generateResponse: get conversation id
374+
375+
note over backend, chat: cwsprChatFullResponseLatency
376+
rect rgb(230, 230, 230, 0.5)
377+
loop for each subsequent chunk
378+
backend->>backend: generate next chunk
379+
backend->>generateResponse: chunk received
380+
generateResponse->>vscode: send chunk to display
381+
vscode->>chat: display chunk
382+
end
383+
backend->>generateResponse: final chunk received
384+
end
385+
generateResponse->>vscode: send chunk to display
386+
vscode->>chat: display chunk
387+
```
388+
389+
### cwsprChatTimeToFirstUsableChunk
390+
391+
The time between the initial server request, including creating the conversation stream, and the first usable result
392+
393+
```mermaid
394+
sequenceDiagram
395+
participant user as User
396+
participant chat as Chat UI
397+
participant vscode as VSCode extension host
398+
participant generateResponse as Generate response
399+
participant backend as Q service backend
400+
401+
user->>chat: Presses enter with message
402+
chat->>vscode: Tell VSCode to generate a response
403+
vscode->>generateResponse: start generating
404+
note over backend, generateResponse: cwsprChatTimeToFirstUsableChunk
405+
rect rgb(230, 230, 230, 0.5)
406+
generateResponse->>backend: start stream
407+
backend->>backend: create conversation id
408+
backend->>generateResponse: get conversation id
409+
backend->>backend: generate first chunk
410+
backend->>generateResponse: chunk received
411+
end
412+
generateResponse->>vscode: send chunk to display
413+
vscode->>chat: display chunk
414+
loop for each subsequent chunk
415+
backend->>backend: generate next chunk
416+
backend->>generateResponse: chunk received
417+
generateResponse->>vscode: send chunk to display
418+
vscode->>chat: display chunk
419+
end
420+
```
421+
422+
### cwsprChatFullServerResponseLatency
423+
424+
The time between the initial server request, including creating the conversation stream, and the final response from the server
425+
426+
```mermaid
427+
sequenceDiagram
428+
participant user as User
429+
participant chat as Chat UI
430+
participant vscode as VSCode extension host
431+
participant generateResponse as Generate response
432+
participant backend as Q service backend
433+
434+
user->>chat: Presses enter with message
435+
chat->>vscode: Tell VSCode to generate a response
436+
vscode->>generateResponse: start generating
437+
note over backend, chat: cwsprChatFullServerResponseLatency
438+
rect rgb(230, 230, 230, 0.5)
439+
generateResponse->>backend: start stream
440+
backend->>backend: create conversation id
441+
backend->>generateResponse: get conversation id
442+
loop for each subsequent chunk
443+
backend->>backend: generate next chunk
444+
backend->>generateResponse: chunk received
445+
generateResponse->>vscode: send chunk to display
446+
vscode->>chat: display chunk
447+
end
448+
backend->>generateResponse: final chunk received
449+
end
450+
generateResponse->>vscode: send chunk to display
451+
vscode->>chat: display chunk
452+
```
453+
454+
### cwsprChatTimeToFirstDisplay
455+
456+
The time between the user pressing enter and when the first piece of data is displayed to the user
457+
458+
```mermaid
459+
sequenceDiagram
460+
participant user as User
461+
participant chat as Chat UI
462+
participant vscode as VSCode extension host
463+
participant generateResponse as Generate response
464+
participant backend as Q service backend
465+
note over backend, user: cwsprChatTimeToFirstDisplay
466+
rect rgb(230, 230, 230, 0.5)
467+
user->>chat: Presses enter with message
468+
chat->>vscode: Tell VSCode to generate a response
469+
vscode->>generateResponse: start generating
470+
generateResponse->>backend: start stream
471+
backend->>backend: create conversation id
472+
backend->>generateResponse: get conversation id
473+
backend->>backend: generate first chunk
474+
backend->>generateResponse: chunk received
475+
generateResponse->>vscode: send chunk to display
476+
vscode->>chat: display chunk
477+
end
478+
loop for each subsequent chunk
479+
backend->>backend: generate next chunk
480+
backend->>generateResponse: chunk received
481+
generateResponse->>vscode: send chunk to display
482+
vscode->>chat: display chunk
483+
end
484+
```
485+
486+
### cwsprChatTimeBetweenDisplays
487+
488+
An array of time when successive pieces of server responses are displayed to the user
489+
490+
```mermaid
491+
sequenceDiagram
492+
participant user as User
493+
participant chat as Chat UI
494+
participant vscode as VSCode extension host
495+
participant generateResponse as Generate response
496+
participant backend as Q service backend
497+
498+
user->>chat: Presses enter with message
499+
chat->>vscode: Tell VSCode to generate a response
500+
vscode->>generateResponse: start generating
501+
generateResponse->>backend: start stream
502+
backend->>backend: create conversation id
503+
backend->>generateResponse: get conversation id
504+
505+
note over backend, chat: cwsprChatTimeBetweenDisplays
506+
rect rgb(230, 230, 230, 0.5)
507+
loop for each subsequent chunk
508+
backend->>backend: generate next chunk
509+
backend->>generateResponse: chunk received
510+
generateResponse->>vscode: send chunk to display
511+
vscode->>chat: display chunk
512+
chat->>vscode: record display timestamp
513+
end
514+
end
515+
```
516+
517+
### cwsprChatFullDisplayLatency
518+
519+
The time between the user pressing enter and the entire response being rendered
520+
521+
```mermaid
522+
sequenceDiagram
523+
participant user as User
524+
participant chat as Chat UI
525+
participant vscode as VSCode extension host
526+
participant generateResponse as Generate response
527+
participant backend as Q service backend
528+
529+
note over backend, user: cwsprChatFullDisplayLatency
530+
rect rgb(230, 230, 230, 0.5)
531+
user->>chat: Presses enter with message
532+
chat->>vscode: Tell VSCode to generate a response
533+
vscode->>generateResponse: start generating
534+
generateResponse->>backend: start stream
535+
backend->>backend: create conversation id
536+
backend->>generateResponse: get conversation id
537+
generateResponse->>backend: start stream
538+
backend->>backend: create conversation id
539+
loop for each subsequent chunk
540+
backend->>backend: generate next chunk
541+
backend->>vscode: send chunk to display
542+
vscode->>chat: display chunk
543+
end
544+
end
545+
546+
```
547+
262548
## Crash Monitoring
263549

264550
We make an attempt to gather information regarding when the IDE crashes, then report it to telemetry. This is the diagram of the steps that take place.

0 commit comments

Comments
 (0)