Skip to content

Commit 4c7ae0a

Browse files
jpinkney-awsjpinkney-aws
committed
docs(amazonq): Add latency metric diagrams for amazon q chat
1 parent 5e8ad31 commit 4c7ae0a

File tree

1 file changed

+285
-0
lines changed

1 file changed

+285
-0
lines changed

‎docs/telemetry-perf.md‎

Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,6 +259,291 @@ How long it took from when the user stopped pressing a key to when they were sho
259259
end
260260
```
261261

262+
## Amazon Q Chat
263+
264+
### amazonq_chatRoundTrip
265+
266+
Measures sequential response times in Q chat, from user input to message display. Tracks time intervals between key events: editor receiving the message, feature processing, and final message rendering
267+
268+
```mermaid
269+
sequenceDiagram
270+
participant User
271+
participant chat as Chat UI
272+
participant vscode as VSCode
273+
participant event as Event Recorder
274+
participant partner as Partner team code
275+
participant telemetry
276+
277+
User->>chat: Write chat message and press enter
278+
chat->>vscode: send message with timestamp
279+
vscode->>event: record chatMessageSent/editorReceivedMessage timestamps
280+
vscode->>partner: forward chat message
281+
partner->>event: record featureReceivedMessage timestamp
282+
partner->>partner: call backend/get response
283+
partner->>vscode: forward response contents
284+
vscode->>chat: display message
285+
chat->>vscode: send stop-chat-message-telemetry event
286+
vscode->>event: record messageDisplayed timestamp
287+
vscode->>telemetry: emit amazonq_chatRoundTrip
288+
```
289+
290+
### cwsprChatTimeToFirstChunk
291+
292+
The time between when the conversation id was created and when we got back the first usable result
293+
294+
```mermaid
295+
sequenceDiagram
296+
participant user as User
297+
participant chat as Chat UI
298+
participant vscode as VSCode
299+
participant generateResponse as Generate response
300+
participant backend
301+
302+
user->>chat: Presses enter with message
303+
chat->>vscode: Tell VSCode to generate a response
304+
vscode->>generateResponse: start generating
305+
generateResponse->>backend: start stream
306+
backend->>backend: create conversation id
307+
backend->>generateResponse: get conversation id
308+
note over backend, generateResponse: cwsprChatTimeToFirstChunk
309+
rect rgb(230, 230, 230, 0.5)
310+
backend->>backend: generate first chunk
311+
backend->>generateResponse: chunk received
312+
end
313+
generateResponse->>vscode: send chunk to display
314+
vscode->>chat: display chunk
315+
loop for each subsequent chunk
316+
backend->>backend: generate next chunk
317+
backend->>generateResponse: chunk received
318+
generateResponse->>vscode: send chunk to display
319+
vscode->>chat: display chunk
320+
end
321+
```
322+
323+
### cwsprChatTimeBetweenChunks
324+
325+
An array of time when successive chunks of data are received from the server
326+
327+
```mermaid
328+
sequenceDiagram
329+
participant user as User
330+
participant chat as Chat UI
331+
participant vscode as VSCode
332+
participant generateResponse as Generate response
333+
participant backend
334+
335+
user->>chat: Presses enter with message
336+
chat->>vscode: Tell VSCode to generate a response
337+
vscode->>generateResponse: start generating
338+
generateResponse->>backend: start stream
339+
backend->>backend: create conversation id
340+
backend->>generateResponse: get conversation id
341+
342+
loop for each subsequent chunk
343+
note over backend, generateResponse: cwsprChatTimeBetweenChunks
344+
rect rgb(230, 230, 230, 0.5)
345+
backend->>backend: generate next chunk
346+
backend->>generateResponse: chunk received
347+
generateResponse->>generateResponse: record timestamp
348+
end
349+
350+
generateResponse->>vscode: send chunk to display
351+
vscode->>chat: display chunk
352+
end
353+
```
354+
355+
### cwsprChatFullResponseLatency
356+
357+
The time between when the conversation id was created and the final response from the server was received
358+
359+
```mermaid
360+
sequenceDiagram
361+
participant user as User
362+
participant chat as Chat UI
363+
participant vscode as VSCode
364+
participant generateResponse as Generate response
365+
participant backend
366+
367+
user->>chat: Presses enter with message
368+
chat->>vscode: Tell VSCode to generate a response
369+
vscode->>generateResponse: start generating
370+
generateResponse->>backend: start stream
371+
backend->>backend: create conversation id
372+
backend->>generateResponse: get conversation id
373+
374+
note over backend, chat: cwsprChatFullResponseLatency
375+
rect rgb(230, 230, 230, 0.5)
376+
loop for each subsequent chunk
377+
backend->>backend: generate next chunk
378+
backend->>generateResponse: chunk received
379+
generateResponse->>vscode: send chunk to display
380+
vscode->>chat: display chunk
381+
end
382+
backend->>generateResponse: final chunk received
383+
end
384+
generateResponse->>vscode: send chunk to display
385+
vscode->>chat: display chunk
386+
```
387+
388+
### cwsprChatTimeToFirstUsableChunk
389+
390+
The time between the initial server request, including creating the conversation id, and the first usable result
391+
392+
```mermaid
393+
sequenceDiagram
394+
participant user as User
395+
participant chat as Chat UI
396+
participant vscode as VSCode
397+
participant generateResponse as Generate response
398+
participant backend
399+
400+
user->>chat: Presses enter with message
401+
chat->>vscode: Tell VSCode to generate a response
402+
vscode->>generateResponse: start generating
403+
note over backend, generateResponse: cwsprChatTimeToFirstUsableChunk
404+
rect rgb(230, 230, 230, 0.5)
405+
generateResponse->>backend: start stream
406+
backend->>backend: create conversation id
407+
backend->>generateResponse: get conversation id
408+
backend->>backend: generate first chunk
409+
backend->>generateResponse: chunk received
410+
end
411+
generateResponse->>vscode: send chunk to display
412+
vscode->>chat: display chunk
413+
loop for each subsequent chunk
414+
backend->>backend: generate next chunk
415+
backend->>generateResponse: chunk received
416+
generateResponse->>vscode: send chunk to display
417+
vscode->>chat: display chunk
418+
end
419+
```
420+
421+
### cwsprChatFullServerResponseLatency
422+
423+
The time between the initial server request, including creating the conversation id, and the final response from the server
424+
425+
```mermaid
426+
sequenceDiagram
427+
participant user as User
428+
participant chat as Chat UI
429+
participant vscode as VSCode
430+
participant generateResponse as Generate response
431+
participant backend
432+
433+
user->>chat: Presses enter with message
434+
chat->>vscode: Tell VSCode to generate a response
435+
vscode->>generateResponse: start generating
436+
note over backend, chat: cwsprChatFullServerResponseLatency
437+
rect rgb(230, 230, 230, 0.5)
438+
generateResponse->>backend: start stream
439+
backend->>backend: create conversation id
440+
backend->>generateResponse: get conversation id
441+
loop for each subsequent chunk
442+
backend->>backend: generate next chunk
443+
backend->>generateResponse: chunk received
444+
generateResponse->>vscode: send chunk to display
445+
vscode->>chat: display chunk
446+
end
447+
backend->>generateResponse: final chunk received
448+
end
449+
generateResponse->>vscode: send chunk to display
450+
vscode->>chat: display chunk
451+
```
452+
453+
### cwsprChatTimeToFirstDisplay
454+
455+
The time between the user pressing enter and when the first chunk of data is displayed to the user
456+
457+
```mermaid
458+
sequenceDiagram
459+
participant user as User
460+
participant chat as Chat UI
461+
participant vscode as VSCode
462+
participant generateResponse as Generate response
463+
participant backend
464+
note over backend, user: cwsprChatTimeToFirstDisplay
465+
rect rgb(230, 230, 230, 0.5)
466+
user->>chat: Presses enter with message
467+
chat->>vscode: Tell VSCode to generate a response
468+
vscode->>generateResponse: start generating
469+
generateResponse->>backend: start stream
470+
backend->>backend: create conversation id
471+
backend->>generateResponse: get conversation id
472+
backend->>backend: generate first chunk
473+
backend->>generateResponse: chunk received
474+
generateResponse->>vscode: send chunk to display
475+
vscode->>chat: display chunk
476+
end
477+
loop for each subsequent chunk
478+
backend->>backend: generate next chunk
479+
backend->>generateResponse: chunk received
480+
generateResponse->>vscode: send chunk to display
481+
vscode->>chat: display chunk
482+
end
483+
```
484+
485+
### cwsprChatTimeBetweenDisplays
486+
487+
An array of time when successive chunks of server responses are displayed to the user
488+
489+
```mermaid
490+
sequenceDiagram
491+
participant user as User
492+
participant chat as Chat UI
493+
participant vscode as VSCode
494+
participant generateResponse as Generate response
495+
participant backend
496+
497+
user->>chat: Presses enter with message
498+
chat->>vscode: Tell VSCode to generate a response
499+
vscode->>generateResponse: start generating
500+
generateResponse->>backend: start stream
501+
backend->>backend: create conversation id
502+
backend->>generateResponse: get conversation id
503+
504+
note over backend, chat: cwsprChatTimeBetweenDisplays
505+
rect rgb(230, 230, 230, 0.5)
506+
loop for each subsequent chunk
507+
backend->>backend: generate next chunk
508+
backend->>generateResponse: chunk received
509+
generateResponse->>vscode: send chunk to display
510+
vscode->>chat: display chunk
511+
chat->>vscode: record display timestamp
512+
end
513+
end
514+
```
515+
516+
### cwsprChatFullDisplayLatency
517+
518+
The time between the user pressing enter and the entire response being rendered
519+
520+
```mermaid
521+
sequenceDiagram
522+
participant user as User
523+
participant chat as Chat UI
524+
participant vscode as VSCode
525+
participant generateResponse as Generate response
526+
participant backend
527+
528+
note over backend, user: cwsprChatFullDisplayLatency
529+
rect rgb(230, 230, 230, 0.5)
530+
user->>chat: Presses enter with message
531+
chat->>vscode: Tell VSCode to generate a response
532+
vscode->>generateResponse: start generating
533+
generateResponse->>backend: start stream
534+
backend->>backend: create conversation id
535+
backend->>generateResponse: get conversation id
536+
generateResponse->>backend: start stream
537+
backend->>backend: create conversation id
538+
loop for each subsequent chunk
539+
backend->>backend: generate next chunk
540+
backend->>vscode: send chunk to display
541+
vscode->>chat: display chunk
542+
end
543+
end
544+
545+
```
546+
262547
## Crash Monitoring
263548

264549
We make an attempt to gather information regarding when the IDE crashes, then report it to telemetry. This is the diagram of the steps that take place.

0 commit comments

Comments
 (0)