Skip to content

Commit bfac1b8

Browse files
authored
docs: update apisix-upstream_response_time and request_llm_model in access log info (apache#12583)
1 parent 1934d56 commit bfac1b8

File tree

2 files changed

+10
-6
lines changed

2 files changed

+10
-6
lines changed

docs/en/latest/plugins/ai-proxy-multi.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -940,6 +940,8 @@ For verification, the behaviours should be consistent with the verification in [
940940

941941
The following example demonstrates how you can log LLM request related information in the gateway's access log to improve analytics and audit. The following variables are available:
942942

943+
* `request_llm_model`: LLM model name specified in the request.
944+
* `apisix_upstream_response_time`: Time taken for APISIX to send the request to the upstream service and receive the full response
943945
* `request_type`: Type of request, where the value could be `traditional_http`, `ai_chat`, or `ai_stream`.
944946
* `llm_time_to_first_token`: Duration from request sending to the first token received from the LLM service, in milliseconds.
945947
* `llm_model`: LLM model.
@@ -951,7 +953,7 @@ Update the access log format in your configuration file to include additional LL
951953
```yaml title="conf/config.yaml"
952954
nginx_config:
953955
http:
954-
access_log_format: "$remote_addr - $remote_user [$time_local] $http_host \"$request_line\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\" \"$apisix_request_id\" \"$request_type\" \"$llm_time_to_first_token\" \"$llm_model\" \"$llm_prompt_tokens\" \"$llm_completion_tokens\""
956+
access_log_format: "$remote_addr - $remote_user [$time_local] $http_host \"$request_line\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $apisix_upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\" \"$apisix_request_id\" \"$request_type\" \"$llm_time_to_first_token\" \"$llm_model\" \"$request_llm_model\" \"$llm_prompt_tokens\" \"$llm_completion_tokens\""
955957
```
956958
957959
Reload APISIX for configuration changes to take effect.
@@ -993,7 +995,7 @@ Next, create a Route with the `ai-proxy-multi` Plugin and send a request. For in
993995
In the gateway's access log, you should see a log entry similar to the following:
994996

995997
```text
996-
192.168.215.1 - - [21/Mar/2025:04:28:03 +0000] api.openai.com "POST /anything HTTP/1.1" 200 804 2.858 "-" "curl/8.6.0" - - - "http://api.openai.com" "5c5e0b95f8d303cb81e4dc456a4b12d9" "ai_chat" "2858" "gpt-4" "23" "8"
998+
192.168.215.1 - - [21/Mar/2025:04:28:03 +0000] api.openai.com "POST /anything HTTP/1.1" 200 804 2.858 "-" "curl/8.6.0" - - - 5765 "http://api.openai.com" "5c5e0b95f8d303cb81e4dc456a4b12d9" "ai_chat" "2858" "gpt-4" "gpt-4" "23" "8"
997999
```
9981000

999-
The access log entry shows the request type is `ai_chat`, time to first token is `2858` milliseconds, LLM model is `gpt-4`, prompt token usage is `23`, and completion token usage is `8`.
1001+
The access log entry shows the request type is `ai_chat`, Apisix upstream response time is `5765` milliseconds, time to first token is `2858` milliseconds, Requested LLM model is `gpt-4`. LLM model is `gpt-4`, prompt token usage is `23`, and completion token usage is `8`.

docs/en/latest/plugins/ai-proxy.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -388,6 +388,8 @@ You should receive a response similar to the following:
388388

389389
The following example demonstrates how you can log LLM request related information in the gateway's access log to improve analytics and audit. The following variables are available:
390390

391+
* `request_llm_model`: LLM model name specified in the request.
392+
* `apisix_upstream_response_time`: Time taken for APISIX to send the request to the upstream service and receive the full response
391393
* `request_type`: Type of request, where the value could be `traditional_http`, `ai_chat`, or `ai_stream`.
392394
* `llm_time_to_first_token`: Duration from request sending to the first token received from the LLM service, in milliseconds.
393395
* `llm_model`: LLM model.
@@ -399,7 +401,7 @@ Update the access log format in your configuration file to include additional LL
399401
```yaml title="conf/config.yaml"
400402
nginx_config:
401403
http:
402-
access_log_format: "$remote_addr $remote_user [$time_local] $http_host \"$request_line\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\" \"$apisix_request_id\" \"$request_type\" \"$llm_time_to_first_token\" \"$llm_model\" \"$llm_prompt_tokens\" \"$llm_completion_tokens\""
404+
access_log_format: "$remote_addr - $remote_user [$time_local] $http_host \"$request_line\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $apisix_upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\" \"$apisix_request_id\" \"$request_type\" \"$llm_time_to_first_token\" \"$llm_model\" \"$request_llm_model\" \"$llm_prompt_tokens\" \"$llm_completion_tokens\""
403405
```
404406
405407
Reload APISIX for configuration changes to take effect.
@@ -441,7 +443,7 @@ Now if you create a Route and send a request following the [Proxy to OpenAI exam
441443
In the gateway's access log, you should see a log entry similar to the following:
442444

443445
```text
444-
192.168.215.1 - [21/Mar/2025:04:28:03 +0000] api.openai.com "POST /anything HTTP/1.1" 200 804 2.858 "-" "curl/8.6.0" - "http://api.openai.com" "5c5e0b95f8d303cb81e4dc456a4b12d9" "ai_chat" "2858" "gpt-4" "23" "8"
446+
192.168.215.1 - - [21/Mar/2025:04:28:03 +0000] api.openai.com "POST /anything HTTP/1.1" 200 804 2.858 "-" "curl/8.6.0" - - - 5765 "http://api.openai.com" "5c5e0b95f8d303cb81e4dc456a4b12d9" "ai_chat" "2858" "gpt-4" "gpt-4" "23" "8"
445447
```
446448

447-
The access log entry shows the request type is `ai_chat`, time to first token is `2858` milliseconds, LLM model is `gpt-4`, prompt token usage is `23`, and completion token usage is `8`.
449+
The access log entry shows the request type is `ai_chat`, Apisix upstream response time is `5765` milliseconds, time to first token is `2858` milliseconds, Requested LLM model is `gpt-4`. LLM model is `gpt-4`, prompt token usage is `23`, and completion token usage is `8`.

0 commit comments

Comments
 (0)