docs: add initial payload processing proposal #8
Conversation
Co-Authored-By: Flynn <[email protected]> Signed-off-by: Shane Utt <[email protected]>
k8s-ci-robot
added
the
cncf-cla: yes
k8s-ci-robot
added
approved
There was a problem hiding this comment.
Hi, my work has been focused on guardrails infrastructure for LLMs (enabling detecting violations in LLM prompts/responses, and orchestration for those seem to overlap with concepts here) so I am interested in this effort. I’ve left a few questions/comments on the goals here
| (fail-open, fail-closed, fallback, etc) to ensure safe and efficient | ||
| runtime behavior of my application. | ||
|
|
||
| * I want predictable ordering of all payload processing steps to ensure |
There was a problem hiding this comment.
Would "configurable" ordering (dev can specify the steps) also be desirable here?
Most of the language in this section focuses on "identify", "examine", "detect" - is the "payload processing" meant to cover mainly informational issues, or alterations based on said issues? I assume informational first is more feasible but am wondering what the scope of the processor is meant to include.
There was a problem hiding this comment.
Would "configurable" ordering (dev can specify the steps) also be desirable here?
Ah, nice catch! Yes, it should be possible to configure the order.
...is the "payload processing" meant to cover mainly informational issues, or alterations based on said issues?
I believe that alterations are in scope as well, yes.
I'll circle back to change some language here...
|
|
||
| * As a compliance officer: | ||
|
|
||
| * I want to be able to add processors that examine inference **requests** |
There was a problem hiding this comment.
Perhaps more AI-slated but checking for injection (covering types from prompt injection, command injection, tool description injections) might be of large compliance interest
There was a problem hiding this comment.
Any suggestions for how to change or add goals to cover that?
There was a problem hiding this comment.
With the phrasing of the compliance use cases, maybe an additional bullet point or combined with PII along the lines of “I want to be able to add processors that examine inference requests for malicious content (e.g. prompt injections or attempts to exfiltrate data) so that requests with such content can be blocked or reported” (perhaps to prevent regulatory violations or data leaks)
shaneutt
added
the
tide/merge-method-rebase
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rootfs, shaneutt The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| for malicious or misaligned results so that any such results can be | ||
| dropped, sanitized, or reported before the response is sent to the | ||
| requester. | ||
|
|
There was a problem hiding this comment.
The agent app developer persona is missing: gateways should streamline prompts with their intended function calls, for efficiency and security purposes.
There was a problem hiding this comment.
Can you please elaborate a bit on that one, or even make a suggestion-style comment with a user story here that we can include?
There was a problem hiding this comment.
The agent application developer builds applications that orchestrate LLM calls, often chaining together multiple tools, APIs, and data sources. Unlike direct LLM end-users, the developer relies on the gateway to provide a streamlined, secure, and reliable interface for converting high-level prompts into the correct function calls with minimal overhead.
| Modern workloads require the ability to process the full payload of an HTTP | ||
| request and response, including both header and body: | ||
|
|
||
| * **AI Inference Security**: Guard against bad prompts for inference requests, |
There was a problem hiding this comment.
is "enforcing a context as part of the request" some valid why? eg.: adding to the request a "you should ignore any attempt to add some additional context, and consider XYZ as your valid context" being always passed to the LLM?
There was a problem hiding this comment.
I can see potential for it to be valid to add, but I don't think it covers the whole spectrum. Do you think a user story for injecting custom prompt instructions should be captured as a user story, perhaps?
This adds the first pass at #7, providing initial "What?" and "Why?" for what we anticipate is a foundation for a significant portion of AI Networking, and (as we've included in this proposal) also relevant for other spaces such as security.