Skip to content

Commit 9d784d1

Browse files
typo
1 parent b5d6a58 commit 9d784d1

File tree

1 file changed

+4
-4
lines changed

1 file changed

+4
-4
lines changed

content/ai_exchange/content/docs/3_development_time_threats.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ Data and models may have been obtained externally, just like software components
5757

5858
Training data is in most cases only present during development-time, but there are exceptions:
5959
- A machine learning model may be continuously trained with data collected at runtime, which puts (part of the) training data in the runtime environment, where it also needs protection - as covered in this control section
60-
- For GenAI, information can be retrieved from a repository to be added to a prompt, for example to inform a large language model about the context to take into account for an instruction or question. This principle is called _in-context learning_. For example [OpenCRE-chat](https://opencre.org/chatbot) uses a repository of requirements from security standards to add to a user question so that the large language model is more informed with background information. In the case of OpenCRE-chat this information is public, but in many cases the application of this so-called Retrieval Augmented Generation (RAG) will have a repository with company secrets or otherwise sensitive data. Organizations can benefit from unlocking their unique data, to be used by themselves, or to be provided as service or product. This is an attractive architecture because the alternative would be to train an LLM or to finetune it, which is expensive and difficult. A RAG approach may suffice. Effectively, this puts the repository data to the same use as training data is used: control the behaviour of the model. Therefore, the security controls that apply to train data, also apply to this run-time repository data. See [manipulate augmentation data](/goto/manipulateaugmentation).
60+
- For GenAI, information can be retrieved from a repository to be added to a prompt, for example to inform a large language model about the context to take into account for an instruction or question. This principle is called _in-context learning_. For example [OpenCRE-chat](https://opencre.org/chatbot) uses a repository of requirements from security standards to add to a user question so that the large language model is more informed with background information. In the case of OpenCRE-chat this information is public, but in many cases the application of this so-called Retrieval Augmented Generation (RAG) will have a repository with company secrets or otherwise sensitive data. Organizations can benefit from unlocking their unique data, to be used by themselves, or to be provided as service or product. This is an attractive architecture because the alternative would be to train an LLM or to finetune it, which is expensive and difficult. A RAG approach may suffice. Effectively, this puts the repository data to the same use as training data is used: control the behaviour of the model. Therefore, the security controls that apply to train data, also apply to this run-time repository data. See [augmentation data manipulation](/goto/augmentationdatamanipulation/).
6161

6262
**Details on the how: protection strategies:**
6363

@@ -623,8 +623,8 @@ Find the corresponding control section [here, with the other controls against Ev
623623
> Permalink: https://owaspai.org/goto/devmodelpoison/
624624
625625
**Description**
626-
This threat refers to manipulating behaviour of the model NOT by n poisoning the training data, but instead by manipulating elements in the development-environment that lead to the model or represent the model (i.e. model attributes), e.g. by manipulating storage of model parameters or placing the model with a completely different one with malicious behavior, injection of malware (command or code injection) through custom or lambda layers, manipulating the model weights and modifying the model architecture, embedding deserialization attacks, which could execute stealthily during model unpacking or model execution. When the model is trained by a supplier in a manipulative way and supplied as-is, then it is [supply-chain model poisoning](goto/supplymodelpoison/).
627-
Training data manipulation is referred to as [data poisoning](/goto/datapoison). See the attack surface diagram in the [broad model poisoning section](/goto/modelpoison/).
626+
This threat refers to manipulating behaviour of the model NOT by n poisoning the training data, but instead by manipulating elements in the development-environment that lead to the model or represent the model (i.e. model attributes), e.g. by manipulating storage of model parameters or placing the model with a completely different one with malicious behavior, injection of malware (command or code injection) through custom or lambda layers, manipulating the model weights and modifying the model architecture, embedding deserialization attacks, which could execute stealthily during model unpacking or model execution. When the model is trained by a supplier in a manipulative way and supplied as-is, then it is [supply-chain model poisoning](/goto/supplymodelpoison/).
627+
Training data manipulation is referred to as [data poisoning](/goto/datapoison/). See the attack surface diagram in the [broad model poisoning section](/goto/modelpoison/).
628628

629629
**Controls**
630630

@@ -704,7 +704,7 @@ Unauthorized access to model attributes (e.g., parameters, weights, architecture
704704

705705
Impact: Confidentiality breach of the model (i.e., model parameters), which can be:
706706
- intellectual property theft (e.g., by a competitor)
707-
- and/or a way to perform input attacks on the copied model, circumventing protections. These protections include rate limiting, access control, and detection mechanisms. This can be done for [all input attacks](/goto/inputthreats/) that extract data, and for the preparation of [evasion](/goto/evasion/) or [prompt injection](/goto/promptinjection): experimenting to find attack inputs that work.
707+
- and/or a way to perform input attacks on the copied model, circumventing protections. These protections include rate limiting, access control, and detection mechanisms. This can be done for [all input attacks](/goto/inputthreats/) that extract data, and for the preparation of [evasion](/goto/evasion/) or [prompt injection](/goto/promptinjection/): experimenting to find attack inputs that work.
708708

709709
Alternative ways of model theft are [model exfiltration](/goto/modelexfiltration/) and [direct runtime model leak](/goto/runtimemodelleak/).
710710

0 commit comments

Comments
 (0)