Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 4 additions & 7 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,11 +149,13 @@ The following terms are used in this section:

- **inputTypes** _array of string_, OPTIONAL

Input types that the model supports, such as "text", "image", "audio", "video", etc.
An array of strings specifying the data types that the model can accept as input.
The allowed values are: "text", "image", "audio", "video", or "embedding". For input types that are not explicitly defined, the value "other" value should be used.

- **outputTypes** _array of string_, OPTIONAL

Output types that the model supports, such as "text", "image", "audio", "video", etc.
An array of strings specifying the data types that the model can produce as output.
The allowed values are: "text", "image", "audio", "video", or "embedding". For output types that are not explicitly defined, the value "other" value should be used.

- **knowledgeCutoff** _string_, OPTIONAL

Expand All @@ -167,10 +169,6 @@ The following terms are used in this section:

Whether the model can use external tools or APIs to perform tasks.

- **embedding** _boolean_, OPTIONAL

Whether the model can perform embedding tasks.

- **reward** _boolean_, OPTIONAL

Whether the model is a reward model.
Expand Down Expand Up @@ -220,7 +218,6 @@ Here is an example model artifact configuration JSON document:
"knowledgeCutoff": "2024-05-21T00:00:00Z",
"reasoning": true,
"toolUsage": false,
"embedding": false,
"reward": false,
"languages": ["en", "zh"]
}
Expand Down
35 changes: 27 additions & 8 deletions specs-go/v1/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -98,12 +98,34 @@ type ModelDescriptor struct {
type Modality string

const (
TextModality Modality = "text"
ImageModality Modality = "image"
AudioModality Modality = "audio"
VideoModality Modality = "video"
// TextModality indicates that the model can process or generate text.
// If present in InputTypes, the model accepts text as input.
// If present in OutputTypes, the model produces text as output.
TextModality Modality = "text"

// ImageModality indicates that the model can process or generate images.
// If present in InputTypes, the model accepts images as input.
// If present in OutputTypes, the model produces images as output.
ImageModality Modality = "image"

// AudioModality indicates that the model can process or generate audio.
// If present in InputTypes, the model accepts audio as input.
// If present in OutputTypes, the model produces audio as output.
AudioModality Modality = "audio"

// VideoModality indicates that the model can process or generate video.
// If present in InputTypes, the model accepts video as input.
// If present in OutputTypes, the model produces video as output.
VideoModality Modality = "video"

// EmbeddingModality indicates that the model can process or generate embeddings.
// If present in InputTypes, the model accepts embeddings as input.
// If present in OutputTypes, the model produces embeddings as output.
// For a dedicated "embedding model", EmbeddingModality should be present in its OutputTypes.
Comment on lines +121 to +124
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This comment is excellent and clearly explains how to use the embedding modality. However, as this is user-facing documentation for the specification, it should be in docs/config.md. This would make it more discoverable for users of the spec. Please move this explanation to the inputTypes/outputTypes section in docs/config.md.

EmbeddingModality Modality = "embedding"
OtherModality Modality = "other"

// OtherModality indicates that the model supports a modality not explicitly listed.
OtherModality Modality = "other"
)

// ModelCapabilities defines the special capabilities that the model supports
Expand All @@ -124,9 +146,6 @@ type ModelCapabilities struct {
// such as a calculator, a search engine, etc.
ToolUsage *bool `json:"toolUsage,omitempty"`

// Embedding indicates whether the model can perform embedding tasks
Embedding *bool `json:"embedding,omitempty"`

// Reward indicates whether the model is a reward model
Reward *bool `json:"reward,omitempty"`

Expand Down