Commit 53c4647
authored
Implement
## Problem
There are a number of new features which are available for the `2025-10`
API release, and the SDK needs to be updated to support them.
Features:
- BYOC
- Dedicated Read Nodes
- Index Schema
- Fetch and Update vectors by metadata filter
- Create Namespace
## Solution
Implement `2025-10` features.
### BYOC (Bring Your Own Cloud)
- `BYOCSpec` has been added to `models.go`, and `IndexSpec` has been
updated to include this as a possible union type. Code has been updated
to deserialize the proper specification when receiving results from
operations.
- `CreateBYOCIndex` method has been added to `Client`, and
`CreateBYOCIndexRequest` type has been added to `client.go`.
- `PrivateHost` is already available on `IndexModel`.
### DRN (Dedicated Read Nodes)
- `ReadCapacityParams` and `ReadCapacityDedicatedConfig` types added to
`client.go`. These can now be used in `CreateServerlessIndexRequest`,
`CreateIndexForModelRequest`, and `ConfigureIndexParams` for creating
and configuring indexes for DRN.
- `ReadCapacity`, `ReadCapacityOnDemand`, `ReadCapacityDedicated`,
`ReadCapacityScaling`, `ReadCapacityManualScaling`, and
`ReadCapacityStatus` have been added to `models.go`. These are used when
deserializing the `ReadCapacity` on `ServerlessSpec`.
`ReadCapacityScaling` is also used in `ReadCapacityDedicatedConfig`.
### Index Schema
- `MetadataSchema` and `MetadataSchemaField` have been added to
`models.go`. `MetadataSchema` has been added to
`CreateServerlessIndexRequest`, `CreateIndexForModelRequest,
`CreateBYOCIndexRequest`, `ServerlessSpec`, and `Backup`.
- The `Schema` can be applied to an index when creating a new index, or
for individual namespaces when calling the new
`IndexConnection.CreateNamespace` method.
### Fetch and Update vectors by metadata
- `FetchVectorsByMetadata` has been added to `IndexConnection`, and
`FetchVectorsByMetadataRequest` and `FetchVectorsByMetadataResponse`
have been added to `index_connection.go`.
- `UpdateVectorsByMetadata` has been added to `IndexConnection`, and
`UpdateVectorsByMetadataRequest` and `UpdateVectorsByMetadataResponse`
have been added to `index_connection.go`.
### Namespaces
- `CreateNamespace` has been added to `IndexConnection`, and
`CreateNamespaceParams` has been added to `index_connection.go`.
- `Prefix` has been added to `ListNamespacesParams`, and `TotalCount`
has been added to `ListNamespacesResponse`.
## Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] This change requires a documentation update
- [ ] Infrastructure change (CI configs, etc)
- [ ] Non-code change (docs, etc)
- [ ] None of the above: (explain here)
## Test Plan
New unit tests for utility functions added in `client_test.go` and
`index_connection_test.go`.
Integration tests have been updated and added to cover new
functionality.
There should ultimately be no breaking changes to the existing
interface, so integrations should run as expected. You'll want to test
the new functionality around creating and configuring and index schema,
updating and filtering with metadata, and creating a new namespace.
You can find code snippets in the integration tests and doc comments.
Here's some example code:
### Create serverless index with dedicated ReadCapacity and Schema:
```go
ctx := context.Background()
clientParams := pinecone.NewClientParams{
ApiKey: "YOUR_API_KEY",
}
pc, err := pinecone.NewClient(clientParams)
if err != nil {
panic(fmt.Errorf("Failed to create Client: %v", err))
}
dimension := int32(128)
metric := pinecone.Cosine
// Define a schema for metadata indexing
schema := &pinecone.MetadataSchema{
Fields: map[string]pinecone.MetadataSchemaField{
"category": {Filterable: true},
"rating": {Filterable: true},
"year": {Filterable: true},
},
}
// Configure dedicated read capacity
readCapacity := &pinecone.ReadCapacityParams{
Dedicated: &pinecone.ReadCapacityDedicatedConfig{
NodeType: "t1",
Scaling: &pinecone.ReadCapacityScaling{
Manual: &pinecone.ReadCapacityManualScaling{
Replicas: 1,
Shards: 1,
},
},
},
}
// Create the serverless index
idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{
Name: "my-serverless-index",
Dimension: &dimension,
Metric: &metric,
Cloud: pinecone.Aws,
Region: "us-east-1",
ReadCapacity: readCapacity,
Schema: schema,
})
if err != nil {
log.Fatalf("Failed to create serverless index: %v", err)
}
fmt.Printf("Successfully created index: %s\n", idx.Name)
```
### Configuring an index to update ReadCapacity
```go
ctx := context.Background()
// Update the read capacity configuration
updatedReadCapacity := &pinecone.ReadCapacityParams{
Dedicated: &pinecone.ReadCapacityDedicatedConfig{
NodeType: "t1",
Scaling: &pinecone.ReadCapacityScaling{
Manual: &pinecone.ReadCapacityManualScaling{
Replicas: 2, // Scale up replicas
Shards: 2, // Scale up shards
},
},
},
}
// Configure the index with updated read capacity
updatedIdx, err := pc.ConfigureIndex(ctx, "my-serverless-index", pinecone.ConfigureIndexParams{
ReadCapacity: updatedReadCapacity,
})
if err != nil {
log.Fatalf("Failed to configure index: %v", err)
}
fmt.Printf("Successfully updated index read capacity. Status: %s\n",
updatedIdx.Spec.Serverless.ReadCapacity.Dedicated.Status.State)
```
### Create a new namespace with a different schema
```go
ctx := context.Background()
// Get the index connection
idx, err := pc.DescribeIndex(ctx, "my-serverless-index")
if err != nil {
log.Fatalf("Failed to describe index: %v", err)
}
idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: idx.Host})
if err != nil {
log.Fatalf("Failed to create IndexConnection: %v", err)
}
// Define a slightly different schema for the new namespace
// (e.g., adding a new field, removing a field, or keeping some fields)
namespaceSchema := &pinecone.MetadataSchema{
Fields: map[string]pinecone.MetadataSchemaField{
"category": {Filterable: true},
"rating": {Filterable: true},
"tags": {Filterable: true}, // New field not in index schema
// Note: "year" is not included in this namespace schema
},
}
// Create the namespace with the custom schema
namespace, err := idxConnection.CreateNamespace(ctx, &pinecone.CreateNamespaceParams{
Name: "my-custom-namespace",
Schema: namespaceSchema,
})
if err != nil {
log.Fatalf("Failed to create namespace: %v", err)
}
fmt.Printf("Successfully created namespace: %s with %d records\n",
namespace.Name, namespace.RecordCount)
```
### Update / Fetch vectors by metadata
```go
ctx := context.Background()
idxConnectionWithNamespace, err := pc.Index(pinecone.NewIndexConnParams{
Host: idx.Host,
Namespace: "my-custom-namespace",
})
if err != nil {
log.Fatalf("Failed to create IndexConnection with namespace: %v", err)
}
vectors := []*pinecone.Vector{
{
Id: "vector-1",
Values: &[]float32{0.1, 0.2, 0.3, 0.4}, // Example values
Metadata: func() *pinecone.Metadata {
m, err := pinecone.NewMetadata(map[string]interface{}{
"category": "electronics",
"rating": 4.5,
"tags": []string{"popular", "trending"},
})
if err != nil {
panic(fmt.Errorf("error creating MetadataSchema: %w", err))
}
return m
}(),
},
{
Id: "vector-2",
Values: &[]float32{0.2, 0.3, 0.4, 0.5},
Metadata: func() *pinecone.Metadata {
m, _ := pinecone.NewMetadata(map[string]interface{}{
"category": "electronics",
"rating": 3.8,
"tags": []string{"new"},
})
if err != nil {
panic(fmt.Errorf("error creating MetadataSchema: %w", err))
}
}(),
},
}
_, err = idxConnectionWithNamespace.Upsert(ctx, vectors, nil)
if err != nil {
log.Fatalf("Failed to upsert vectors: %v", err)
}
// Update vectors by metadata filter
// Update all vectors with category "electronics" and rating >= 4.0
filterMap := map[string]interface{}{
"$and": []interface{}{
map[string]interface{}{
"category": map[string]interface{}{"$eq": "electronics"},
},
map[string]interface{}{
"rating": map[string]interface{}{"$gte": 4.0},
},
},
}
filter, err := pinecone.NewMetadataFilter(filterMap)
if err != nil {
log.Fatalf("Failed to create metadata filter: %v", err)
}
// Update metadata for matching vectors
updateMetadata, err := pinecone.NewMetadata(map[string]interface{}{
"category": "electronics",
"rating": 5.0, // Update rating to 5.0
"tags": []string{"popular", "trending", "featured"}, // Add new tag
})
if err != nil {
log.Fatalf("Failed to create update metadata: %v", err)
}
updateRes, err := idxConnectionWithNamespace.UpdateVectorsByMetadata(ctx, &pinecone.UpdateVectorsByMetadataRequest{
Filter: filter,
Metadata: updateMetadata,
})
if err != nil {
log.Fatalf("Failed to update vectors by metadata: %v", err)
}
fmt.Printf("Updated %d vector(s) matching the filter\n", updateRes.MatchedRecords)
// Fetch vectors by metadata filter
// Fetch all vectors with category "electronics" and rating >= 4.5
fetchFilterMap := map[string]interface{}{
"$and": []interface{}{
map[string]interface{}{
"category": map[string]interface{}{"$eq": "electronics"},
},
map[string]interface{}{
"rating": map[string]interface{}{"$gte": 4.5},
},
},
}
fetchFilter, err := pinecone.NewMetadataFilter(fetchFilterMap)
if err != nil {
log.Fatalf("Failed to create fetch filter: %v", err)
}
limit := uint32(10)
fetchRes, err := idxConnectionWithNamespace.FetchVectorsByMetadata(ctx, &pinecone.FetchVectorsByMetadataRequest{
Filter: fetchFilter,
Limit: &limit,
})
if err != nil {
log.Fatalf("Failed to fetch vectors by metadata: %v", err)
}
fmt.Printf("Fetched %d vector(s) matching the filter:\n", len(fetchRes.Vectors))
for id, vector := range fetchRes.Vectors {
fmt.Printf(" Vector ID: %s, Metadata: %v\n", id, vector.Metadata)
}
```
---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
- https://app.asana.com/0/0/1211666868889403
- https://app.asana.com/0/0/1211666868889406
- https://app.asana.com/0/0/12116668688894002025-10 Features: DRN, BYOC, Index Schema / Metadata Update & Filter, CreateNamespace (#125)1 parent 5d64bbd commit 53c4647
File tree
10 files changed
+2817
-792
lines changed- codegen
- internal/gen/db_data/rest
- pinecone
10 files changed
+2817
-792
lines changedSubmodule apis updated from 158e273 to d5ac931
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
3 | 18 | | |
4 | 19 | | |
5 | 20 | | |
6 | 21 | | |
7 | 22 | | |
8 | 23 | | |
| 24 | + | |
9 | 25 | | |
10 | 26 | | |
11 | 27 | | |
12 | 28 | | |
13 | 29 | | |
14 | 30 | | |
| 31 | + | |
15 | 32 | | |
16 | 33 | | |
17 | 34 | | |
18 | 35 | | |
19 | 36 | | |
| 37 | + | |
20 | 38 | | |
21 | 39 | | |
| 40 | + | |
22 | 41 | | |
23 | 42 | | |
24 | 43 | | |
0 commit comments