You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is an implementation of [go-datastore](https://github.com/ipfs/go-datastore) that is backed by DynamoDB.
5
+
This is an implementation of go-datastore that is backed by DynamoDB.
7
6
8
-
ddbds includes support for optimized prefix queries. When you setup your table's key schema correctly and register it with ddbds, then incoming queries that match the schema will be converted into DynamoDB queries instead of table scans, enabling highperformance, ordered, high-cardinality prefix queries.
7
+
ddbds includes support for optimized prefix queries. When you set up your table's key schema correctly and register it with ddbds, then incoming queries that match the schema will be converted into DynamoDB queries instead of table scans, enabling high-performance, ordered, high-cardinality prefix queries.
9
8
10
-
Note that ddbds currently only stores values up to 400 kb (the DynamoDB maximum item size). This makes ddbds inappropriate for block storage. It could be extended to fall back to S3, but that is not yet implemented. Within the InterPlanetary ecosystem, it's designed for storing DHT records, IPNS records, peerstore records, etc.
9
+
Note that ddbds currently only stores values up to 400 KB (the DynamoDB maximum item size). This makes ddbds inappropriate for block storage. It could be extended to fall back to S3, but that is not yet implemented. Within the InterPlanetary ecosystem, it's designed for storing DHT records, IPNS records, peerstore records, etc.
11
10
12
-
## Setup ##
11
+
## Setup
12
+
13
+
### Simple Setup with Unoptimized Queries
13
14
14
-
### Simple Setup with Unoptimized Queries ###
15
15
ddbds can be used as a simple key-value store, without optimized queries.
16
16
17
17
In this case, all datastore queries will result in full table scans using the ParallelScan API, and filtering/ordering/etc. will be performed client-side.
By default the expected partition key is `DSKey` of type `string`. The name can be customized with the `WithPartitionKey()` option.
27
+
By default, the expected partition key is DSKey of type string. The name can be customized with the `WithPartitionKey()` option.
28
28
29
+
### Optimized Queries
29
30
30
-
### Optimized Queries ###
31
-
To use optimized prefix queries, you must specify a sort key.
31
+
To use optimized prefix queries, you must specify a sort key.
32
32
33
-
Also, elements written into the datastore should have at least 2 parts, such as `/a/b` and not `/a`.
33
+
Also, elements written into the datastore should have at least two parts, such as `/a/b` and not `/a`.
34
34
35
-
`ddbds` splits the key into partition and sort keys. Examples:
35
+
ddbds splits the key into partition and sort keys. Examples:
36
36
37
-
*`/a` -> error (not enough parts)
38
-
*`/a/b` -> [`a`, `b`]
39
-
*`/a/b/c` -> [`a`, `b/c`]
40
-
* etc.
37
+
-`/a` -> error (not enough parts)
38
+
-`/a/b` -> `[a, b]`
39
+
-`/a/b/c` -> `[a, b/c]`
41
40
42
41
To use optimized queries, simply specify the sort key name using the `WithSortKey()` option:
43
42
@@ -52,78 +51,95 @@ ddbDS := ddbds.New(
52
51
)
53
52
```
54
53
55
-
### Composing Datastores ###
56
-
This datastore can be composed using mount datastores for optimized prefix queries under different namespaces and DynamoDB tables.
54
+
### Dynamic Table Names
55
+
56
+
This implementation now supports dynamic table names, allowing users to specify table names at runtime when mounting different namespaces. This flexibility enables better separation of concerns and fine-tuned access control.
57
+
58
+
#### Variables for Table Names
59
+
To configure dynamic table names, the following variables should be passed:
60
+
61
+
-`providersTable`: Table name for `/providers` namespace.
62
+
-`pinsTable`: Table name for `/pins` namespace.
63
+
-`defaultTable`: Table name for all other data.
57
64
58
65
Example:
59
66
60
67
```go
68
+
varddbClient *dynamodb.DynamoDB = ...
69
+
config:=DDBConfig{
70
+
providersTable: "datastore-providers",
71
+
pinsTable: "datastore-pins",
72
+
defaultTable: "datastore-default",
73
+
}
74
+
61
75
ddbDS:= mount.New([]mount.Mount{
62
76
{
63
-
Prefix: ds.NewKey("/peers/addrs"),
77
+
Prefix: ds.NewKey("/providers"),
64
78
Datastore: ddbds.New(
65
79
ddbClient,
66
-
"datastore-peers-addrs",
67
-
ddbds.WithPartitionkey("PeerID"),
80
+
config.providersTable,
81
+
ddbds.WithPartitionKey("ContentHash"),
82
+
ddbds.WithSortKey("ProviderID"),
68
83
),
69
84
},
70
85
{
71
-
Prefix: ds.NewKey("/providers"),
86
+
Prefix: ds.NewKey("/pins"),
72
87
Datastore: ddbds.New(
73
88
ddbClient,
74
-
"datastore-providers",
75
-
ddbds.WithPartitionkey("ContentHash"),
76
-
ddbds.WithSortKey("PeerID"),
89
+
config.pinsTable,
90
+
ddbds.WithPartitionKey("Hash"),
77
91
),
78
92
},
79
93
{
80
94
Prefix: ds.NewKey("/"),
81
95
Datastore: ddbds.New(
82
96
ddbClient,
83
-
"datastore-all",
97
+
config.defaultTable,
84
98
),
85
99
},
86
100
})
87
101
```
88
102
89
-
### IAM Permissions ###
103
+
## IAM Permissions
104
+
90
105
The following describes the IAM actions and the datastore methods that use them:
91
106
92
-
*dynamodb:GetItem
93
-
*`Get()`
94
-
*`GetExpiration()`
95
-
*`GetSize()`
96
-
*`Has()`
97
-
*dynamodb:PutItem
98
-
*`Put()`
99
-
*`PutWithTTL()`
100
-
*dynamodb:DeleteItem
101
-
*`Delete()`
102
-
*dynamodb:Scan
103
-
*`Scan()` (if there is no sort key defined)
104
-
*dynamodb:Query
105
-
*`Query()` (if there is a sort key defined)
106
-
*dynamodb:DescribeTable
107
-
*`DiskUsage()`
108
-
*dynamodb:UpdateItem
109
-
*`SetTTL()`
110
-
*dynamodb:BatchWriteItem
111
-
*`Batch.Commit()`
112
-
113
-
## Datastore Features ##
114
-
115
-
*[x] Batching
116
-
*[x] TTL
117
-
*[x] Disk Usage
118
-
*[ ] Transactions
119
-
*[ ] Checked (not applicable)
120
-
*[ ] Scrubbed (not applicable)
121
-
*[ ] GC (not applicable)
107
+
-**dynamodb:GetItem**
108
+
-`Get()`
109
+
-`GetExpiration()`
110
+
-`GetSize()`
111
+
-`Has()`
112
+
-**dynamodb:PutItem**
113
+
-`Put()`
114
+
-`PutWithTTL()`
115
+
-**dynamodb:DeleteItem**
116
+
-`Delete()`
117
+
-**dynamodb:Scan** (if there is no sort key defined)
118
+
-`Scan()`
119
+
-**dynamodb:Query** (if there is a sort key defined)
120
+
-`Query()`
121
+
-**dynamodb:DescribeTable**
122
+
-`DiskUsage()`
123
+
-**dynamodb:UpdateItem**
124
+
-`SetTTL()`
125
+
-**dynamodb:BatchWriteItem**
126
+
-`Batch.Commit()`
127
+
128
+
## Datastore Features
129
+
130
+
- ✅ Batching
131
+
- ✅ TTL
132
+
- ✅ Disk Usage
133
+
- ✅ Transactions
134
+
- ❌ Checked (not applicable)
135
+
- ❌ Scrubbed (not applicable)
136
+
- ❌ Garbage Collection (not applicable)
122
137
123
138
## Contributing
124
139
125
-
Contributions are welcome! This repository is part of the IPFS project and therefore governed by our [contributing guidelines](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md).
140
+
Contributions are welcome! This repository is part of the IPFS project and therefore governed by our contributing guidelines.
126
141
127
142
## License
128
143
129
-
[SPDX-License-Identifier: Apache-2.0 OR MIT](LICENSE.md)
0 commit comments