Skip to content

Commit 2b4d364

Browse files
committed
update readme
1 parent 61d40af commit 2b4d364

File tree

3 files changed

+137
-17
lines changed

3 files changed

+137
-17
lines changed

README.md

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,40 @@ Note that our table store was ```PocoTableStore<Employee, int, int>```, but that
188188
```var record = tableStore.GetRecord(142, "user");```
189189
which is both clear and provides type safety.
190190

191+
#### Sequential Keys
192+
`SequentialKeyMapper` was introduced in v2.6 and is a bit different from other key mappers because the output isn't meant for point lookups. This key mapper assigns keys in sequential order (forward or backward). Because Azure Table Storage orders rows by row key, a sequential key allows you to use Azure Table Storage as a log.
193+
194+
Coupled with TODO , you can do things like saving a historical record when mutating your main table entity.
195+
196+
Example:
197+
```csharp
198+
var pKeyMapper = new KeyMapper<Employee, int>(e => e.Id.ToString(), int.Parse, e => e.Id, id => id.ToString());
199+
200+
var rKeyMapper = new SequentialKeyMapper<Employee, int>(true);
201+
202+
var keysConverter = new CalculatedKeysConverter<Employee, int, int>(pKeyMapper, rKeyMapper);
203+
204+
tableStore = new PocoTableStore<Employee, int, int>("TestEmployee", "UseDevelopmentStorage=true", keysConverter);
205+
206+
var employee = new Employee
207+
{
208+
CompanyId = 1,
209+
Id = 242443,
210+
Name = "1",
211+
Department = new Department { Id = 22, Name = "Executive" }
212+
};
213+
214+
tableStore.Insert(employee);
215+
216+
employee.Name = "2";
217+
tableStore.Insert(employee);
218+
219+
employee.Name = "3";
220+
tableStore.Insert(employee);
221+
222+
// order will be 3, 2, 1 because we are sorting in sequential order
223+
```
224+
191225
### Further Filtering (Beyond Partition & Row Keys)
192226
New to v1.2, we now include the ability to filter on properties outside of partition and row keys. Please note that this filtering occurs outside of table storage, so please consider using at least the partition key for best results.
193227

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# TableStorage.Abstractions.POCO.SecondaryIndexes
2+
[![Build status](https://ci.appveyor.com/api/projects/status/fx9j8yc06s9ib4n9?svg=true)]
3+
4+
This project builds on top of [TableStorage.Abstractions.POCO](https://github.com/giometrix/TableStorage.Abstractions.POCO) to introduce "secondary indexes" to [Azure Table Storage](https://github.com/giometrix/TableStorage.Abstractions.POCO). Internally this library uses an [intra/inter partition (or table) secondary index pattern](https://docs.microsoft.com/en-us/azure/storage/tables/table-storage-design-patterns). When data gets mutated on your table store, the library takes care of reflecting the change in your secondary indexes.
5+
6+
## Caveats And Notes
7+
1. Indexes are managed through a library, _not_ Table Storage, thus data mutated outside of the library will not automatically be reflected in your indexes.
8+
2. Though Azure Table Storage does offer transactions within partitions, this library does not leverage this at this time.
9+
3. This library is intended for Azure Table Storage, not CosmosDB, which offers an Azure Table Storage API. CosmosDB does offer secondary indexes, so this library may not be as useful there.
10+
11+
## Examples
12+
Note that it may be useful to read about [TableStorage.Abstractions.POCO](https://github.com/giometrix/TableStorage.Abstractions.POCO) to better understand the examples below.
13+
14+
All of the examples will use the following classes:
15+
```csharp
16+
public class Employee
17+
{
18+
public int CompanyId { get; set; }
19+
public int Id { get; set; }
20+
public string Name { get; set; }
21+
public Department Department { get; set; }
22+
}
23+
24+
public class Department
25+
{
26+
public int Id { get; set; }
27+
public string Name { get; set; }
28+
}
29+
```
30+
### Instantiation
31+
Indexes are just regular `PocoTableStore`s so you instantiate them like any other `PocoTableStore`. Here we instantiate the entity store and an index store. The `PocoTableStore` named `TableStore` will store records using `CompanyId` as a partition key, and `Id` as the row key. The `PocoTableStore` named `IndexStore` will store records using `CompanyId` as the partition key, and `Name` as the row key. In this example they use different tables.
32+
33+
``` csharp
34+
TableStore = new PocoTableStore<Employee, int, int>("IXTestEmployee", "UseDevelopmentStorage=true", e => e.CompanyId, e => e.Id);
35+
36+
IndexStore = new PocoTableStore<Employee, int, string>("IXTestEmployeeNameIndex", "UseDevelopmentStorage=true", e => e.CompanyId, e => e.Name);
37+
```
38+
39+
Next we tie them together by using `AddIndex()`. Indexes must be given a name so that you can specify which index to use when querying. Hete we name our index "Name."
40+
41+
``` charp
42+
TableStore.AddIndex("Name", IndexStore);
43+
```
44+
After adding the index, mutations that happen on `TableStore` will result in mutations in `IndexStore`. For instance, if we insert a record as seen below, we can expect to find a corresponding record in `IndexStore.`
45+
46+
``` charp
47+
var employee = new Employee
48+
{
49+
Name = "Test",
50+
CompanyId = 99,
51+
Id = 99,
52+
Department = new Department { Id = 5, Name = "Test" }
53+
};
54+
TableStore.Insert(employee);
55+
```
56+
57+
### Fetching Data
58+
To fetch a single data point from the index, we use the `GetRecordByIndex` (or `GetRecordByIndexAsync`) extension method on the entity `PocoTableStore` (note that we are doing this on the main data store, not on the index, as a convenience):
59+
``` charp
60+
var e = TableStore.GetRecordByIndex("Name", 99, "Test");
61+
```
62+
63+
Sometimes it may be useful to fetch all of the records from a partition for an index, such as historical data (described later). Example:
64+
``` csharp
65+
var records = await TableStore.GetByIndexPartitionKeyAsync("Name", 99);
66+
```
67+
68+
One use of this pattern can be to store the current entity in the main entity store, and to keep historical data in a separate table. Here is an example of this pattern:
69+
``` csharp
70+
var pKeyMapper = new KeyMapper<Employee, int>(e => e.Id.ToString(), int.Parse, e => e.Id, id => id.ToString());
71+
72+
var rKeyMapper = new SequentialKeyMapper<Employee, int>(true);
73+
74+
var keysConverter = new CalculatedKeysConverter<Employee, int, int>(pKeyMapper, rKeyMapper);
75+
76+
var logStore =
77+
new PocoTableStore<Employee, int, int>("IXLogIndex", "UseDevelopmentStorage=true", keysConverter);
78+
79+
TableStore.AddIndex("Log", logStore);
80+
81+
```
82+
In the example above we create an index called "Log", which will use `Id` as the partition key and a decreasing sequence number for row key (so that the most recent record is always on top).
83+
84+
If we want to fetch the history for employee 99, we do the following:
85+
``` csharp
86+
var records = TableStore.GetByPartitionKey(99);
87+
```
88+
89+
### Removing An Index
90+
To remove an index without deleting data, use the `Reindex()` or `ReindexAsync()` extension method.
91+
92+
### Dropping An Index
93+
To remove _and_ drop an index without deleting data, use the `DropIndex()` or `DropIndexAsync()` extension method. Deleting the original table will also drop all indexes on that table.
94+
95+
### Seeding Or Reindexing
96+
If you are adding an index to an existing table that already has data, or if for some reason data gets out of sync, you can use the `Reindex()` extension method, shown below. Note that this method is not yet optimized (for instance no batching is currently used). On my machine home internet connection, and data size, it took 22 minutes to index 1 million rows.
97+
98+
``` charp
99+
await TableStore.ReindexAsync("Name", maxDegreeOfParallelism: 20, recordsIndexedCallback: i=>count = i);
100+
```
101+
Call backs are available to get status updates and errors.

tests/TableStorage.Abstractions.POCO.Tests/PocoTableStoreTests.cs

Lines changed: 2 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1894,26 +1894,11 @@ public void insert_record_with_sequential_row_key_sorts_correctly()
18941894

18951895
Clock.AsMockClock().Adjust(1000);
18961896

1897-
employee = new Employee
1898-
{
1899-
CompanyId = 1,
1900-
Id = 242443,
1901-
Name = "2",
1902-
Department = new Department { Id = 22, Name = "Executive" }
1903-
};
1904-
1897+
employee.Name = "2";
19051898
tableStore.Insert(employee);
19061899
Clock.AsMockClock().Adjust(1000);
19071900

1908-
employee = new Employee
1909-
{
1910-
CompanyId = 1,
1911-
Id = 242443,
1912-
Name = "3",
1913-
Department = new Department { Id = 22, Name = "Executive" }
1914-
};
1915-
1916-
1901+
employee.Name = "3";
19171902
tableStore.Insert(employee);
19181903
Clock.AsMockClock().Adjust(1000);
19191904

0 commit comments

Comments
 (0)