|
| 1 | +# TableStorage.Abstractions.POCO.SecondaryIndexes |
| 2 | +[] |
| 3 | + |
| 4 | +This project builds on top of [TableStorage.Abstractions.POCO](https://github.com/giometrix/TableStorage.Abstractions.POCO) to introduce "secondary indexes" to [Azure Table Storage](https://github.com/giometrix/TableStorage.Abstractions.POCO). Internally this library uses an [intra/inter partition (or table) secondary index pattern](https://docs.microsoft.com/en-us/azure/storage/tables/table-storage-design-patterns). When data gets mutated on your table store, the library takes care of reflecting the change in your secondary indexes. |
| 5 | + |
| 6 | +## Caveats And Notes |
| 7 | +1. Indexes are managed through a library, _not_ Table Storage, thus data mutated outside of the library will not automatically be reflected in your indexes. |
| 8 | +2. Though Azure Table Storage does offer transactions within partitions, this library does not leverage this at this time. |
| 9 | +3. This library is intended for Azure Table Storage, not CosmosDB, which offers an Azure Table Storage API. CosmosDB does offer secondary indexes, so this library may not be as useful there. |
| 10 | + |
| 11 | +## Examples |
| 12 | +Note that it may be useful to read about [TableStorage.Abstractions.POCO](https://github.com/giometrix/TableStorage.Abstractions.POCO) to better understand the examples below. |
| 13 | + |
| 14 | +All of the examples will use the following classes: |
| 15 | +```csharp |
| 16 | +public class Employee |
| 17 | +{ |
| 18 | + public int CompanyId { get; set; } |
| 19 | + public int Id { get; set; } |
| 20 | + public string Name { get; set; } |
| 21 | + public Department Department { get; set; } |
| 22 | +} |
| 23 | + |
| 24 | +public class Department |
| 25 | +{ |
| 26 | + public int Id { get; set; } |
| 27 | + public string Name { get; set; } |
| 28 | +} |
| 29 | +``` |
| 30 | +### Instantiation |
| 31 | +Indexes are just regular `PocoTableStore`s so you instantiate them like any other `PocoTableStore`. Here we instantiate the entity store and an index store. The `PocoTableStore` named `TableStore` will store records using `CompanyId` as a partition key, and `Id` as the row key. The `PocoTableStore` named `IndexStore` will store records using `CompanyId` as the partition key, and `Name` as the row key. In this example they use different tables. |
| 32 | + |
| 33 | +``` csharp |
| 34 | +TableStore = new PocoTableStore<Employee, int, int>("IXTestEmployee", "UseDevelopmentStorage=true", e => e.CompanyId, e => e.Id); |
| 35 | + |
| 36 | +IndexStore = new PocoTableStore<Employee, int, string>("IXTestEmployeeNameIndex", "UseDevelopmentStorage=true", e => e.CompanyId, e => e.Name); |
| 37 | +``` |
| 38 | + |
| 39 | +Next we tie them together by using `AddIndex()`. Indexes must be given a name so that you can specify which index to use when querying. Hete we name our index "Name." |
| 40 | + |
| 41 | +``` charp |
| 42 | +TableStore.AddIndex("Name", IndexStore); |
| 43 | +``` |
| 44 | +After adding the index, mutations that happen on `TableStore` will result in mutations in `IndexStore`. For instance, if we insert a record as seen below, we can expect to find a corresponding record in `IndexStore.` |
| 45 | + |
| 46 | +``` charp |
| 47 | +var employee = new Employee |
| 48 | +{ |
| 49 | + Name = "Test", |
| 50 | + CompanyId = 99, |
| 51 | + Id = 99, |
| 52 | + Department = new Department { Id = 5, Name = "Test" } |
| 53 | +}; |
| 54 | +TableStore.Insert(employee); |
| 55 | +``` |
| 56 | + |
| 57 | +### Fetching Data |
| 58 | +To fetch a single data point from the index, we use the `GetRecordByIndex` (or `GetRecordByIndexAsync`) extension method on the entity `PocoTableStore` (note that we are doing this on the main data store, not on the index, as a convenience): |
| 59 | +``` charp |
| 60 | +var e = TableStore.GetRecordByIndex("Name", 99, "Test"); |
| 61 | +``` |
| 62 | + |
| 63 | +Sometimes it may be useful to fetch all of the records from a partition for an index, such as historical data (described later). Example: |
| 64 | +``` csharp |
| 65 | +var records = await TableStore.GetByIndexPartitionKeyAsync("Name", 99); |
| 66 | +``` |
| 67 | + |
| 68 | +One use of this pattern can be to store the current entity in the main entity store, and to keep historical data in a separate table. Here is an example of this pattern: |
| 69 | +``` csharp |
| 70 | +var pKeyMapper = new KeyMapper<Employee, int>(e => e.Id.ToString(), int.Parse, e => e.Id, id => id.ToString()); |
| 71 | + |
| 72 | +var rKeyMapper = new SequentialKeyMapper<Employee, int>(true); |
| 73 | + |
| 74 | +var keysConverter = new CalculatedKeysConverter<Employee, int, int>(pKeyMapper, rKeyMapper); |
| 75 | + |
| 76 | +var logStore = |
| 77 | + new PocoTableStore<Employee, int, int>("IXLogIndex", "UseDevelopmentStorage=true", keysConverter); |
| 78 | + |
| 79 | +TableStore.AddIndex("Log", logStore); |
| 80 | + |
| 81 | +``` |
| 82 | +In the example above we create an index called "Log", which will use `Id` as the partition key and a decreasing sequence number for row key (so that the most recent record is always on top). |
| 83 | + |
| 84 | +If we want to fetch the history for employee 99, we do the following: |
| 85 | +``` csharp |
| 86 | +var records = TableStore.GetByPartitionKey(99); |
| 87 | +``` |
| 88 | + |
| 89 | +### Removing An Index |
| 90 | +To remove an index without deleting data, use the `Reindex()` or `ReindexAsync()` extension method. |
| 91 | + |
| 92 | +### Dropping An Index |
| 93 | +To remove _and_ drop an index without deleting data, use the `DropIndex()` or `DropIndexAsync()` extension method. Deleting the original table will also drop all indexes on that table. |
| 94 | + |
| 95 | +### Seeding Or Reindexing |
| 96 | +If you are adding an index to an existing table that already has data, or if for some reason data gets out of sync, you can use the `Reindex()` extension method, shown below. Note that this method is not yet optimized (for instance no batching is currently used). On my machine home internet connection, and data size, it took 22 minutes to index 1 million rows. |
| 97 | + |
| 98 | +``` charp |
| 99 | +await TableStore.ReindexAsync("Name", maxDegreeOfParallelism: 20, recordsIndexedCallback: i=>count = i); |
| 100 | +``` |
| 101 | +Call backs are available to get status updates and errors. |
0 commit comments