|
1 |
| -dynamodb-lambda-autoscale |
2 |
| -========================= |
3 |
| -Autoscale AWS DynamoDB using an AWS Lambda function |
| 1 | +# dynamodb-lambda-autoscale |
| 2 | +**Autoscale AWS DynamoDB using an AWS Lambda function |
4 | 3 |
|
5 |
| -DynamoDB does not currently have an official way of autoscaling the capacity throughput for reads and writes. |
| 4 | ++ 5 minute setup process |
| 5 | ++ Serverless design |
| 6 | ++ Flexible code over configuration style |
| 7 | ++ Autoscale table and global secondary indexes |
| 8 | ++ Autoscale multiple tables |
| 9 | ++ Optimised performance using concurrent queries |
| 10 | ++ Statistics via 'measured' |
| 11 | ++ AWS credential configuration via 'dotenv' |
| 12 | ++ Optimised lambda package via 'webpack' |
| 13 | ++ ES7 code |
6 | 14 |
|
7 |
| -Features |
8 |
| --------- |
9 |
| - |
10 |
| -- 5 minute setup process |
11 |
| -- Serverless design |
12 |
| -- Flexible code over configuration style |
13 |
| -- Autoscale table and global secondary indexes |
14 |
| -- Autoscale multiple tables |
15 |
| -- Optimised performance using concurrent queries |
16 |
| -- Statistics via 'measured' |
17 |
| -- AWS credential configuration via 'dotenv' |
18 |
| -- Optimised lambda package via 'webpack' |
19 |
| -- ES7 code |
20 |
| - |
21 |
| -Setup |
22 |
| ------ |
| 15 | +## Getting started |
23 | 16 |
|
24 | 17 | 1. Build and package the code
|
25 | 18 | 1. Fork the repo
|
|
34 | 27 | 4. Run 'npm run build'
|
35 | 28 | 5. Verify this has created a 'dist.zip' file
|
36 | 29 | 6. Optionally, run a local test by running 'npm run start'
|
| 30 | + |
| 31 | +## Running on AWS Lambda |
| 32 | + |
| 33 | +1. Follow the steps in 'Running locally' |
37 | 34 | 2. Create an AWS Policy and Role
|
38 | 35 | 1. Create a policy called 'DynamoDBLambdaAutoscale'
|
39 | 36 | 2. Use the following content to give access to dynamoDB, cloudwatch and lambda logging
|
@@ -70,27 +67,79 @@ Setup
|
70 | 67 | 8. Set the Timeout to 5 seconds (higher possibly depending on the amount of tables you have)
|
71 | 68 | 9. Once the function is created, attach a 'scheduled event' event source and make it run every minute
|
72 | 69 |
|
73 |
| -License |
74 |
| -------- |
| 70 | +## Configuration |
| 71 | + |
| 72 | +The default setup of the configuration is to apply autoscaling to all tables, |
| 73 | +allowing for a no touch quick setup. |
| 74 | + |
| 75 | +dynamodb-lambda-autoscale takes a different approach to autoscaling |
| 76 | +configuration compared to other community projects. Rather than making changes |
| 77 | +to a config file dynamodb-lambda-autoscale provides a function 'getTableUpdate' |
| 78 | +which must be implemented. |
| 79 | + |
| 80 | + |
| 81 | +```javascript |
| 82 | +{ |
| 83 | + connection: { |
| 84 | + dynamoDB: { apiVersion: '2012-08-10', region: 'us-east-1' }, |
| 85 | + cloudWatch: { apiVersion: '2010-08-01', region: 'us-east-1' } |
| 86 | + }, |
| 87 | + getTableUpdate: (description, consumedCapacityDescription) => { |
| 88 | + // The 'description' parameter is JSON in the following format |
| 89 | + <http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DescribeTable.html#API_DescribeTable_ResponseSyntax> |
| 90 | + |
| 91 | + // The 'consumedCapacityDescription' follows the same format but details |
| 92 | + // the consumed capacity found via the cloudwatch api |
| 93 | + |
| 94 | + // The return type is either null for no update or JSON in the |
| 95 | + // following format |
| 96 | + <http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateTable.html#API_UpdateTable_ResponseSyntax> |
| 97 | + } |
| 98 | +}; |
| 99 | +``` |
| 100 | + |
| 101 | +The function is given the information such as the table name, current table |
| 102 | +provisioned throughput and the consumed throughput for the past minute. |
| 103 | +Table updates will only be sent to AWS if the values are different for the |
| 104 | +current, this approach follows the popular code first pattern used in React. |
| 105 | + |
| 106 | +In most cases the default 'ConfigurableProvisioner' supplied will provide |
| 107 | +enough functionality out of box such that additional coding is not required. |
| 108 | +The default provisioner provides the following features. |
| 109 | + |
| 110 | +- Separate 'Read' and 'Write' capacity adjustment |
| 111 | +- Separate 'Increment' and 'Decrement' capacity adjustment |
| 112 | +- Read/Write provisioned capacity increased |
| 113 | + - if capacity utilisation > 90% |
| 114 | + - by either 100% or 3 units, which ever is the greater |
| 115 | + - with hard min/max limits of 1 and 10 respectively |
| 116 | +- Read/Write provisioned capacity decreased |
| 117 | + - if capacity utilisation < 30% AND |
| 118 | + - if at least 60 minutes have passed since the last increment AND |
| 119 | + - if at least 60 minutes have passed since the last decrement AND |
| 120 | + - if the adjustment will be at least 3 units AND |
| 121 | + - if we are allowed to utilise 1 of our 4 AWS enforced decrements |
| 122 | + - to the consumed throughput value |
| 123 | + - with a hard min limit of 1 |
75 | 124 |
|
76 |
| -The MIT License (MIT) |
| 125 | +As AWS only allows 4 table decrements in a calendar day we have an intelligent |
| 126 | +algorithm which segments the remaining time to midnight by the amount of |
| 127 | +decrements we have left. This logic allows us to utilise each 4 decrements |
| 128 | +efficiently. The increments are unlimited so the algorithm follows a unique |
| 129 | +'sawtooth' profile, dropping the provisioned throughput all the way down to |
| 130 | +the consumed throughput rather than gradually. Please see |
| 131 | +[RateLimitedDecrement.js](./src/RateLimitedDecrement.js) for full |
| 132 | +implementation. |
77 | 133 |
|
78 |
| -Copyright (c) 2016 for dynamodb-lambda-autoscale |
| 134 | +## Dependencies |
79 | 135 |
|
80 |
| -Permission is hereby granted, free of charge, to any person obtaining a copy |
81 |
| -of this software and associated documentation files (the "Software"), to deal |
82 |
| -in the Software without restriction, including without limitation the rights |
83 |
| -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell |
84 |
| -copies of the Software, and to permit persons to whom the Software is |
85 |
| -furnished to do so, subject to the following conditions: |
| 136 | +dynamodb-lambda-autoscale has the following main dependencies: |
| 137 | ++ aws-sdk - Access to AWS services |
| 138 | ++ winston - Logging |
| 139 | ++ dotenv - Environment variable configuration useful for lambda |
| 140 | ++ measured - Statistics gathering |
86 | 141 |
|
87 |
| -The above copyright notice and this permission notice shall be included in all |
88 |
| -copies or substantial portions of the Software. |
| 142 | +## Licensing |
89 | 143 |
|
90 |
| -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR |
91 |
| -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, |
92 |
| -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE |
93 |
| -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER |
94 |
| -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, |
95 |
| -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE |
96 |
| -SOFTWARE. |
| 144 | +The source code is licensed under the MIT license found in the |
| 145 | +[LICENSE](LICENSE) file in the root directory of this source tree. |
0 commit comments