Skip to content

Commit c36fab2

Browse files
authored
Merge pull request #26 from ExpediaDotCom/fix_rate_limit_issue6
Fix "Rate exceeded" issues for large clusters
2 parents 5fbab05 + 3fea419 commit c36fab2

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+4741
-971
lines changed

Dockerfile

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,16 @@
11
# https://hub.docker.com/r/library/node/
22

3-
FROM node:4.1
3+
FROM node:9.11.1-alpine
4+
5+
# >> FIX:
6+
# Fixes error Ubuntu: "gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable"
7+
# REF: https://gist.github.com/vidhill/0a85dc1848feee4171944dc4d7757895
8+
# REF: https://github.com/nodejs/node-gyp/issues/1105
9+
10+
# build base includes g++ and gcc and Make
11+
RUN apk update && apk add python build-base
12+
13+
# << END FIX
414

515
# Bundle app source
616
COPY . /

README.md

Lines changed: 52 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# c3vis - Cloud Container Cluster Visualizer
22

3-
Helps visualize the resource utilisation of Amazon ECS clusters.
3+
Helps visualize the resource reservation of Amazon ECS clusters.
44

55
Deploying software as “containers” promises to solve many problems with regards to interoperability of environments, speed to deploy, and cost reduction.
66
But understanding where our software lives now becomes more difficult both for development and operations teams.
@@ -16,118 +16,94 @@ Each unique Task Definition is represented as a different color, with the legend
1616
Each Task will contain one or more containers, the task box shows accumulated reserved memory or CPU for all containers in the Task. ECS Services are not currently represented.
1717

1818

19-
## Configure
19+
## Configuration
2020

21-
Displaying live ECS data requires server-side AWS credentials.
21+
See [CONFIGURATION](docs/CONFIGURATION.md) for details on server-side configurable options that affect cache entry TTL and AWS API call throttling.
2222

23-
### Region
23+
## Configuring AWS SDK
2424

25-
Before running, add a file named ```aws_config.json``` to this project root directory. At a minimum set the default region:
25+
See [AWS_SDK_CONFIGURATION](docs/AWS_SDK_CONFIGURATION.md) for instructions
26+
on configuring the AWS SDK for server-side AWS connectivity.
2627

27-
```
28-
{
29-
"region": "<default-region>"
30-
}
31-
```
28+
## Requirements
29+
30+
Node >= 0.12
3231

33-
Alternatively, set the environment variable "AWS_REGION" before starting the server.
32+
## Building and Running
3433

35-
### Credentials
34+
The c3vis server is based on ExpressJS. The client predominantly uses D3.js,
35+
jQuery and Bootstrap.
3636

37-
AWS credentials properties "accessKeyId" and "secretAccessKey" can be added to the aws_config.json file as per https://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html.
37+
Run the following to build and run the server ("package.json" contains instructions to pre-install required node modules):
3838

39-
Otherwise, the credentials will be loaded from the Shared Credentials File or Environment Variables or IAM roles if deployed to an AWS instance.
39+
```
40+
npm start
41+
```
4042

41-
When using an IAM role, ensure the role has the following access:
43+
Now browse to the app at `http://localhost:3000`.
4244

43-
* ecs:listContainerInstances
44-
* ecs:describeContainerInstances
45-
* ecs:listTasks
46-
* ecs:describeTasks
47-
* ecs:describeTaskDefinition
48-
* ecs:listClusters
49-
* ec2:describeInstance
45+
## Testing
5046

51-
Sample IAM Inline Policy:
47+
To run the server-side unit test suite with mocha and chai:
48+
5249
```
53-
{
54-
"Version": "2012-10-17",
55-
"Statement": [
56-
{
57-
"Effect": "Allow",
58-
"Action": [
59-
"ecs:listContainerInstances",
60-
"ecs:describeContainerInstances",
61-
"ecs:listTasks",
62-
"ecs:describeTasks",
63-
"ecs:describeTaskDefinition",
64-
"ecs:listClusters",
65-
"ec2:describeInstances"
66-
],
67-
"Resource": [
68-
"*"
69-
]
70-
}
71-
]
72-
}
50+
npm run test
7351
```
7452

75-
**WARNING:** c3vis makes ECS data from the above API calls (including environment variables in task definitions) available to clients/browsers.
76-
Ensure the c3vis server is available only to users that should have access to this information.
53+
## Usage
7754

55+
### Approach
7856

79-
## Requirements
57+
When a client browser first connects to the c3vis server, the Cluster dropdown will be populated with ECS cluster names for the configured region.
8058

81-
Node >= 0.12
59+
Select from the dropdown to view the resources allocated to that cluster. If no cluster names appear in the dropdown, check the server logs and ensure the correct region is configured (see below).
8260

83-
## Building and Running
61+
The list of clusters and the user's current selection are stored in cookies. Use the ```[refresh list]``` dropdown entry to refresh the list of clusters.
8462

85-
Server is based on ExpressJS. Client uses D3.js.
63+
The Y axis shows total memory or CPU available for the instances. Memory is the default resource type represented. Use the "resourceType" query parameter to toggle between "memory" and "cpu". E.g. ```localhost:3000/?resourceType=cpu```
8664

87-
Run the following to build and run the server ("package.json" contains instructions to pre-install required node modules):
65+
The X axis displays the Private IP Address for each EC2 instance. Right-clicking the IP address shows the context menu with links to browse the instance in the ECS and EC2 consoles.
8866

89-
```
90-
npm start
91-
```
67+
### AWS API Call Throttling
9268

93-
This will run ```npm install``` and ```node --harmony ./bin/www```
94-
(NOTE: ```"--harmony"``` is required for ES6 functionality such as Array.find())
69+
In order to prevent AWS API Rate limiting issues for large clusters, the server:
9570

96-
Now browse to the app at `http://localhost:3000`.
71+
* Introduces a delay between API calls (configurable via `aws.apiDelay` setting)
72+
* Limits the number of items retrieved per page in `list` and `describe` API calls (configurable via `aws.*PageSize`)
73+
* Limits the number of asynchronous API calls it makes at a time (configurable via `aws.maxSimultaneous*Calls`)
9774

98-
### Usage
75+
You can increase or decrease each of these settings to suit each environment c3vis is deployed to.
9976

100-
When a client browser first connects to the c3vis server the Cluster dropdown will be populated with ECS cluster names for the configured region.
77+
### Short Polling, Server-Side Caching and Fetch Status
10178

102-
Select from the dropdown to view the resources allocated to that cluster. If no cluster names appear in the dropdown, check the server logs and ensure the correct region is configured (see below).
79+
For each cluster requested, the server caches cluster data in-memory while the client polls the server until the cache is populated.
10380

104-
The list of clusters and the user's current selection are stored in cookies. Use the ```[refresh list]``` dropdown entry to refresh the list of clusters.
81+
For an explanation on how the client polls the server for cluster data and the applicable fetch statuses, see [SHORT_POLLING_FETCH_STATUS](docs/SHORT_POLLING_FETCH_STATUS.md).
10582

106-
The Y axis shows total memory or CPU available for the instances. Memory is the default resource type represented. Use the "resourceType" query parameter to toggle between "memory" and "cpu". E.g. ```localhost:3000/?resourceType=cpu```
10783

108-
The X axis displays the Private IP Address for each EC2 instance. Right-clicking the IP address shows the context menu with links to browse the instance in the ECS and EC2 consoles.
84+
## Debugging
10985

11086
### Sample Clusters for Testing
11187

11288
From the browser, use a ```"?static=true"``` query parameter to have the server return static test data. Useful for testing when server is unable to connect to AWS.
11389

11490
Browse to `http://localhost:3000/?static=true`.
11591

116-
### Debugging
92+
### Server Debug Logging
11793

118-
Add the following line to server-side Javascript code to add a breakpoint:
94+
To see all debug log entries:
11995

12096
```
121-
debugger;
97+
DEBUG=* npm start
12298
```
12399

124-
then run the debugger with:
100+
To see just API debug log entries:
125101

126102
```
127-
node debug --harmony ./bin/www
103+
DEBUG=api npm start
128104
```
129105

130-
### Running with Docker
106+
## Running with Docker
131107

132108
Build and tag the image:
133109

@@ -141,8 +117,14 @@ Run the container: (can remove ```AWS_ACCESS_KEY_ID``` and ```AWS_SECRET_ACCESS_
141117
docker run -e "AWS_REGION=<region>" -e "AWS_ACCESS_KEY_ID=<accesskey>" -e "AWS_SECRET_ACCESS_KEY=<secretkey>" -p 3000:3000 c3vis
142118
```
143119

120+
E.g. To run with `prod` target environment configuration:
121+
122+
```
123+
docker run -e "TARGET_ENV=prod" -p 3000:3000 c3vis
124+
```
125+
144126

145-
Browse to `<docker localhost>:3000` (e.g. [http://192.168.99.100:3000](http://192.168.99.100:3000))
127+
Browse to [http://localhost:3000](http://localhost:3000)
146128

147129

148130
# Credits
@@ -154,4 +136,4 @@ Created by [Matt Callanan](https://github.com/mattcallanan) with contributions f
154136

155137
This project is available under the [Apache 2.0 License](http://www.apache.org/licenses/LICENSE-2.0.html).
156138

157-
Copyright 2015 Expedia Inc.
139+
Copyright 2018 Expedia Inc.

TODO.md

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,24 @@
1+
## UI
12
* Menubar with:
23
* Selectable Region
34
* Selectable AWS account – requires ability for mutliple config files server side
45
* Show clusters as tabbed view with one cluster per tab
5-
* Add toggle button to switch between memory vs CPU resourceType
66
* Show an exploded view of task with more details when hovering over tasks:
77
* Show containers within tasks
88
* Show memory breakdown across containers
99
* Sliding timebar to see historical data for comparison (like google street view)
1010
* Show container actual memory utilisation vs reserved memory utilisation
11+
* Provide access to more troubleshooting information (such as docker logs, ECS logs)
12+
* Add footer with fetched/expiry timestamp, #instances/services/tasks, Average CPU/Memory Reservation
13+
14+
## Server
1115
* Write a plugin system that lets adopters plugin their own statistics from favourite monitoring tool
1216
* Pluggable backend system that could support other public or private cloud providers
13-
* Provide access to more troubleshooting information (such as docker logs, ECS logs)
14-
* Cache responses server-side to reduce AWS API calls
15-
* Make the data transfer between client and server more efficient - Separate requests for task and instance data and populate graph asynchronously
17+
* Return instances with FETCHED_INSTANCES FetchStatus to allow client to draw instances outline until tasks retrieved asynchronously
1618
* Arrow functions: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions
17-
* Testing
19+
20+
21+
22+
## Testing
1823
* Capture ECS JSON responses for testing and replay with mock AWS ECS server
1924
* https://fbflex.wordpress.com/2013/11/18/mocking-out-amazon-aws-sdk-with-the-betamax-recording-proxy-for-testing/

bin/www

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,13 @@
77
var app = require('../app');
88
var debug = require('debug')('c3vis:server');
99
var http = require('http');
10+
var config = require('../config/config');
1011

1112
/**
1213
* Get port from environment and store in Express.
1314
*/
1415

15-
var port = normalizePort(process.env.PORT || '3000');
16+
var port = normalizePort(config.port);
1617
app.set('port', port);
1718

1819
/**

config/config.js

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
const lodash = require('lodash');
2+
const TARGET_ENV = process.env.TARGET_ENV || 'dev';
3+
4+
function _loadDefaultConfig() {
5+
return require('./defaults.js');
6+
}
7+
8+
function _loadOverrideConfig(targetEnvironment) {
9+
try {
10+
// Extend configuration with environment-specific configuration
11+
console.debug(`Overriding default configuration with '${targetEnvironment}' environment configuration from ${_overrideConfigFilename(targetEnvironment)} (TARGET_ENV=${process.env.TARGET_ENV}, NODE_ENV=${process.env.NODE_ENV})`);
12+
return require(_overrideConfigFilename(targetEnvironment));
13+
} catch (err) {
14+
console.error(`ERROR: Could not load configuration file for target environment '${targetEnvironment}'. Skipping. (${err})`);
15+
return {}
16+
}
17+
}
18+
19+
function _overrideConfigFilename(targetEnvironment) {
20+
return `./env/${targetEnvironment}.js`;
21+
}
22+
23+
module.exports = lodash.merge(_loadDefaultConfig(), _loadOverrideConfig(TARGET_ENV));

config/defaults.js

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
module.exports = {
2+
environmentName: undefined,
3+
port: process.env.PORT || 3000,
4+
clusterStateCacheTtl: 30 * 60 * 1000, // Invalidate clusters in cache after 30 minutes
5+
aws: {
6+
configFile: './aws_config.json',
7+
apiDelay: 100, // milliseconds to pause between AWS API calls to prevent API rate limiting
8+
listInstancesPageSize: 100, // max 100
9+
describeInstancesPageSize: 100, // max 100
10+
listTasksPageSize: 100, // max 100
11+
describeTasksPageSize: 100, // max 100
12+
maxSimultaneousDescribeTasksCalls: 2,
13+
maxSimultaneousDescribeTaskDefinitionCalls: 1,
14+
}
15+
};

config/env/dev.js

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
module.exports = {
2+
// Add dev environment config overrides here and enable at startup with TARGET_ENV=dev environment variable
3+
environmentName: "Development"
4+
};

config/env/prod.js

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
module.exports = {
2+
// Add prod environment config overrides here and enable at startup with TARGET_ENV=prod environment variable
3+
environmentName: "Production"
4+
};

config/env/test.js

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
module.exports = {
2+
// Add test environment config overrides here and enable at startup with TARGET_ENV=test environment variable
3+
environmentName: "Test"
4+
};

docs/AWS_SDK_CONFIGURATION.md

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Configuring AWS SDK
2+
3+
The c3vis server uses the AWS JavaScript SDK to connect to AWS APIs.
4+
5+
As per [Configuring the SDK for JavaScript](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/configuring-the-jssdk.html), the AWS JavaScript SDK will get its configuration from the server's environment.
6+
7+
## Provide Explicit AWS SDK Configuration with `aws_config.json` Configuration File
8+
9+
AWS SDK configuration can be overridden by providing an `aws_config.json` file (this file location is overridable with `aws.configFile` option, see [CONFIGURATION.md](CONFIGURATION.md)).
10+
11+
E.g. to set the region used by c3vis server to `us-east-1`, create an `aws_config.json` file in the root directory with the following:
12+
13+
```
14+
{
15+
"region": "us-east-1"
16+
}
17+
```
18+
19+
The contents of this file override all other sources of AWS SDK configuration.
20+
The settings are applied to the AWS Global Configuration Object using `AWS.config.update()` as per [Using the Global Configuration Object](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/global-config-object.html)
21+
22+
## AWS Region
23+
24+
As per above section, AWS Region can be provided in local `aws_config.json` file.
25+
26+
Otherwise the Region will be configured as per [Setting the AWS Region](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-region.html).
27+
28+
## AWS Credentials
29+
30+
If using `aws_config.json` file as per above section, you can add AWS credentials properties `accessKeyId` and `secretAccessKey` to the `aws_config.json`
31+
See [https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-json-file.html](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-json-file.html).
32+
33+
*NOTE: Storing credentials in plaintext file is not recommended, especially if there is a risk this file could be committed to version control.*
34+
35+
Otherwise, the credentials will be loaded as per priority listed [here](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html).
36+
37+
## IAM Role Permissions
38+
39+
### EC2 IAM Role Permissions
40+
41+
When running c3vis on EC2 instances using an IAM role, ensure the role has the
42+
following permissions:
43+
44+
* `ecs:listContainerInstances`
45+
* `ecs:describeContainerInstances`
46+
* `ecs:listTasks`
47+
* `ecs:describeTasks`
48+
* `ecs:describeTaskDefinition`
49+
* `ecs:listClusters`
50+
* `ec2:describeInstance`
51+
52+
Sample IAM Inline Policy:
53+
```
54+
{
55+
"Version": "2012-10-17",
56+
"Statement": [
57+
{
58+
"Effect": "Allow",
59+
"Action": [
60+
"ecs:listContainerInstances",
61+
"ecs:describeContainerInstances",
62+
"ecs:listTasks",
63+
"ecs:describeTasks",
64+
"ecs:describeTaskDefinition",
65+
"ecs:listClusters",
66+
"ec2:describeInstances"
67+
],
68+
"Resource": [
69+
"*"
70+
]
71+
}
72+
]
73+
}
74+
```
75+
76+
### ECS IAM Task Role
77+
78+
When running c3vis on an ECS cluster, you can use an ECS Task IAM Role, which
79+
can be created using the process documented [here](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html#create_task_iam_policy_and_role).
80+
Ensure the IAM Policy has the permissions listed above.
81+
82+
## Security Warning
83+
84+
**WARNING:** c3vis makes ECS data from the above API calls (including environment variables in task definitions) available to clients/browsers.
85+
Ensure the c3vis server is available only to users that should have access to this information.

0 commit comments

Comments
 (0)