Skip to content

Commit c2ad137

Browse files
authored
Add documentation for environment variables (#177)
* Add documentation for BATCH_RATE_LIMIT_MAX and BATCH_RATE_LIMIT_WINDOW * Document environment variables for logging node heap stats
1 parent d7694a3 commit c2ad137

File tree

1 file changed

+38
-0
lines changed

1 file changed

+38
-0
lines changed

service_config/service.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@
77
- [AUTH\_CURATION\_TEAM](#auth_curation_team)
88
- [AUTH\_GITHUB\_CLIENT](#auth_github_client)
99
- [AUTH\_HARVEST\_TEAM](#auth_harvest_team)
10+
- [BATCH_RATE_LIMIT_MAX](#batch_rate_limit_max)
11+
- [BATCH_RATE_LIMIT_WINDOW](#batch_rate_limit_window)
1012
- [CACHING\_PROVIDER](#caching_provider)
1113
- [CACHING\_REDIS\_SERVICE](#caching_redis_service)
1214
- [CRAWLER\_API\_AUTH\_TOKEN\*\*](#crawler_api_auth_token)
@@ -32,6 +34,8 @@
3234
- [HARVEST\_QUEUE\_PROVIDER](#harvest_queue_provider)
3335
- [HARVEST\_QUEUE\_PREFIX\*\*](#harvest_queue_prefix)
3436
- [HARVESTER\_PROVIDER](#harvester_provider)
37+
- [LOG\_NODE\_HEAPSTATS](#log_node_heapstats)
38+
- [LOG\_NODE\_HEAPSTATS\_INTERVAL\_MS](#log_node_heapstats_interval_ms)
3539
- [MULTIVERSION\_CURATION\_FF](#multiversion_curation_ff)
3640
- [NODE\_ENV](#node_env)
3741
- [RATE\_LIMIT\_MAX](#rate_limit_max)
@@ -94,9 +98,13 @@ The environmental variables for the clearlydefined-api-dev App Service include:
9498
* HARVEST_QUEUE_PREFIX
9599
* HARVEST_QUEUE_PROVIDER
96100
* HARVESTER_PROVIDER
101+
* LOG_NODE_HEAPSTATS
102+
* LOG_NODE_HEAPSTATS_INTERVAL_MS
97103
* NODE_ENV
98104
* RATE_LIMIT_MAX
99105
* RATE_LIMIT_WINDOW
106+
* BATCH_RATE_LIMIT_MAX
107+
* BATCH_RATE_LIMIT_WINDOW
100108
* SEARCH_AZURE_API_KEY
101109
* SEARCH_AZURE_SERVICE
102110
* SEARCH_PROVIDER
@@ -312,6 +320,28 @@ Important to ensure that any other instances of production crawlers that use the
312320

313321
This indicates what type of service we use for harvesting, in this case it's **crawlerQueue**, which corresponds with the [crawlerQueue harvest provider](https://github.com/clearlydefined/service/blob/master/providers/harvest/crawlerQueue.js)
314322

323+
### LOG_NODE_HEAPSTATS
324+
325+
This is an optional flag to `enable` logging of Node's `v8` module's memory usage data using the `getHeapSpaceStatistics` and `getHeapStatistics()` functions.
326+
327+
Value is either `true` or `false`
328+
> Note: if this env var is not present, it equates to `false`
329+
> example:
330+
> `LOG_NODE_HEAPSTATS` = `true`
331+
332+
- [Node.js v8 engine docs - getHeapSpaceStatistics()](https://nodejs.org/docs/v22.12.0/api/v8.html#v8getheapspacestatistics)
333+
334+
- [Node.js v8 engine docs - getHeapStatistics()](https://nodejs.org/docs/v22.12.0/api/v8.html#v8getheapstatistics)
335+
336+
### LOG_NODE_HEAPSTATS_INTERVAL_MS
337+
338+
This is an optional environment variable that sets the interval to log heap statistics (When enabled).
339+
340+
Value is a number in `ms` (`milliseconds`).
341+
> NOTE: The default value is `30000` ms (`30` seconds)
342+
> example:
343+
> `LOG_NODE_HEAPSTATS_INTERVAL_MS` = `10000`
344+
315345
### MULTIVERSION_CURATION_FF
316346

317347
This is a feature flag that indicates whether the [Multi-version curation feature](https://github.com/clearlydefined/service/pull/810) is active.
@@ -334,6 +364,14 @@ When we [use this value in the code](https://github.com/clearlydefined/service/b
334364

335365
So, one IP address can only call the ClearlyDefined API 500 times every 300 seconds.
336366

367+
### BATCH_RATE_LIMIT_MAX
368+
369+
Defines the maximum number of requests allowed from a single IP to the batch endpoints within the batch rate limit window.
370+
371+
### BATCH_RATE_LIMIT_WINDOW
372+
373+
Defines the time window (in seconds) used to apply `BATCH_RATE_LIMIT_MAX` for batch endpoints. This value is multiplied by 1000 internally to convert to milliseconds (same as `RATE_LIMIT_WINDOW`)
374+
337375
### SEARCH_PROVIDER
338376

339377
We use [Azure Cognitive Search](https://docs.microsoft.com/en-us/azure/search/search-what-is-azure-search) to power ClearlyDefined's Search functionality, in this case this is indicated with the string "azure".

0 commit comments

Comments
 (0)