Merge pull request #126 from jkbschmid/licensee-parallelism

Add docs for `CRAWLER_LICENSEE_PARALLELISM`

Author: qtomlinson

service_config/crawler.md

@@ -6,6 +6,7 @@
     - [CRAWLER\_GITHUB\_TOKEN](#crawler_github_token)
     - [CRAWLER\_HOST](#crawler_host)
     - [CRAWLER\_INSIGHTS\_KEY](#crawler_insights_key)
+    - [CRAWLER\_LICENSEE\_PARALLELISM](#crawler_licensee_parallelism)
     - [CRAWLER\_NAME](#crawler_name)
     - [CRAWLER\_QUEUE\_PREFIX](#crawler_queue_prefix)
     - [CRAWLER\_QUEUE\_PROVIDER](#crawler_queue_provider)
@@ -34,6 +35,7 @@ The environmental variables for the cdcrawler-dev App Service include:
 * CRAWLER_GITHUB_TOKEN
 * CRAWLER_HOST
 * CRAWLER_INSIGHTS_KEY
+* CRAWLER_LICENSEE_PARALLELISM
 * CRAWLER_NAME
 * CRAWLER_QUEUE_AZURE_CONNECTION_STRING
 * CRAWLER_QUEUE_PREFIX
@@ -87,6 +89,12 @@ Note that we only use this in the development environment, not in the production
 
 We use [Azure Application Insights](https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview) to monitor the crawler application. This requires a key and this is where it is kept.
 
+### CRAWLER_LICENSEE_PARALLELISM
+
+This is the maximum number of `licensee` processes to run in parallel. `licensee` is a tool to collect license
+information. The default value is `10` and setting it to a smaller value can reduce CPU spikes and lead to the crawler
+having a more uniform CPU usage.
+
 ### CRAWLER_NAME
 
 This is a name to refer to the crawler with. Note that we set it in the App Service in the development environment and in [the Docker file](https://github.com/clearlydefined/crawler/blob/32a0d6b59edfda5d3226c50680e4a8338af395cd/Dockerfile) for the Prod environment.