CBG-4548 allow creation of partitioned indexes

[torcolvin]

CBG-4548 allow creation of partitioned indexes

• create optional database config index.num_partitions indicating how many partitions to create. By default, indexes will not be partitioned.
• partition only the largest indexes, channels and allDocs which scale for the number of docs and the number of channels in each document.
• move num_index_replicas to index.num_replicas in configuration to allow more index options to be created in the future.
  • num_index_replicas is backward compatible
  • move test code to use index.num_replicas or remove the definition when unnecessary (it is set by RestTester)
• prevent xattrs=false from using partitioned indexes, to make testing easier

I think there's test coverage for everything here, but the things I am most worried about is roundtripping /db/_config and making sure that DbConfig.Index doesn't get set automatically so a GET followed by a PUT would result in a mismatched configuration. Additionally, there is concern/requirement to make this work for legacy configuration.

Pre-review checklist

• Removed debug logging (fmt.Print, log.Print, ...)
• Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
• Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Integration Tests

• GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/2998/

[bbrks]

Not sure whether this PR was intended to implement custom partition counts other than 8. The ticket and PR were a bit vague in terms of just stating "opt-in" - left comments below along those lines though just in case it was not intended

[bbrks]

This looks good 👍

• One nit about DefaultDbConfig to ensure the default (logical) value of 1 is shown
• Missing API docs for db config endpoints

[github-actions]

Redocly previews

• Admin API
• Public API
• Metric API
• Diagnostic API

[bbrks]

All looks good, with one suggestion for documentation which I'll leave for @adamcfraser to review the wording of as well.

stale review

[adamcfraser]

Generally looks great, a few minor comments/questions.

[adamcfraser]

Elsewhere if options.NumPartitions<=1, it's treated as "do not use partitions" (i.e. the check seems to consistently be if options.NumPartitions > 1). This might be the right approach, just want to make sure we're not triggering a fatal error when options.NumPartitions isn't initialized (is zero), if it's not actually a fatal condition.

[torcolvin]

I split the validation for this, once you get to this part of the code we do expect >= 1 to make it simpler to not have to convert everytime here and do a conversion from uninitialized to set to one. I thought this was conceptually easier because it forces the caller to set this value, so the caller decides when to actually make the indexes partitionable.

This was easier for me in testing to make sure I was passing this value through the code but I'm not committed to this approach.

[adamcfraser]

Suggested change

	docker run --rm -d --name ${SG_TEST_COUCHBASE_SERVER_DOCKER_NAME} --volume "${WORKSPACE_ROOT}:/workspace" -p 8091-8096:8091-8096 -p 9102:9102 -p 11207:11207 -p 11210:11210 -p 11211:11211 -p 18091-18094:18091-18094 -p 19102 "${COUCHBASE_DOCKER_IMAGE_NAME}"
	docker run --rm -d --name ${SG_TEST_COUCHBASE_SERVER_DOCKER_NAME} --volume "${WORKSPACE_ROOT}:/workspace" -p 8091-8096:8091-8096 -p 9102:9102 -p 11207:11207 -p 11210:11210 -p 11211:11211 -p 18091-18094:18091-18094 -p 19102:19102 "${COUCHBASE_DOCKER_IMAGE_NAME}"

I'm guessing this is needed based on other ports, but I'm not a docker expert.

[torcolvin]

I added some extra ports based on public documentation and fixed this.