Migrate occurrence-ws, gbif-pipelines, common-search, maps and metrics-ws to Elasticsearch 9.3.0(latest)

[fmendezh]

Migrate multiple GBIF services to Elasticsearch 9.3.0 based on the experience gained from the GBIF registry migration. This migration requires coordinated changes across several repositories and careful testing in the labs environment.

Scope

The following services need to be migrated:

• occurrence-ws - Occurrence web service
• occurrence-download - Occurrence small downloads
• gbif/pipelines - Data ingestion pipelines
• metrics-ws - Metrics web service
• vectortile-server - Maps service

Implementation Strategy

Prerequisites

• Review and document lessons learned from the GBIF registry Elasticsearch migration
• Provision a small Elasticsearch 9.3.0 cluster in Kubernetes for testing
  • Initially can use the current core-search installation for small test indices

Branch Strategy

Each component should be implemented in isolated branches to allow independent development and testing:

1. Adapt the GBIF common-search library to support Elasticsearch 9.3.0
2. occurrence-ws: Create feature branch for ES 9.3.0 migration
3. metrics-ws: Create feature branch for ES 9.3.0 migration
4. vectortile-server: Create feature branch for ES 9.3.0 migration
5. gbif/pipelines: Create branch based on pipelines v2 (branch 4.x) for ES 9.3.0 support### Deployment Strategy- All deployments should be done in LABS environment

• DO NOT deploy to DEV environment during migration
• This allows for thorough testing without impacting development workflows

Tasks

1. Infrastructure Tahs

• Provision Elasticsearch 9.3.0 cluster in Kubernetes
• Size: Small cluster for initial testing
• Location: Labs environment
• Option: Leverage existing core-search installation for test indices
• Configure access and connectivity for labs environment, this includes DNS/IP external access.
• Set up monitoring for ES 9.3.0 cluster

2. Common Library Updates

• Update gbif/common-search library to support Elasticsearch 9.3.0
  • Review breaking changes in ES 9.3.0 API
  • Update client dependencies
  • Adapt query builders and search functionality
  • Update index management utilities
  • Add backward compatibility considerations if needed

3. gbif/pipelines Migration

• Create new branch based on pipelines v2 (branch 4.x)
• Update pipeline dependencies for ES 9.3.0
• Adapt ingestion logic to work with ES 9.3.0
• Update indexing operations
• Modify bulk ingestion handling
• Update mapping definitions
• Adjust data transformation if needed
• Test ingestion with small datasets in labs
• Validate data integrity and search functionality

4. occurrence-ws, vectortile and metrics-ws Migration

• Create isolated feature branch
• Update dependencies to use new common-search library
• Update Elasticsearch client to 9.3.0
• Adapt search queries and aggregations
• Update index configurations and mappings
• Test API endpoints against ES 9.3.0 cluster in labs

5. Testing & Validation

• Create test data set for validation
• Execute integration tests for each service
• Compare results between old and new Elasticsearch versions
• Performance benchmarking

7. Documentation

• Update deployment documentation
• Document configuration changes
• Update GBIF tech-docs

Notes

• Use isolated branches for each component to maintain independence
• Labs-only deployments ensure safe testing environment
• Consider creating a migration checklist for each service based on registry experience

[MattBlissett]

While doing this, can we consider reorganizing occurrence/metrics so at the end we have

• One webservice instance that handles ES-backed queries (currently occurrence-ws and metrics-ws)
• Another that handles HBase-backed (verbatim/fragment) and PG-backed (download requests) requests?

It would then be easier to monitor and handle surges in use of either side without affecting the other.

This doesn't necessarily have to be two separate modules, it could also be two deployments of the same service with different names (or whatever) so Varnish can direct to the appropriate one.

[MattBlissett]

We will also need at least a rough deployment plan. We have the disk space on /index/? to store an additional full index in prod, but will need to ensure we don't overload CPU/RAM on the machines while transitioning.

Everything under https://informatics-docs.gbif.org/elasticsearch-overview will need review/updates.