## Why Closes #223 We want to move the indexing-service from the current lambda-based deployment to a fargate task running in ECS. The first step to achieve this is merging the different functions done by the lambdas into a single, containerized service. ## How The indexing-service features are implemented via the following lambdas: - `getclaim` - `getclaims` - `getdiddocument` - `getroot` - `postclaims` - `notifier` - `remotesync` - `providercache` ### HTTP endpoint lambdas `getclaim`, `getclaims`, `getdiddocument`, `getroot` and `postclaims` are all HTTP lambdas triggered by requests to API Gateway endpoints. These can be easily replaced by a standard HTTP server exposing the different routes. https://github.com/storacha/indexing-service/blob/e91e53712af9a5470cbe321418055be6507037db/cmd/aws.go#L83 ### `notifier` and `remotesync` These two lambdas work in tandem. `notifier` fetches the latest advert from IPNI and compares that with the current chain. For adverts that had already been ingested by IPNI, it notifies the `remotesync` lambda, that will set the corresponding provider info entries as expirable in the cache. It's ok for these entries to be evicted from the cache now that they can be found in IPNI. `notifier` is triggered by a periodic CloudWatch event, which is replaced by a regular ticker. It triggers the `remotesync` lambda by issuing a SNS notification. Since both components will be running in the same service now, they can be just connected together. The `setupIPNIPipeline` function creates a `Notifier` and a `RemoteSyncer` and sets them up so that the syncer is invoked directly when needed. https://github.com/storacha/indexing-service/blob/e91e53712af9a5470cbe321418055be6507037db/cmd/aws.go#L87-L109 ### `providercache` The main goal of the indexing-service is to cache content claims before they are available through IPNI. Caching index claims can take a long time for big indexes, as an entry is stored for each shard. Therefore, this is done asynchronously. When the indexing service fetches an index, it will store it in an S3 bucket and put a message in a SQS queue with the corresponding details. The `providercache` lambda will be triggered by messages in the SQS caching queue, and do the actual store in the cache. Given that the indexing-service will run as a monolithic service, the SQS queue could be replaced with some local queue. Using the SQLite-backed job queue used in Piri would be an option, since we want the queue to be persistent and survive service restarts. I kept the SQS queue for now to simplify this PR as much as possible. This also allows the queue to be shared by multiple tasks when the service is scaled horizontally, sharing the load to cache indexes. But we can consider getting rid of the SQS queue if an alternative would be more cost-efficient. SQS queues have the ability to trigger a lambda. They can also trigger ECS tasks by defining an EventBridge Pipe, but I'm not sure whether the pipe can target an already running task or it spawns its own. Again, to simplify the changes in this PR I created a simple poller that fetches batches of messages periodically from the queue and caches them. This approach can be revisited in the future if we prefer a push-based mechanism. As part of the implementation of this new component, I moved some interface definitions from the `blobindexlookup` package to the `providercacher` package and added new methods to the `CachingQueue` interface so that caching jobs can also be read and deleted from the queue. ## Known issues - Service initialization is a bit weird right now because I didn't touch the function that builds the service for the lambdas, as that needs to be working until the deployment configuration is updated. Don't pay too much attention to that since it will be taken care of afterwards (see #225). - The `notifier` lambda was configured to check IPNI [every minute](https://github.com/storacha/indexing-service/blob/e91e53712af9a5470cbe321418055be6507037db/deploy/app/lambda.tf#L388-L392). However, the IPNI notifier has a fixed polling interval of 30 seconds. If this is relevant, let me know and I'll update the IPNI notifier implementation to accept a configurable interval. - The caching queue poller has a default poll interval of 5 seconds and it will fetch batches of 10 messages. Feel free to suggest alternative values. ## Testing For now, this needs to be deployed to be tested, so testing will be done in staging once available.

Post-deployment-strategy-change cleanup (code) #225

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions