All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- a new
filmdrop-titilermodule that provides a lightweight implementation of the most recent release of TiTiler (239). This is a new module part of a significant migration and should undergo extensive functionality and security testing before production use.
-
Added SSM parameters for Cirrus pre-batch and post-batch lambda function ARNs (
CIRRUS_PRE_BATCH_LAMBDA_ARNandCIRRUS_POST_BATCH_LAMBDA_ARN) -
Added SSM parameter for Cirrus process queue ARN (
CIRRUS_PROCESS_QUEUE_ARN)
-
A new stac_server_inputs.custom_vpce_id var added. If provided, the user is indicating that they have an existing vpc endpoint that the stac server api gateway (and supporting resources) should allow to ingress
-
A new titiler_inputs.custom_vpce_id var added. If provided, the user is indicating that they have an existing vpc endpoint that the titiler api gateway (and supporting resources) should allow to ingress
-
A new titiler_inputs.allowed_extensions_enabled var added. Set to
false(default istrue) to exclude the GDAL'sCPL_VSIL_CURL_ALLOWED_EXTENSIONSenv var from the titiler API Lambda environment. This allows GDAL to access extension-less proxy hrefs when asset proxying is enabled in stac-server. -
A new input variable and resource block offers ability to optionally deploy cirrus api lambda with provisioned concurrency (240)
-
The default version of cirrus lambda version was bumped from 1.1.2 -> 1.3.0, and the package stac-server infrastructure version from 2.0.0 -> 2.0.2
- stac-server OpenSearch
opensearch_cluster_instance_countis now editable to >0 && <3 for lower-cost deployment scenarios; seeopensearch_cluster_zone_awareness_enabledandopensearch_cluster_availability_zone_countdescriptions in the terraform-aws-stac-server repository
- stac-server OpenSearch
-
stac_server_inputs.api_provisioned_concurrencyenables provisioned concurrency on the stac-server API lambda -
titiler_inputs.api_provisioned_concurrencyoffers the ability to optionally deploy the titiler lambda with provisioned concurrency (245) -
stac_server_inputs.opensearch_logsoptionally enables various logging types on the opensearch cluster
-
Minor changes (#218) to how TiTiler behaves when is_private_endpoint = true, supporting communications with its API Gateway and VPC Endpoint
-
TiTiler module will note three IAM related resources being created: aws_iam_role_policy.titiler-mosaic-lambda-inline-policy, aws_iam_role_policy_attachment.lambda_basic_execution, aws_iam_role_policy_attachment.lambda_vpc_access. This is a result of AWS provider deprecations to managed_policy and inline_policy properties. The TiTiler role in question will updated in place, resulting in no material changes to the role.
-
Removing internal stac-server terraform module code and pointed to new location as stac-server terraform module code now lives in it's own stac-server terraform repo (216).
-
Removing internal console-ui terraform module and pointing to an externally sourced module filmdrop-ui (216).
-
cirrus apilambda extracted into its own module, and can now be optionally deployed (or not) along with other cirrus built in lambdas when deploying the larger cirrus ecosystem (248)
-
Corrected permissions in Workflow Metrics CloudWatch write policy to include necessary wildcard, and in the read policy to include listing metrics. (#237)
-
TiTiler's API Gateway points to the published version of it's API Lambda, so that provisioned concurrency function as expeted
- If you have stac server, cirrus, or titiler (as private) deployed`, Terraform will note that a null_resource.enable_access_logs is being destroyed for each. This is expected. Previously, api gateway access logs were enabled via this null resource; they are now enabled via an aws_api_gateway_stage resource.
- Changed all
console-uireferences tofilmdrop-uias part of module extraction and renaming to align with broader FilmDrop ecosystem naming conventions. All resources withconsole-uinaming will be destroyed/re-created with the new name convention 216
- See MIGRATION.md for important updates to the AWS provider version to v6, and the terraform-aws-stac-server module to v2
- See MIGRATION.md for important information regarding the name change of the module
console-uitofilmdrop-ui - See MIGRATION.md for important information regardint the change to optional deployment of
cirrus apilambda
-
Added feeders as configuration (177). For common Cirrus "feeder" use cases, S3 and SNS, this feature allows for defining trigger -> sqs -> lambda infrastructure via config in a manner similar to tasks and workflows. Note: existing projects will see two additional output values
CIRRUS_PROCESS_QUEUE_ARNandCIRRUS_PROCESS_QUEUE_URLincluded in three existing maps -
Added
workflow_metrics_timestream_enabledability to disable Timestream for LiveAnalytics, which has been deprecated. Note: existing projects will see destruction and creation of Lambda IAM resources. These should result in no material change to actual roles/policies; they're a destructuring of larger policies into smaller modular policies -
Added Cirrus Workflow Metrics resources and configuration per #191, and in congruence with cirrus-geo#329 to enable continued support of the cirrus-dashboard metrics page (#191).
- Added missing
ephemeral_storage_mbfield tocirrus_taskstyped definition inmodules/cirrus/typed-definitions/inputs.tfto match thetask_config.lambdaobject schema inmodules/cirrus/task/inputs.tf
-
Moved stac-server to a standalone module https://github.com/Element84/terraform-aws-stac-server
-
Terraform >= v1.13.0 is now required
-
Added support for custom environment variables in STAC Server Lambda functions (
api_lambda,ingest_lambda,pre_hook_lambda). Users can now pass custom environment variables via the optionalenvironment_variablesparameter, enabling support for STAC Server v4.4.0+ features likeENABLE_CONTEXT_EXTENSIONandENABLE_THUMBNAILS. This enhancement is fully backward compatible. -
If using the optional
cirrus_inputs.lambda_versionorcirrus_inputs.lambda_zip_filepathto denote a specific version of Cirrus, you must additionally define acirrus_inputs.lambda_pyversion. Cirrus geo versions are now tied to specific Python runtime versions; see the cirrus-geo releases for details.
-
Cirrus Lambda builtins (api, process, update_state, pre_batch, post_batch) now require a minimum of 512 MB memory. If you've explicitly set memory values lower than that in your
tfvarsan error will occur, noting you need to increase the value. If you're using the defaults, the nextterraform applyrun will update your Lambdas in place. -
Updated Terraform version dictated in .terraform-version to latest stable 1.13.4. While not technically a semver breaking change, you may want to review the Terraform upgrade guides for 1.8, 1.9, 1.10, 1.11, 1.12, and 1.13
- Added back support for deploying with custom Cirrus lambda zip
- Option to enable private dns for execute-api VPC endpoint created for stac-server
- Added inputs to configure stac-server with enable_collections_authx (stac-server>=v4.4.0)
- Added support for cirrus tasks with mutable tags
- Moved Cirrus task related permissions into separate IAM policy
- Added to TF outputs mapping of Cirrus lambda/batch tasks to IAM role arns
- Added CIRRUS_PAYLOAD_BUCKET as builtin template variable
- Added CIRRUS_DATA_BUCKET to parameter store
- Download specified version of Cirrus lambda zip from GitHub during deployment
- Support for Cirrus template variables sourced from SSM parameters
- Support for External AWS accounts Publish to Cirrus Proccess Queue
- Updated Pangeo base image for Daskhub from 2022 to 2025 with dependency locking
- stac-server configurations for items_max_limit and enable_response_compression
- Outputs for the stac-server ingest and api lambda functions including their name and arn
- Reverting unintended
filmdrop_ui_config_fileandfilmdrop_ui_logo_filedefault value changes from the inputs.tf
- Support for PRIVATE-type TiTiler API Gateways
- Added Custom Domain Name support for TiTiler, Stac and Cirrus Private API Gateways
- Enforced character limit on cirrus
resource_prefix
- Added capability to pass a custom S3 bucket for Console UI and Cirrus Dashboard S3 websites
- Added outputs for Console UI and Cirrus Dashboard S3 Bucket names
- stac_server: set ENABLE_INGEST_ACTION_TRUNCATE on the ingest lambda rather than the api lambda
- Input to configure stac-server with ENABLE_INGEST_ACTION_TRUNCATE
- removing step function permission from
update-stateIAM policy due to change in howupdate-statelambda gets errors in cirrus v1.0.0 release
- The variable
enable_collections_authxdefaulted to true, should have defaulted to false.
- Added inputs to configure stac-server with stac_id, stac_title, stac_description, and enable_collections_authx
- Added input to allow configuration of stac-server api gateway method authorization type
- Corrected resource string for
update-stateandprocesslambda function roles to accessStateDBindexes.
- Added configuration
stac_server_inputs.opensearch_version - Added output for stac-server gateway id
- Updated default OpenSearch version to 2.17, from 2.13
- Updated default stac-server version to v3.10.0, from v3.8.0
- Updated default stac-server Lambda runtime to Node 20, from 18
- Corrected resource string for
cirrusAPI lambda function role to accessStateDBindexes.
stac_versionis no longer supported for configuration
- Exposing stac_server_ingest_sns_topic_arn via outputs
- Exposed stac_server_lambda_iam_role_arn via outputs
- Adding capability to change the Stac-Server OpenSearch availability_zone_count via input parameter
- Documentation for the
cirrusmodule and itstask-batch-compute,task, andworkflowsubmodules - Direct invocations of the
cirrusmodule can now specify a custom resource prefix - A list of additional security groups can now be added to the VPCes for PRIVATE-type stac-server and/or cirrus API gateways
- Cirrus workflows can now have custom permissions applied to the state machine's execution IAM role
- Cirrus workflows can now invoke any AWS service that provides a state machine integration
- The cirrus
task-batch-computesubmodule now supports parameterization of its definition YAMLs through templating - The cirrus
workflowsubmodule now supports parameterization of its definition YAMLs and state machine JSONs through templating - The
stac-serverAPI'sSTAC_VERSIONenvironment variable can now be specified via input variablestac_version - Cirrus deployment parameter store and optional assumable management role intended for the cirrus CLI tool
- Renamed all instances of
cirrus_prefixtoresource_prefixas a preliminary step for adopting the latter as an input variable across all modules
- Cirrus workflows no longer require at least one cirrus task reference
- Cirrus
pre-batchandpost-batchlambdas now work correctly with a payload bucket managed bycirrus'sbasemodule
- Updated FilmDrop Analytics eks kubernetes version to 1.32 and autoscaler version to 1.32.0
- Removed classical WAF support as it is being deprecated by AWS in favor of the newer AWS WAFv2
- Removed max terraform required version constraints
- Fixed incorrect default
cirrusvariables
- Fixed occasional planning-time issue with a
data.aws_subnetblock in thecirrusandstac-servermodules
- Removing monokai from daskhub Dockerfile which was causing the jupyterhub image build to fail
- Custom cirrus lambda dist ZIP can now be used instead of the default
- Custom stac-server lambda dist ZIPs and configuration overrides can now be used for the
api,ingest, andpre-hooklambdas - Support for PRIVATE-type cirrus and stac-server API Gateways
- Ephemeral storage option for cirrus lambda tasks
- Inputs to the
cirrusmodule'stask-batch-compute,task, andworkflowsubmodules are now defined via YAML files instead of HCL object lists - Cirrus workflow's
template_variablesconfig is removed in favor of referencing cirrus task output attributes directly
- Fixed the Cirrus
update-statelambda permissions to allow:- Pushing messages to the Cirrus
publishSNS topic - Creating objects in the Cirrus
payloadS3 bucket
- Pushing messages to the Cirrus
- Fixed Cirrus workflow state machine permissions to allow creating state transition events
- Fixed constant state drift caused by multiple
aws_api_gateway_accountresources (one instac-server, one incirrus)
- Final migration of Cirrus IaC to Terraform for compatibility with
cirrus-geov1.0.0a0 and beyond. The following modules were created to manage all Cirrus Task and Workflow resources through input variables:modules.cirrus.task_batch_computemodules.cirrus.taskmodules.cirrus.workflow
- Moved
modules.cirrus.functionsmodule tomodules.cirrus.builtin_functions - Moved
modules.cirrus.base-builtinsmodule tomodules.cirrus.base
- Base Cirrus alarms
- Default FilmDrop Warning and Critical SNS Topics
- Builtin lambdas added to cirrus module along with script to update deployment zip
- API Gateway infrastructure for Cirrus API
- Creation of Cirrus Data and Payload S3 bucket if none is defined via inputs
- Consolidated WAF rules into a single one by default for cost savings
- Rolled back vpc infrastructure changes to support creation of VPC if
deploy_vpcis set totrue.
- Default to stac-server 3.8.0 and OpenSearch 2.13
- For both
stac_server_inputsandtitiler_inputs, renamedstac_server_and_titiler_s3_arnstoauthorized_s3_arns. private_subnets_az_to_id_mapnow correctly using ID as the map value instead of previous cidr_blockpublic_subnets_az_to_id_mapnow correctly using ID as the map value instead of previous cidr_block
- titiler-mosaicjson configuration parameter
mosaic_tile_timeout
- VPC and subnets are no longer created by the FD VPC module, since IDs must now be provided
for preexisting resources. If
deploy_vpcwas set totrueon a previous terrform apply, then this update will to attempt to delete the VPC and subnets, which will fail due to resource dependencies. The TF state will need to be manually updated to remove these references without deleting the underlying AWS resources.
- Added Cirrus terraform base resource set and new cirrus terraform module
titiler_inputs.mosaic_titiler_release_tagis nowtitiler_inputs.versioncirrus_dashboard_inputs.cirrus_dashboard_releaseis nowcirrus_dashboard_inputs.versionconsole_ui_inputs.filmdrop_ui_releaseis nowconsole_ui_inputs.version
- sample data bucket module has been removed, as it was unused in any projects
- Add default values to console-ui inputs to allow tflint validation.
- Allow 7 instead of 5 characters for
environment
- Fixed filmdrop built-in vpc output references and mappings
- Default to filmdrop-ui version v5.3.0
- Default to stac-server v3.7.0
- Adding support for stac-server API Lambda environment configuration:
- Access-Control-Allow-Origin:
CORS_ORIGIN - Access-Control-Allow-Credentials:
CORS_CREDENTIALS - Access-Control-Allow-Methods:
CORS_METHODS - Access-Control-Allow-Headers:
CORS_HEADERS
- Access-Control-Allow-Origin:
- Added consistent naming for CloudFront Basic Auth and other resources.
- Added GitHub Actions workflow to test new commits to main branch and new releases.
- Added GitHub Actions workflow manual trigger to test new commits and PRs.
- Added ci.tfvars with minimal configuration for GitHub Action testing, no CloudFront, no Analytics and stac-server with OpenSearch Serverless.
- Updated changelog to adhere to spec.
- Updated Terraform AWS Provider minimum version to 5.47.
- Updated CloudFront deployment as optional for FilmDrop UI.
- Updated CloudFront deployment as optional for Cirrus Dashboard.
- Updated FilmDrop Analytics eks kubernetes version to 1.29 and autoscaler version to 1.29.0.
- Added FilmDrop Analytics cleanup capability.
- Changed mosaic titiler lambda bucket to generate unique bucket name.
- Changed stac-server security group to generate unique sg name.
- Added VPC support for titiler-mosaicjson.
- Fixing basic auth CloudFront function.
- Changed Cirrus Dashboard variables for explicitly requiring inputs for cirrus_api_endpoint and metrics_api_endpoint.
- Adding support for STAC_API_URL env variable for stac-server lambdas.
- Adding support for creating a BasicAuth CloudFront function.
- Fixed input parameters for creating CloudFront functions.
- Adding support for custom origin port for load balancer endpoints.
- Uses v5.0.0 of the filmdrop-ui by default
- Added flag to deploy stac-server resources, including OpenSearch within or outside the vpc, defaults to within vpc.
- Enabling stac-server post ingest sns publishing
- Added self-managed, managed and fargate node group capability to eks module
- Fixed custom domain alias and certificate creation for filmdrop endpoints
- Fixed analytics dask helm installation
- Update to require version 1.6.x or 1.7.x of Terraform (instead of ~>1.6.6).
- Update to default to stac-server v3.7.0 (from v3.2.0)
- Update stac-server to use OpenSearch 2.11
- Updated terraform supported version to 1.6.6
- Updating public_subnets_cidr_map name variable name to public_subnets_az_to_id_map
- Updating private_subnets_cidr_map name variable name to private_subnets_az_to_id_map
- Updating analytics load balancer subnets
- Updating analytics ebs csi driver repo
- Added historic and ongoing ingest capability as stac-server submodules
- Added capability for optional CloudFront deployment for stac-server, with a parameter in the stac_server_inputs
- Added OpenSearch Serverless capability to stac-server module
- Added READMEs for titiler and mosaic-titiler linking to unit test instructions and general documentation
- Added flop CLI utility for creating and interacting with FilmDrop test environments
- Added built-in validation of project_name and environment parameters
- Adding FilmDrop profiles for deploying components via flags and enabling a 1-step deployment via tf-modules repo
- Fixes kubectl version on codebuild for jupyterhub analytics module
- Default cirrus_dashboard_release_tag to v0.5.1
- Update stac-server to use OpenSearch 2.9
- The jupyterhub-dask-eks module no longer takes an input
kubernetes_cluster_name, but now requires a parameterenvironment. Resource names that previously usedkubernetes_cluster_namenow construct those using theproject_nameandenvironmentvariables - All jupyterhub-dask-eks and mosaic-titiler module CodeBuild projects are now set with a concurrency of 1.
- The jupyterhub-dask-eks module no longer takes inputs
filmdrop_analytics_jupyterhub_admin_credentials_secretorfilmdrop_analytics_dask_secret_tokens, but instead constructs these from theproject_nameandenvironmentas${var.project_name}-${var.environment}-admin-credentialsand${var.project_name}-${var.environment}-dask-token - jupyterhub-dask-eks CodeBuild project must be manually run, instead of it being run automatically in response to a configuration change.
- jupyterhub-dask-eks configuration bucket has been renamed from
jupyter-config-${random_id.suffix.hex}tofd-${var.project_name}-${var.environment}-jd-cfg-${random_id.suffix.hex} - jupyterhub-dask-eks AWS EKS version has been updated to 1.25
- lowercase aws_s3_bucket.jupyter_dask_source_config S3 bucket name
- add .snyk file to ignore rules for public S3 buckets and open auth to API gateway
- The stac-server module renamed numerous resources to use project/stage naming format. See README.md for upgrade instructions it you have a preexisting stac-server OpenSearch cluster than needs to be preserved upon taking this update.
- Various issues fixed related to stac-server resource name changes
- Removed invalid default values for stac-server variables
vpc_security_group_idsandvpc_subnet_ids
- fix args being passed to the cloudfront/custom module which were removed in a lint/cleanup commit
- console-ui.filmdrop_ui_release must be gte 4.x, e.g.,
v4.0.1
- Defining a non-empty
stac_server_s3_bucket_arnsandtitiler_s3_bucket_arnsparameters is no longer required. This can now be empty. - Refined public bucket access policies
- SSM secret creation will now fail if a secret with the same name already exists
- add an optional env var config for mosaic lambda (request_host_header_override) that will overridethe host header in the event, so that responses crafted use the desired external-facing domain instead of internal API gateway
- set force_delete on daskhub and titiler ECR repos to allow automated destroy
- set force_destroy access logs and logs archive S3 buckets to allow automated destroy
- OpenSearch Service linked role is no longer managed by these modules, but instead should be created using the bootstrap project.
- Adding configurable disk_size and capacity_type to eks node groups
- Add WAF rules to check requests for mosaic titiler - consumer must set "waf_allowed_url" to enable (updated to allow OPTIONS)
- Sets element 84 distribution email as maintainer for daskhub dockerfile
- explicitly set bash interpreter for local-exec shell scripts, and don't swallow errors (-e)
- add a trigger for trigger_console_ui_upgrade on config file contents
- add status check to wait for console ui codebuild to complete and return success/failure
- Updated wget TLS to v1.3 (Github began denying the default)
- updated TF resource schemas to work with AWS provider v5
- Add cloudwatch alarms to mosaic titiler module
- Removes stac ingest policy assignment within the stac server ingest sqs resource
- Utilizes the stac ingest sqs arn to build the correct access policy
- Reverts removal of base_infra/alerts
- Added support jupyterhub deployment as http load balancer if no acm certificates are specified
- Add a variable for stac-server to use the correct root path when cloudfront is used
- For stac-server, OpenSearch 2.7 will be used instead of 2.3.
- truncate S3 cloudfront content_bucket name to 63 characters
- add optional
stac_server_s3_bucket_arnsconfig input list to stac-server to grant S3 GetObject permissions - Fix inconsistent plan in jupyterhub trigger by triggering from S3 events
- Try to fix yet another S3 race condition applying versioning/replication config
- Export API gateway ID from mosaic-titiler to fix hard-coded config in top-level TF file
- Fixes jupyterhub race condition for bucket ACLs
- Console UI bucket ACL creation now depends on the ownership permission being applied first
- Mosaic TiTitler module now creates directory and checks for wget
- FilmDrop UI >= 3.0.0 is now required. The configuration file is now
./public/config/config.jsoninstead of./src/assets/config.js
- DNS validation capability to allow for cloudfront urls instead of custom aliases
- Adding missing dns_validation input variable on the s3_website module
- Cloudfront default alias
- Adding dns validation capability to allow for cloudfront urls instead of custom aliases
- Multiple deployments now works correctly
- Explicitly sets titiler docker image local exec interpreter
- Fixed EKS permissions and adding SSM Bastion
- Prevent recreation of SSM bastion host
- Adding missing dns_validation input variable on the s3_website module
- Cloudfront default alias
- Partial support for multiple deployments
- Many modules now require
project_nameto be defined, includingbase_infra/log_archive,cloudfront/s3_website,cloudfront/lb_endpoint,cloudfront/apigw_endpoint, andstac-server - Module
titlerrequiresproject_name,prefix, andtitiler_stagevariables set (may be required from an earlier release) - Module
jupyterhub-dask-eksrequiredsdaskhub_stagevariables set (may be required from an earlier release)
- Add dependency of user_init_lambda_zip for the opensearch lambda
- TiTiler bucket ownership Rules
- Pin TiTiler FastAPI version to 0.95 to fix routing issue
- EKS permissions
- Support for FilmDrop UI custom logos
- Bucket permissions
- Changed Filmdrop UI config from .env to config.js
- EKS module
Many changes, see commit history
- Fixed issue with stac-server opensearch user-init lambda not building except on initial deploy.
- Added support for deploying the stac-server auth key pre-hook lambda. This will deploy
by default (tbd not having it deploy). Setting
stac_server_auth_pre_hook_enabledorstac_server_pre_hook_lambda_arnwill cause it not to be used. When enabled, this uses an AWS Secrets Manager secret namedstac-server-${stage}-api-auth-keysto store a JSON value that contains a mapping of key (token) values to permission values. Currently, the only permission allowed iswrite, which allows read of everything and write if the Transaction Extension is enabled.
- Start of changelog