v1.5.3

What's Changed

The following pull requests solve issues related to node12 being deprecated. This upgrade is necessary for a correct deployment of data.all.

• Updated CDK Version to fix issue with cdkproxy/ dataset stack creations by @gmuslia in #476
• update auth-at-edge semantic version to latest 2.1.5 by @dlpzx in #480

Full Changelog: v1.5.2...v1.5.3

v1.5.2

What's Changed

IMPORTANT: the following is a security fix. We encourage users using the data.all Pipelines feature with GitHub templates option to upgrade.

• hotfix: Remove GitHub template option from data.all Pipelines by @dlpzx in #472

Other fixes:

• fix: Fixes issue with existing cognito callbacks by @gmuslia in #464
• fix: Fix lambda/ECS IAM permissions for AOSS by @kukushking in #467
• fix: Upgrade aurora engine version to 11.16 by @kimengu-david in #471
• fix: 465 - Update Aurora default Parameter Group to 'default.aurora-postgresql11'. by @rbernotas in #466
• fix: Bump requests from 2.27.1 to 2.31.0 in /backend by @dependabot in #469
• fix: Bump requests from 2.27.1 to 2.31.0 in /backend/dataall/cdkproxy by @dependabot in https://github.com/awslabs/aws-dataall
• fix: Bump starlette from 0.25.0 to 0.27.0 and upgrade fastapi by @dlpzx in #460

New Contributors

• @rbernotas made their first contribution in #466
• @kimengu-david made their first contribution in #471
• @kimengu-david made their first contribution in #471

Full Changelog: v1.5.1...v1.5.2

v1.5.1

What's Changed

• Solve deployment bug #433 CloudFront logs does not enable ACL access by @akaitoua in #437

• Modify docker-compose yaml to read region and default region from env… by @dlpzx in #446

• Bump flask from 2.0.3 to 2.3.2 in /backend by @dependabot in #439

• Bump flask from 2.0.3 to 2.3.2 in /backend/dataall/cdkproxy by @dependabot in #438

• Bump pymdown-extensions from 8.1.1 to 10.0 in /documentation/userguide by @dependabot in #456

New Contributors

• @akaitoua made their first contribution in #437

Full Changelog: v1.5.0...v1.5.1

v1.5.0

What's Changed

New features:

Check each PR for a complete description of the feature.

• Support OpenSearch Serverless by @kukushking in #292
• Include Pivot Role as part of environment stack (avoiding manual pivot role creation) by @dlpzx in #355
• Configurable restricted VPC for tooling resources by @dlpzx in #337
• Better handling of missing default VPCs and added VPC creation for SageMaker domains by @dlpzx in #427

Bug-Fixes:

• Fix dev Docker images base by @AmrSaber in #387
• Fix get AWS credentials from environment tab by @dlpzx in #391
• Added waiting conditions for slow creation of access points in sharing folders @dlpzx in #392
• Fix shared dbs worksheet list (duplicates) by @noah-paige in #402
• Fix sharing update of storage location by @dlpzx in #404
• Backwards compatibility V1.5 fixes and documentation #431

⚠️ ⚠️ ⚠️ Important ⚠️ ⚠️ ⚠️

Breaking changes

Both the environment and the dataset stacks have been updated in this release. We need to update environment stacks BEFORE creating new datasets or updating existing ones in the environment. There are 3 ways of updating your stacks:

1. Automatically (daily task) - There is an schedule ECS task that updates stacks daily. it has been modified to update environments and then datasets. Until the task is executed environments and datasets won't reflect the latest status of the code and creation of new datasets will fail.
2. Automatically (add CICD stage) - We have introduced an optional CICD stage that triggers the ECS stack-updater task from the CICD pipeline. You need to set enable_update_dataall_stacks_in_cicd_pipeline to true in the cdk.json file to enable this stage (check #355 for more details). In this case the only downtime will be the time in which the CICD pipeline is running.
3. Manually - In data.all console, go to the environment window > Stack tab > click on Update. Once it has completed, go to the required dataset window > Stack tab > click on Update.

Migrating to OpenSearch serverless

If you have deployed data.all with Amazon OpenSearch and would like to migrate to Amazon OpenSearch Serverless,
you would need to migrate the indexes to your new cluster. Although data.all currently does not provide an automated
migration tool, it is possible to do so manually using the following approaches:

• Migrate your indexes to Amazon OpenSearch Serverless with Logstash.
• Migrating Amazon OpenSearch Service indexes using remote reindex

Migrating from manual pivot roles to automatically created pivot roles

If you already have environments which use a manually created Pivot Role and want to upgrade to automatically create the pivot Roles as part of the environment stack, you just need to add the enable_pivot_role_auto_create parameter to cdk.json and set it to true. While the CICD pipeline is upgrading you will experience downtimes because backend, frontend and environment and dataset stacks are updated in different CodeBuild stages. For this upgrade we recommend you to enable_update_dataall_stacks_in_cicd_pipeline to update the environment and dataset stacks, otherwise you can either wait for the daily task or manually update all stacks as explained above.

Special thanks to @kukushking, @nikpodsh, @noah-paige , @AmrSaber for their contributions!
Full Changelog: v1.4.3...v1.5.0

v1.4.3

What's Changed

• Pin alembic version to 'alembic==1.9.4' by @dlpzx in #354
• Bugfix default value cascade:false by @dlpzx in #363
• Bump webpack from 5.75.0 to 5.76.1 in /frontend by @dependabot in #371
• Upgrade sqlalchemy 13.16 -> 1.3.24 and starlette 0.19.1 -> 0.25.0, ariadne 0.13 -> 0.17, fastapi 0.78 -> 0.92 by @dlpzx in #379
• BUGFIX - Add dependency in dataset stack by @dlpzx in #385

Full Changelog: v1.4.2...v1.4.3

v1.4.2

What's Changed

• Fix missing 'glue:DeleteResourcePolicy' to pivotRole in YAML and CDK by @dlpzx in #334
• Fix SageMaker Studio creation. Added kms:createGrant to environmentGroups (no admin groups) by @dlpzx in #335
• Fix Dashboard Quicksight button leads to an inaccessibility page by @wolanlu in #339
• Add ability to bring own certificate to internet_facing=True deployments by @noah-paige in #342
• Fix pivot role logs actions privileges by @dosiennik in #350

Full Changelog: v1.4.1...v1.4.2

v1.4.1

What's Changed

Bug Fixes and enhancements

• Fix dataset dropdown list in worksheet view by @noah-paige in #304
• Bump werkzeug from 2.0.3 to 2.2.3 in /tests and /cdkproxy by @dependabot in #315
• Add cross-region sharing error message/exception by @dlpzx in #322
• Check existing Lake Formation data lake location and make dataset resources names unique by @dlpzx in #324

Full Changelog: v1.4.0...v1.4.1

v1.4.0

What's Changed

New features:

• Sharing State Machines - embedded logic of sharing for more robust sharing workflow. Added testing for sharing managers. by @dlpzx and @degoldner in #267
• Added UI visibility of shared access points and databases by @dlpzx in #289
• Added Pipeline filter in pipeline menu search bar by @noah-paige in #259

Bug-Fixes:

• Added get_dataset_folders and delete folders index in delete datasets by @dlpzx in #280
• Added missing ram:DisassociateResourceShare permission to pivotRole by @dlpzx in #286

Breaking Changes:

Data.all PivotRole needs the ram:DisassociateResourceShare permission. Make sure your PivotRoles include this permission in their policies.

Special thanks to @noah-paige, @degoldner, @luxatwoodmark for their contributions!

Full Changelog: v1.3.1...v1.4.0

v1.3.1

What's Changed

Bug Fixes

• Update the region in which parameters are looked for by @louishourcade in #266
• Fix unauthorized data preview by adding table permissions by @noah-paige in #265
• Added missing PivotRole permissions by @dlpzx in #273

Package updates

• Bump json5 from 1.0.1 to 1.0.2 in /frontend by @dependabot in #269
• Fix Dependabot alert (Prototype Pollution in JSON5 via Parse Method) … by @dlpzx in #270
• Added nth-check resolution to point only to ^2.0.1 versions by @dlpzx in #272

Full Changelog: v1.3.0...v1.3.1

v1.3.0

What's Changed

New features:

• Data.all pipelines enhacements and handle update by @noah-paige in #230
• Introduction of Consumer roles to ease data consumption by @dlpzx in #231
• Configurable Inheritance of Environment tags by @dlpzx in #203
• Configurable Quicksight reader vs anonymous sessions by @dlpzx in #218

Enhancements and Bug-Fixes:

• Add delete and clear notifications by @noah-paige in #252
• Added Quicksight documentation including capacity pricing by @dlpzx in #254
• Added PivotRole IAM Role CDK stack by @dlpzx in #253

Breaking Changes:

Data.all PivotRole has been granted additional permissions for the new features. Make sure that your environments are using the latest version of the pivot role template provided in the code (https://github.com/awslabs/aws-dataall/blob/v1.2.0/deploy/pivot_role/pivotRole.yaml)

Full Changelog: v1.2.3...v1.3.0