Releases · open-metadata/OpenMetadata

Support for Postgres as OpenMetadata Store #4601
UI Improvements in 0.10.1 Release #4600
Support JWT Token Generation for Bot Accounts #4637
UI Ingestion Improvements - Support for Dashboards & Messaging Services #4843
Security: Fix Azure SSO and support refresh tokens (#4988) by @vivekratnavel in #4989

New Contributors

@cometta made their first contribution in #2916
@snyk-bot made their first contribution in #3130
@Beetelbrox made their first contribution in #3185
@sfirke made their first contribution in #3361
@mosiac1 made their first contribution in #3599
@troyel made their first contribution in #3695
@dvecerdea made their first contribution in #3800
@Curiosady made their first contribution in #3892
@barloc made their first contribution in #4037
@Vj-L made their first contribution in #4092
@TeraInferno made their first contribution in #3720
@vitoravancini made their first contribution in #4733
@habibrhmn made their first contribution in #4800
@filip-xmargin made their first contribution in #4926
@mohitdeuex made their first contribution in #4956

Full Changelog: 0.10.0-release...0.10.1-release

Contributors

cometta, vivekratnavel, and 14 other contributors

Assets 5

26 Apr 17:33

akash-jain-10

0.10.0-release

5200834

OpenMetadata 0.10.0-release

What's Changed

Support for Database Schema

OpenMetadata supports databases, service name databases, and tables. We’ve added Database Schema as part of the FQN. For each external data source, we ingest the database, as well as the tables that are contained underneath the schemas.

Support for Hard Delete

OpenMetadata supported soft deletions. Now, we also support the hard deletion of entities through the UI, APIs, and ingestion. Hard deleting an entity removes the entity and all of its relationships. This will also generate a change event.

Deploy Ingestion from UI

OpenMetadata has refactored the service connections to simplify the ingestion jobs from both the ingestion framework as well as the UI. We now use the pydantic models automatically generated from the JSON schemas for the connection definition. The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata.

Download DBT Manifest Files from Amazon S3 or Google Cloud Storage

Previously, when ingesting the models and lineage from DBT, we passed the path of the DBT manifest and catalog files directly into the workflow. We’ve worked on improving the quality of life of DBT. Now, we can dynamically download these files from Amazon S3 or Google Cloud Storage. This way we can have any other process to connect to the DBT, extract the catalog, and put it into any cloud service. We just need the path name and workflow job details from the metadata extraction to be able to ingest metadata.

JSON Schema based Connection Definition

Each service (database, dashboard, messaging, or pipeline service) has its own configuration specifications, with some unique requirements for some services. Instead of the ad hoc definitions of the source module in Python for each connector, we’ve worked on the full refactoring of the ingestion framework. We now use the pydantic models automatically generated from the JSON schemas for the connection definition.

Airflow Rest APIs

The Airflow REST APIs have been refactored. With our API centric model, we are creating a custom airflow rest API directly on top of Airflow using plugins. This passes the connection information to automatically generate all the dags and prepares handy methods to help us test the connection to the source before creating the service.

UI Changes

The UI improvements are directed towards providing a consistent user experience.
Hard Deletion of Entities: With the support for the hard deletion of entities, we can permanently delete tables, topics, or services. When the entity is hard deleted, the entity and all its relationships are removed. This generates an ‘EntityDeleted’ change event.
Dynamic “Add Service” Forms: The ‘Add Service’ form is automatically generated in the UI based on the JSON schema specifications for the various connectors that are supported in OpenMetadata.
UI Support for Database Schema as part of FQN: The database schema has been introduced in the 0.10 release. All the entity pages now support Database Schema in the UI.
Lineage Editor: Improvements have been made to the lineage editor.
Teams: While signing up in OpenMetadata, the teams with restricted access are hidden and only the joinable teams are displayed.
Team Owner: An Owner field has been added to the Team entity. Only team owners can update the teams.
Activity Feeds: The Activity Feeds UI supports infinite scrolling.
Add User: A user can be added from the Users page.

Security Changes

Support Refresh Tokens for Auth0 and Okta SSO

The JWT tokens generated by the SSO providers expire by default in about an hour, making the user re-login often. In this release, we’ve added support for refresh tokens for Auth0 and Okta SSO. The tokens are refreshed silently behind the scenes to provide an uninterrupted user experience. In the future releases, we’ll continue to stabilize authentication and add refresh tokens for the other SSO providers.

Custom OIDC SSO

OpenMetadata now supports integration with your custom built OIDC SSO for authentication. This is supported both on the front end for user authentication as well as on the ingestion side.

Azure SSO

Support has been added for Azure SSO on Airflow.

New Contributors

@cometta made their first contribution in #2916
@Beetelbrox made their first contribution in #3185
@sfirke made their first contribution in #3361
@OnkarVO7 made their first contribution in #3408
@mosiac1 made their first contribution in #3599
@troyel made their first contribution in #3695
@dvecerdea made their first contribution in #3800
@Curiosady made their first contribution in #3892
@MilanBariya made their first contribution in #3949
@barloc made their first contribution in #4037
@Vj-L made their first contribution in #4092
@TeraInferno made their first contribution in #3720

Full Changelog: 0.8.1-release...0.10.0-release

Contributors

cometta, TeraInferno, and 10 other contributors

Assets 5

08 Apr 14:20

akash-jain-10

0.9.1-release

4b9798e

OpenMetadata 0.9.1-release

Collaboration

Conversations in the main feed
Users can ask each other questions, add suggestions and replies
Table details - Click through on usage to see who or what services are using it, and what queries are pulling from it.

Data Quality

Ability to create and monitor the test cases
Data Quality Tests support with Json Schemas and APIs
UI Integration to enable users to write tests and run them on Airflow

Glossary

Glossaries are a Controlled Vocabulary in an organization used to define the concepts and terminologies specific to a particular domain.
API & Schemas to support Glossary
UI support to add Glossary and Glossary Terms.
Support for using Glossary terms to annotate Entities and Search using Glossary Terms

Connectors

Apache Iceberg
Azure SQL
Clickhouse
Clickhouse Usage
Databricks
Databricks Usage
Delta Lake
DynamoDB
IBM DB2
Power BI
MSSQL Usage
SingleStore
Apache Atlas, Import Metadata from Apache Atlas into OpenMetadata
Amundsen, Import Metadata from Amundsen into OpenMetadata

Lineage

DataSource SQL Parsing support to extract Lineage
View Lineage support

Pipeline

Capture pipeline status as it happens

Security

Security policies through the UI
Configuration personas and authorization based on policies
AWS SSO support

New Contributors

@cometta made their first contribution in #2916
@Beetelbrox made their first contribution in #3185
@sfirke made their first contribution in #3361

Full Changelog: 0.8.1-release...0.9.1-release

Contributors

cometta, sfirke, and Beetelbrox

Assets 3

10 Mar 19:25

akash-jain-10

0.9.0-release

fa44ed1

OpenMetadata 0.9.0-release

What's Changed

Handle description and tag updates NPE by @harshach in #2856
Generating Typescript Types from Json Schema. by @Sachin-chaurasiya in #2851
Fix #2835, #2836 Lineage entities overlap and Descriptions in Lineage view do not have formatting. by @Sachin-chaurasiya in #2840
Fix #2778 UI : Add Queries tab for table details page. by @Sachin-chaurasiya in #2793
Fix: #2521 - Add owner support for all services and database by @darth-coder00 in #2858
Tableau Lineage Implementation by @ayush-shah in #2850
Table Constraints Added - Ingestion by @ayush-shah in #2854
Add support for default role (#2631) by @mithmatt in #2676
Fix #2837: Add a Feed Count API with entityLink filter support by @vivekratnavel in #2860
Fix #2845 - Init Profiler Workflow by @pmbrull in #2862
Fix access control rule userRoleAttr to point to roleName by @mithmatt in #2644
Support swapping default roles by @mithmatt in #2868
Extend team to support defaultRoles by @mithmatt in #2885
fix: cherry pick for 0.8.3 by @akash-jain-10 in #2893
fix #2869: snowflake-usage processor fixed by @ulixius9 in #2896
Fix #2902 - Check for empty payload by @pmbrull in #2903
Fix #2867: Add support for GET /feed/{id} by @vivekratnavel in #2892
Fix #2894 - Profiler Processor & Metrics by @pmbrull in #2900
Fix : Roles page is showing blank UI when switching to the users tab. by @Sachin-chaurasiya in #2922
fix incorrect query by @cometta in #2916
Fix #2890 Show columns icons in Lineage UI enhancement. by @Sachin-chaurasiya in #2924
Fix #2901: fixed error message by @ulixius9 in #2917
Removed ColumnDataTypeDisplay check with ColumnDataType by @ayush-shah in #2928
Fix #2871: Make search magnifying glass clickable to search by @Sachin-chaurasiya in #2923
Fixes #2899 - Not getting proper relation between parent and child gl… by @sureshms in #2941
Fix #2953: NoopAuthorizer should create a anonymous user by @harshach in #2940
Changing selenium version by @kshinde2512 in #2934
Added Error Handling in Policy tags - Bigquery by @ayush-shah in #2933
Fix #2905: Added MSSQL Usage Connector by @ulixius9 in #2948
Refactor EntityReference list population with id list for a given entity type by @mithmatt in #2956
Fix #2865: Thread should have its own message and posts should only be the replies by @vivekratnavel in #2944
Support returning teams for GET /roles by @mithmatt in #2958
Fixes #2942 - Fix bugs reported by Sonar issue on release 0.9 by @sureshms in #2943
Fix #2878 & #2877 - Implement Metrics and Validate Composed Metrics by @pmbrull in #2926
Bigquery lineage mapping by @codingwithabhi in #2849
Fix #2897 - Profiler CLI by @pmbrull in #2967
Fix #2873: DataQuality test specifications and APIs by @harshach in #2906
fix: Bump up release version 0.8.4 for docker compose by @akash-jain-10 in #2968
Fix #2539: redshift lineage mapping by @ulixius9 in #2828
Adding Webhooks selenium tests by @kshinde2512 in #2911
Assign default roles from Teams to User by @mithmatt in #2947
Fix #2966: JWT filter should compare times in UTC to check the validity by @vivekratnavel in #2971
Update all *.java in openmetadata-ui to use lombok.LOG by @mithmatt in #2973
Use RequiredArgsConstructor in openmetadata-ui/*.java by @mithmatt in #2974
fix #2809: Auto populate View Lineage to Tables by @ulixius9 in #2977
Fix #2965 - Additional Properties set to False by @pmbrull in #2976
Use lombok Getter to simplify openmetadata-ui/*.java files by @mithmatt in #2975
Fix #2981 - Update Profile to match TableProfile by @pmbrull in #2982
Fix #2991: Add ES support for glossary by @harshach in #2993
Fixes #2962 - Clean up fields parameter choices in GET APIs by @sureshms in #2963
Fix #2946: Presto Ingestion issues fix by @ulixius9 in #2986
Fix #2937: added clickhouse usage connector by @ulixius9 in #2987
Fix Okta Authentication and Validation - Ingestion by @ayush-shah in #2955
Fix #2996: ChangeEventHandler throws NPE at times by @vivekratnavel in #2998
Metabase lineage by @codingwithabhi in #3003
Fix #2834: Merge the change events 'delete' and 'add' as 'update' for Activity Feed by @vivekratnavel in #3006
updated chrome drivers and added options for headless chrome by @parthp2107 in #3009
Fix #2807: Merge openmetadata.yaml and openmetadata-server.yaml by @harshach in #2808
Fix #3012 - Dependency breaking Airflow Container by @pmbrull in #3015
Fix: #2938 Remove serviceType from service UI route by @Sachin-chaurasiya in #2939
Feat collaboration by @Sachin-chaurasiya in #2859
Fix #2984: added azure sso auth by @ulixius9 in #3002
Fixes #3021 - Restore a soft-deleted container entity along with the … by @sureshms in #3022
Fixes #3007 - Clean up Sonar flagged issues by @sureshms in #3008
Fix #3023: Keep ElasticSearch version to 7.13 by @harshach in #3024
Power BI connector by @ayush-shah in #3019
Feat : Display a chat icon next to data entities that have threads by @Sachin-chaurasiya in #3026
Atlas connector by @codingwithabhi in #2855
ES index: ignore any errors during the es index by @harshach in #1220
WIP - Selenium fix by @parthp2107 in #3014
Fix #3037: metadata --version doesn't work by @amiorin in #3038
Fix #3040 - py_format recipe includes more paths by @pmbrull in #3043
Fix: #2491 - improvements in google, okta and azure auth by @darth-coder00 in #2929
fix #3035: handled none entity by @ulixius9 in #3036
Add UpdateTeam permission support by @mithmatt in #2994
Fix #3041: Handled AttributeError by @ulixius9 in #3042
Fix: azure auth callback from UI by @darth-coder00 in #3048
Fix #2989: Add delete apis for tests by @harshach in #3020
Fix #3047: OpenMetadata Server fails to run when updating config from Environment variable by @harshach in #3049
Fix #2866: Add support for PATCH to feed API to be able to resolve a thread by @vivekratnavel in #3027
*...