feat(docs): enhance metadata model entity documentation with field tables and SDK examples (#15095)

Author: shirshanka

docs-website/sidebars.js

@@ -1,5 +1,6 @@
 // note: to handle errors where you don't want a markdown file in the sidebar, add it as a comment.
 // this will fix errors like `Error: File not accounted for in sidebar: ...`
+// smoke-test/tests/library_examples/README.md
 module.exports = {
   overviewSidebar: [
     // Getting Started.
@@ -682,14 +683,27 @@ module.exports = {
       defaultStyle: true,
     },
     {
-      "DataHub's Open Metadata Standard": [
-        "docs/modeling/metadata-model",
-        "docs/what/mxe",
+      type: "category",
+      label: "Open Source DataHub Metadata Standard",
+      collapsed: false,
+      items: [
+        {
+          type: "doc",
+          label: "The Metadata Model",
+          id: "docs/modeling/metadata-model",
+        },
         {
-          Entities: [
+          type: "doc",
+          label: "Core Metadata Events",
+          id: "docs/what/mxe",
+        },
+        {
+          type: "category",
+          label: "Entity Docs",
+          items: [
             {
               type: "autogenerated",
-              dirName: "docs/generated/metamodel/entities", // '.' means the current docs folder
+              dirName: "docs/generated/metamodel/entities",
             },
           ],
         },

docs/api/datahub-apis.md

@@ -93,7 +93,7 @@ Here's an overview of what each API can do.
 | Read MLModel                                             | ✅ [[Guide]](/docs/api/tutorials/ml.md#read-mlmodel)                          | ✅ [[Guide]](/docs/api/tutorials/ml.md#read-mlmodel)                                                                                           | ✅      |
 | Read MLModelGroup                                        | ✅ [[Guide]](/docs/api/tutorials/ml.md#read-mlmodelgroup)                     | ✅ [[Guide]](/docs/api/tutorials/ml.md#read-mlmodelgroup)                                                                                      | ✅      |
 | Read MLPrimaryKey                                        | ✅ [[Guide]](/docs/api/tutorials/ml.md#read-mlprimarykey)                     | ✅ [[Guide]](/docs/api/tutorials/ml.md#read-mlprimarykey)                                                                                      | ✅      |
-| Create Data Product                                      | 🚫                                                                            | ✅ [[Code]](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/create_dataproduct.py)                  | ✅      |
+| Create Data Product                                      | 🚫                                                                            | ✅ [[Code]](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/dataproduct_create.py)                  | ✅      |
 | Create Lineage Between Chart and Dashboard               | 🚫                                                                            | ✅ [[Code]](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/lineage_chart_dashboard.py)             | ✅      |
 | Create Lineage Between Dataset and Chart                 | 🚫                                                                            | ✅ [[Code]](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/lineage_dataset_chart.py)               | ✅      |
 | Create Lineage Between Dataset and DataJob               | 🚫                                                                            | ✅ [[Code]](https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/lineage_dataset_job_dataset.py)         | ✅      |

docs/api/tutorials/applications.md

@@ -55,7 +55,7 @@ If you see the following response, the operation was successful:
 <TabItem value="python" label="Python">
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_application.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/application_create_full.py show_path_as_comment }}
 ```
 
 </TabItem>
@@ -146,7 +146,7 @@ If you see the following response, the operation was successful:
 <TabItem value="python" label="Python">
 
 ```python
-{{ inline /metadata-ingestion/examples/library/add_application.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/application_add.py show_path_as_comment }}
 ```
 
 </TabItem>
@@ -189,7 +189,7 @@ Expected Response:
 <TabItem value="python" label="Python">
 
 ```python
-{{ inline /metadata-ingestion/examples/library/remove_application.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/application_remove.py show_path_as_comment }}
 ```
 
 </TabItem>

docs/api/tutorials/assertions.md

@@ -1423,7 +1423,7 @@ If you see the following response, the operation was successful:
 <TabItem value="python" label="Python">
 
 ```python
-{{ inline /metadata-ingestion/examples/library/delete_assertion.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/assertion_delete.py show_path_as_comment }}
 ```
 
 </TabItem>

docs/api/tutorials/container.md

@@ -32,7 +32,7 @@ For detailed steps, please refer to [Datahub Quickstart Guide](/docs/quickstart.
 <TabItem value="python" label="Python" default>
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_container.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/container_create.py show_path_as_comment }}
 ```
 
 </TabItem>

docs/api/tutorials/dashboard-chart.md

@@ -23,35 +23,35 @@ For detailed steps, please refer to [Datahub Quickstart Guide](/docs/quickstart.
 ## Create Chart
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_chart.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/chart_create_simple.py show_path_as_comment }}
 ```
 
 ### Link Chart with Datasets
 
 You can associate datasets with the chart by providing the dataset URN in the `input_datasets` parameter. This will create lineage between the chart and the datasets, so you can track the data sources used by the chart.
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_chart_complex.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/chart_create_complex.py show_path_as_comment }}
 ```
 
 ## Create Dashboard
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_dashboard.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/dashboard_create_simple.py show_path_as_comment }}
 ```
 
 ### Link Dashboard with Charts, Dashboards, and Datasets
 
 You can associate charts, dashboards, and datasets with the dashboard by providing their URNs in the `charts`, `dashboards`, and `input_datasets` parameters, respectively. This will create lineage between the dashboard and the associated entities.
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_dashboard_complex.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/dashboard_create_complex.py show_path_as_comment }}
 ```
 
 ## Read Chart
 
 ```python
-{{ inline /metadata-ingestion/examples/library/read_chart.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/chart_read.py show_path_as_comment }}
 ```
 
 #### Expected Output
@@ -65,7 +65,7 @@ You can associate charts, dashboards, and datasets with the dashboard by providi
 ## Read Dashboard
 
 ```python
-{{ inline /metadata-ingestion/examples/library/read_dashboard.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/dashboard_read.py show_path_as_comment }}
 ```
 
 #### Expected Output

docs/api/tutorials/dataflow-datajob.md

@@ -25,7 +25,7 @@ For detailed steps, please refer to [Datahub Quickstart Guide](/docs/quickstart.
 <TabItem value="python" label="Python" default>
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_dataflow.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/dataflow_create.py show_path_as_comment }}
 ```
 
 </TabItem>
@@ -39,14 +39,14 @@ DataJob must be associated with a DataFlow. You can create a DataJob by providin
 <TabItem value="create-datajob" label="Create DataJob with a DataFlow Object" default>
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_datajob.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/datajob_create_full.py show_path_as_comment }}
 ```
 
 </TabItem>
 <TabItem value="create-datajob-urn" label="Create DataJob with DataFlow URN">
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_datajob_with_flow_urn.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/datajob_create_with_flow_urn.py show_path_as_comment }}
 ```
 
 </TabItem>
@@ -55,7 +55,7 @@ DataJob must be associated with a DataFlow. You can create a DataJob by providin
 ## Read DataFlow
 
 ```python
-{{ inline /metadata-ingestion/examples/library/read_dataflow.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/dataflow_read.py show_path_as_comment }}
 ```
 
 #### Example Output
@@ -69,7 +69,7 @@ DataJob must be associated with a DataFlow. You can create a DataJob by providin
 ## Read DataJob
 
 ```python
-{{ inline /metadata-ingestion/examples/library/read_datajob.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/datajob_read.py show_path_as_comment }}
 ```
 
 #### Example Output

docs/api/tutorials/datasets.md

@@ -109,7 +109,7 @@ Expected Response:
 <TabItem value="python" label="Python" default>
 
 ```python
-{{ inline /metadata-ingestion/examples/library/delete_dataset.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/dataset_delete.py show_path_as_comment }}
 ```
 
 </TabItem>

docs/api/tutorials/domains.md

@@ -64,7 +64,7 @@ Expected Response:
 <TabItem value="python" label="Python">
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_domain.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/domain_create.py show_path_as_comment }}
 ```
 
 </TabItem>
@@ -105,7 +105,7 @@ curl --location --request POST 'http://localhost:8080/api/graphql' \
 <TabItem value="python" label="Python">
 
 ```python
-{{ inline /metadata-ingestion/examples/library/create_nested_domain.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/domain_create_nested.py show_path_as_comment }}
 ```
 
 </TabItem>

docs/api/tutorials/lineage.md

@@ -35,7 +35,7 @@ You can create lineage between two datasets, data jobs, dashboards, or charts. T
 #### Add Entity Lineage Between Datasets
 
 ```python
-{{ inline /metadata-ingestion/examples/library/add_lineage_dataset_to_dataset.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/lineage_dataset_add.py show_path_as_comment }}
 ```
 
 #### Add Entity Lineage Between Datajobs
@@ -98,7 +98,7 @@ Check out more information on how we handle SQL parsing below.
 If you provide a `transformation_text` to `add_lineage`, DataHub will create a query node that represents the transformation logic. This is useful for tracking how data is transformed between datasets.
 
 ```python
-{{ inline /metadata-ingestion/examples/library/add_lineage_dataset_to_dataset_with_query_node.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/lineage_dataset_add_with_query_node.py show_path_as_comment }}
 ```
 
 Transformation text can be any transformation logic, Python scripts, Airflow DAG code, or any other code that describes how the upstream dataset is transformed into the downstream dataset.
@@ -124,15 +124,15 @@ The `get_lineage()` method allows you to retrieve lineage for a given entity.
 This will return the direct upstream entity that the dataset depends on. By default, it retrieves only the immediate upstream entities (1 hop).
 
 ```python
-{{ inline /metadata-ingestion/examples/library/get_lineage_basic.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/lineage_get_basic.py show_path_as_comment }}
 ```
 
 #### Get Downstream Lineage for a Dataset Across Multiple Hops
 
 To get upstream/downstream entities that are more than one hop away, you can use the `max_hops` parameter. This allows you to traverse the lineage graph up to a specified number of hops.
 
 ```python
-{{ inline /metadata-ingestion/examples/library/get_lineage_with_hops.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/lineage_get_with_hops.py show_path_as_comment }}
 
 ```
 
@@ -165,13 +165,13 @@ results = [
 You can retrieve column-level lineage by specifying the `source_column` parameter. This will return lineage paths that include the specified column.
 
 ```python
-{{ inline /metadata-ingestion/examples/library/get_column_lineage.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/lineage_column_get.py show_path_as_comment }}
 ```
 
 You can also pass `SchemaFieldUrn` as the `source_urn` to get column-level lineage.
 
 ```python
-{{ inline /metadata-ingestion/examples/library/get_column_lineage_from_schemafield.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/lineage_column_get_from_schemafield.py show_path_as_comment }}
 
 ```
 
@@ -211,7 +211,7 @@ For more details on how to interpret the results, see [Interpreting Lineage Resu
 You can filter by platform, type, domain, environment, and more.
 
 ```python
-{{ inline /metadata-ingestion/examples/library/get_lineage_with_filter.py show_path_as_comment }}
+{{ inline /metadata-ingestion/examples/library/lineage_get_with_filter.py show_path_as_comment }}
 ```
 
 You can check more details about the available filters in the [Search SDK documentation](./sdk/search_client.md#filter-based-search).