Feature/multilp reltype (#124)

* composite key identification added

* added test cases

* moved distinct count to connector level

* lint

* storing distinct count

* added composite key cache to store distinct_count

* added langfuse observability

* implement multi lp

* tests

* refactored predicted links

* test cases pass

* added asserts for accuracy and intersect ratios

* added date for lp, connectors implementation, rate limiter

* add basketball dataset

* mcp server get schema refactored

* dataproduct refactor

* added uniqueness ratio

* Link type identification added to data product

* fix relationship type order mismatch bug

* rc1

* databricks refactor

* composite key graph visualization

* sqlserver optimized queries

* snowflake fixes

* graph update

* added legend. upgraded version

* added docs

* updated composite edge colour in streamlit

* added example notebook

* removed unnecessary print statements

* linting

* updated cardinality test case

Author: raphael-intugle

README.md

@@ -110,11 +110,13 @@ For a detailed, hands-on introduction to the project, please see our quickstart
 | **Native Databricks with AI/BI Genie [ Tech Manufacturing ]** | [`quickstart_native_databricks.ipynb`](notebooks/quickstart_native_databricks.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Intugle/data-tools/blob/main/notebooks/quickstart_native_databricks.ipynb) |
 | **Streamlit App**       | [`quickstart_streamlit.ipynb`](notebooks/quickstart_streamlit.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Intugle/data-tools/blob/main/notebooks/quickstart_streamlit.ipynb) |
 | **Conceptual Search**   | [`quickstart_conceptual_search.ipynb`](notebooks/quickstart_conceptual_search.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Intugle/data-tools/blob/main/notebooks/quickstart_conceptual_search.ipynb) |
+| **Composite Relationships Prediction**   | [`quickstart_basketball_composite_links.ipynb`](notebooks/quickstart_basketball_composite_links.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Intugle/data-tools/blob/main/notebooks/quickstart_basketball_composite_links.ipynb) |
+
 These datasets will take you through the following steps:
 
 *   **Generate Semantic Model** → The unified layer that transforms fragmented datasets, creating the foundation for connected intelligence.
     *   **1.1 Profile and classify data** → Analyze your data sources to understand their structure, data types, and other characteristics.
-    *   **1.2 Discover links & relationships among data** → Reveal meaningful connections (PK & FK) across fragmented tables.
+    *   **1.2 Discover links & relationships among data** → Reveal meaningful connections (PK & FK), including composite keys, across fragmented tables.
     *   **1.3 Generate a business glossary** → Create business-friendly terms and use them to query data with context.
     *   **1.4 Enable semantic search** → Intelligent search that understands meaning, not just keywords—making data more accessible across both technical and business users.
     *   **1.5 Visualize semantic model**→ Get access to enriched metadata of the semantic layer in the form of YAML files and visualize in the form of graph

docsite/docs/core-concepts/semantic-intelligence/dataset.md

@@ -39,7 +39,10 @@ The library organizes metadata using Pydantic models, but you can access it thro
 -   **Table-Level Metadata**: Accessed via `dataset.source.table`.
     -   `.name: str`
     -   `.description: str`
-    -   `.key: Optional[str]`
+    -   `.key: Optional[PrimaryKey]`
+-   **`PrimaryKey`**: Defines the primary key for a table.
+    -   `.columns: List[str]` (A list of column names that make up the primary key.)
+    -   `.distinct_count: Optional[int]` (The number of distinct values in the primary key column(s).)
 -   **Column-Level Metadata**: Accessed via the `dataset.columns` dictionary, where keys are column names.
     -   `[column_name].description: Optional[str]`
     -   `[column_name].type: Optional[str]` (for example, 'integer', 'date')
@@ -68,7 +71,7 @@ print(f"Schema: {customers_dataset.source.schema}")
 # Access table-level metadata
 print(f"Table Name: {customers_dataset.source.table.name}")
 print(f"Table Description: {customers_dataset.source.table.description}")
-print(f"Primary Key: {customers_dataset.source.table.key}")
+print(f"Primary Key Columns: {customers_dataset.source.table.key.columns}")
 
 # Access column-level metadata using the 'columns' dictionary
 email_column = customers_dataset.columns['email']

docsite/docs/core-concepts/semantic-intelligence/link-prediction.md

@@ -5,7 +5,7 @@ title: Link Prediction
 
 # Link prediction
 
-Link prediction is one of the most powerful features of the Intugle Data Tools library. It's the process of automatically discovering meaningful relationships and potential join keys between different, isolated datasets. This turns a collection of separate tables into a connected semantic graph, which is the foundation for building unified data products.
+Link prediction is one of the most powerful features of the Intugle Data Tools library. It's the process of automatically discovering meaningful relationships and potential join keys between different, isolated datasets. This turns a collection of separate tables into a connected semantic graph, which is the foundation for building unified data products. The library now supports the prediction of **composite key** relationships, where multiple columns together form a link between tables.
 
 ## The LinkPredictor class
 
@@ -28,7 +28,7 @@ links_list = predictor_instance.links
 To use the `LinkPredictor` manually, you must give it a list of fully profiled `DataSet` objects.
 
 ```python
-from intugle.analysis.models import DataSet,
+from intugle.analysis.models import DataSet
 from intugle.link_predictor.predictor import LinkPredictor
 
 
@@ -52,7 +52,7 @@ predictor.predict(save=True)
 # The discovered links are stored as a list of PredictedLink objects in the `links` attribute
 links_list = predictor.links
 for link in links_list:
-    print(f"Found link from {link.from_dataset}.{link.from_column} to {link.to_dataset}.{link.to_column}")
+    print(f"Found link from {link.from_dataset}.{link.from_columns} to {link.to_dataset}.{link.to_columns}")
 ```
 
 ### Caching mechanism
@@ -74,7 +74,7 @@ A utility function that converts the `links` list into a Pandas DataFrame. This
 links_df = predictor.get_links_df()
 
 # Display the DataFrame
-# columns: from_dataset, from_column, to_dataset, to_column
+# columns: from_dataset, from_columns, to_dataset, to_columns
 print(links_df)
 ```
 

docsite/docs/examples.md

@@ -19,6 +19,7 @@ For a detailed, hands-on introduction to the project, please see our quickstart
 | **Native Snowflake with Cortex Analyst [ Tech Manufacturing ]** | [`quickstart_native_snowflake.ipynb`](https://github.com/Intugle/data-tools/blob/main/notebooks/quickstart_native_snowflake.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Intugle/data-tools/blob/main/notebooks/quickstart_native_snowflake.ipynb) |
 | **Native Databricks with AI/BI Genie [ Tech Manufacturing ]** | [`quickstart_native_databricks.ipynb`](https://github.com/Intugle/data-tools/blob/main/notebooks/quickstart_native_databricks.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Intugle/data-tools/blob/main/notebooks/quickstart_native_databricks.ipynb) |
 | **Streamlit App**       | [`quickstart_streamlit.ipynb`](https://github.com/Intugle/data-tools/blob/main/notebooks/quickstart_streamlit.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Intugle/data-tools/blob/main/notebooks/quickstart_streamlit.ipynb) |
+| **Composite Relationships Prediction**   | [`quickstart_basketball_composite_links.ipynb`](https://github.com/Intugle/data-tools/blob/main/notebooks/quickstart_basketball_composite_links.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Intugle/data-tools/blob/main/notebooks/quickstart_basketball_composite_links.ipynb) |
 
 These datasets will take you through the following steps: