Features/final touches (#16)

* feat: Duckdb compatible

- Added duckdb adapter
- Changes made in sql generator for query compatiblity
- Added details in source

* feat: updated model version compatible to scikit-learn==1.7.1, xgboost==3.0.4

* feat: added fallback for np.float128 as it's not supported by all systems

* added Knowledge Builder Module

* added DataProductBuilder, updated documentation

* DataProductBuilder typo fix

* added macos instructions

* updated readme to correct API KEY

* added dev1

* added httpfs manually for duckdb mac

* updated version to 0.1.2dev2

* added SSL certs for nltk downloads for mac users

* added retry to output parser, removed mcp documentation

* updated documenations

* added virtual env docs

* included all tables

---------

Co-authored-by: juhel-phanju-intugle <juhel@intugle.ai>
Co-authored-by: JaskaranIntugle <jaskaran@intugle.ai>

Author: 3 people

.gitignore

@@ -211,3 +211,4 @@ testing_base
 models
 
 settings.json
+archived/

README.md

@@ -10,45 +10,63 @@
 [![Open Issues](https://img.shields.io/github/issues-raw/Intugle/data-tools)](https://github.com/Intugle/data-tools/issues)
 [![GitHub star chart](https://img.shields.io/github/stars/Intugle/data-tools?style=social)](https://github.com/Intugle/data-tools/stargazers)
 
-*Automated Data Profiling, Link Prediction, and Semantic Layer Generation*
+*Transform Fragmented Data into Connected Semantic Layer*
 
 ## Overview
 
-Intugle provides a set of GenAI-powered Python tools to simplify and accelerate the journey from raw data to insights. This library empowers data and business teams to build an intelligent semantic layer over their data, enabling self-serve analytics and natural language queries. By automating data profiling, link prediction, and SQL generation, Intugle helps you build data products faster and more efficiently than traditional methods.
+Intugle’s GenAI-powered open-source Python library builds an intelligent semantic layer over your existing data systems. At its core, it discovers meaningful links and relationships across data assets — enriching them with profiles, classifications, and business glossaries. With this connected knowledge layer, you can enable semantic search and auto-generate queries to create unified data products, making data integration and exploration faster, more accurate, and far less manual.
 
 ## Who is this for?
 
-This tool is designed for both **data teams** and **business teams**.
-
-*   **Data teams** can use it to automate data profiling, schema discovery, and documentation, significantly accelerating their workflow.
-*   **Business teams** can use it to gain a better understanding of their data and to perform self-service analytics without needing to write complex SQL queries.
+*   **Data Engineers & Architects** often spend weeks manually profiling, classifying, and stitching together fragmented data assets. With Intugle, they can automate this process end-to-end, uncovering meaningful links and relationships to instantly generate a connected semantic layer.
+*   **Data Analysts & Scientists** spend endless hours on data readiness and preparation before they can even start the real analysis. Intugle accelerates this by providing contextual intelligence, automatically generating SQL and reusable data products enriched with relationships and business meaning.
+*   **Business Analysts & Decision Makers** are slowed down by constant dependence on technical teams for answers. Intugle removes this bottleneck by enabling natural language queries and semantic search, giving them trusted insights on demand.
 
 ## Features
 
-*   **Automated Data Profiling:** Generate detailed statistics for each column in your dataset, including distinct count, uniqueness, completeness, and more.
-*   **Datatype Identification:** Automatically identify the data type of each column (e.g., integer, string, datetime).
-*   **Key Identification:** Identify potential primary keys in your tables.
-*   **LLM-Powered Link Prediction:** Use GenAI to automatically discover relationships (foreign keys) between tables.
-*   **Business Glossary Generation:** Generate a business glossary for each column, with support for industry-specific domains.
-*   **Semantic Layer Generation:** Create YAML files that defines your semantic layer, including models (tables) and their relationships.
-*   **SQL Generation:** Generate SQL queries from the semantic layer, allowing you to query your data using business-friendly terms.
+*   **Semantic Intelligence:** Transform raw, fragmented datasets into an intelligent semantic graph that captures entities, relationships, and context — the foundation for connected intelligence.
+*   **Business Glossary & Semantic Search:** Auto-generate a business glossary and enable search that understands meaning, not just keywords — making data more accessible across technical and business users.
+*   **Smart SQL & Data Products:** Instantly generate SQL and reusable data products enriched with context, eliminating manual pipelines and accelerating data-to-insight.
 
 ## Getting Started
 
 ### Installation
 
+Before installing, it is recommended to create a virtual environment:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+```
+
+Then, install the package:
+
 ```bash
 pip install intugle
 ```
 
+#### macOS
+
+For macOS users, you may need to install the `libomp` library:
+
+```bash
+brew install libomp
+```
+
+If you installed Python using the official installer from python.org, you may also need to install SSL certificates by running the following command in your terminal. Please replace `3.XX` with your specific Python version. This step is not necessary if you installed Python using Homebrew.
+
+```bash
+/Applications/Python\ 3.XX/Install\ Certificates.command
+```
+
 ### Configuration
 
 Before running the project, you need to configure a LLM. This is used for tasks like generating business glossaries and predicting links between tables.
 
 You can configure the LLM by setting the following environment variables:
 
 *   `LLM_PROVIDER`: The LLM provider and model to use (e.g., `openai:gpt-3.5-turbo`) following LangChain's [conventions](https://python.langchain.com/docs/integrations/chat/)
-*   `OPENAI_API_KEY`: Your API key for the LLM provider.
+*   `API_KEY`: Your API key for the LLM provider. The exact name of the variable may vary from provider to provider.
 
 Here's an example of how to set these variables in your environment:
 
@@ -59,66 +77,53 @@ export OPENAI_API_KEY="your-openai-api-key"
 
 ## Quickstart
 
-For a detailed, hands-on introduction to the project, please see the [`quickstart.ipynb`](quickstart.ipynb) notebook. It will walk you through the entire process of profiling your data, predicting links, generating a semantic layer, and querying your data.
-
-## Usage
-
-The core workflow of the project involves the following steps:
-
-1.  **Load your data:** Load your data into a DataSet object.
-2.  **Run the analysis pipeline:** Use the `run()` method to profile your data and generate a business glossary.
-3.  **Predict links:** Use the `LinkPredictor` to discover relationships between your tables.
-
-    ```python
-    from intugle import LinkPredictor
+For a detailed, hands-on introduction to the project, please see the [`quickstart.ipynb`](quickstart.ipynb) notebook. It will walk you through the entire process of building a semantic layer, including:
 
-    # Initialize the predictor
-    predictor = LinkPredictor(datasets)
+*   **Building a Knowledge Base:** Use the `KnowledgeBuilder` to automatically profile your data, generate a business glossary, and predict links between tables.
+*   **Accessing Enriched Metadata:** Learn how to access the profiling results and business glossary for each dataset.
+*   **Visualizing Relationships:** Visualize the predicted links between your tables.
+*   **Generating Data Products:** Use the semantic layer to generate data products and retrieve data.
+*   **Serving the Semantic Layer:** Learn how to start the MCP server to interact with your semantic layer using natural language.
 
-    # Run the prediction
-    results = predictor.predict()
-    results.show_graph()
-    ```
-
-5.  **Generate SQL:** Use the `SqlGenerator` to generate SQL queries from the semantic layer.
-
-    ```python
-    from intugle import SqlGenerator
+## Usage
 
-    # Create a SqlGenerator
-    sql_generator = SqlGenerator()
+The core workflow of the project involves using the `KnowledgeBuilder` to build a semantic layer, and then using the `DataProductBuilder` to generate data products from that layer.
 
-    # Create an ETL model
-    etl = {
-        name": "test_etl",
-        fields": [
-           {"id": "patients.first", "name": "first_name"},
-           {"id": "patients.last", "name": "last_name"},
-           {"id": "allergies.start", "name": "start_date"},
-        ,
-        filter": {
-           "selections": [{"id": "claims.departmentid", "values": ["3", "20"]}],
-        ,
-    }
+```python
+from intugle import KnowledgeBuilder, DataProductBuilder
 
-    # Generate the query
-    sql_query = sql_generator.generate_query(etl_model)
-    print(sql_query)
-    ```
+# Define your datasets
+datasets = {
+    "allergies": {"path": "path/to/allergies.csv", "type": "csv"},
+    "patients": {"path": "path/to/patients.csv", "type": "csv"},
+    # ... add other datasets
+}
 
-For detailed code examples and a complete walkthrough, please refer to the [`quickstart.ipynb`](quickstart.ipynb) notebook.
+# Build the knowledge base
+kb = KnowledgeBuilder(datasets, domain="Healthcare")
+kb.build()
 
-### MCP Server
+# Create a DataProductBuilder
+dp_builder = DataProductBuilder()
 
-This tool also includes an MCP server that exposes your semantic layer as a set of tools that can be used by an LLM client. This enables you to interact with your semantic layer using natural language to generate SQL queries, discover data, and more.
+# Define an ETL model
+etl = {
+    "name": "patient_allergies",
+    "fields": [
+        {"id": "patients.first", "name": "first_name"},
+        {"id": "patients.last", "name": "last_name"},
+        {"id": "allergies.description", "name": "allergy"},
+    ],
+}
 
-To start the MCP server, run the following command:
+# Generate the data product
+data_product = dp_builder.build(etl)
 
-```bash
-intugle-mcp
+# View the data product as a DataFrame
+print(data_product.to_df())
 ```
 
-You can then connect to the server from any MCP client, such as Claude Desktop or Gemini CLI, at `http://localhost:8000/semantic_layer/mcp`.
+For detailed code examples and a complete walkthrough, please refer to the [`quickstart.ipynb`](quickstart.ipynb) notebook.
 
 ## Contributing