KMDS refactoring.

Author: rajivsam

.github/workflows/ci.yml

@@ -0,0 +1,62 @@
+name: CI
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11"]
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install uv
+      uses: astral-sh/setup-uv@v3
+      with:
+        version: "latest"
+    - name: Load cached venv
+      id: cached-uv-dependencies
+      uses: actions/cache@v3
+      with:
+        path: .venv
+        key: venv-${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('**/uv.lock') }}
+    - name: Install dependencies
+      run: uv sync --no-install-project
+    - name: Run tests
+      run: uv run pytest
+
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: "3.10"
+    - name: Install uv
+      uses: astral-sh/setup-uv@v3
+      with:
+        version: "latest"
+    - name: Load cached venv
+      id: cached-uv-dependencies
+      uses: actions/cache@v3
+      with:
+        path: .venv
+        key: venv-${{ runner.os }}-3.10-${{ hashFiles('**/uv.lock') }}
+    - name: Install dependencies
+      run: uv sync
+    - name: Build package
+      run: uv build
+    - name: Upload build artifacts
+      uses: actions/upload-artifact@v3
+      with:
+        name: dist
+        path: dist/

.gitignore

@@ -62,4 +62,6 @@ _solr/schema.xml
 _src/*
 local/*
 .env
+.venv/
+.pytest_cache/
 

Changelog.md

@@ -7,3 +7,8 @@
 * Wiki page on design perspective added
 * Examples for Minio based file read and write added. Docker files added.
 * Examples of generating semantic meta-data for clean data represenations that are used for modelling is illustrated with woodwork.
+
+[0.0.3.0]
+* Removed cloud features and packages related to documenting meta-data for datasets, these can be captured by code generation tools now
+* Refactored examples to illustrate the use of generative AI in data science and or data analysis projects
+* Created better example documentation using generative AI tools.

README.md

@@ -11,15 +11,11 @@
 
 </div>
 
-### What is this tool used for?
+### What is the tool and why do you need it?
 
-This tool is used for knowledge management in data science. In software design, conceptual domain models inform lower-level models. In data science work, experiments yield knowledge that informs modeling choices. Data science models are almost always informed by a variety of analysis experiments. Experimentation and organization of knowledge artifacts supporting modeling decisions is a requirement for reproducible data science models and improving model quality and performance. ([video](https://www.youtube.com/watch?v=ckr8YQJxF9I)).
+This tool is used for knowledge management in data science. As data scientists, incremental experimentation is a way of life. The problem is we have a lot of them and even small projects accumalate context, decisions and rationale over time. This is not a problem if we have both the need for experimentation (the design question or issue) and the results documented over time, but this tends to be done in an adhoc manner, so when its time to rebuild or revisit a particular question, we can't find the research and the results related to it. This is the need this tool fulfils.
 
-Please see [knowledge application development context](https://github.com/rajivsam/KMDS/blob/main/feature_documentation/knowledge_management_in_DS.md) for a description of a typical knowledge application development setting.
-
-### Why do you need this tool?
-
-The above narrative suggests that ability to retrieve knowledge about experiments and historical models in an adhoc manner is critical in data science. It is. It is also grossly underserved. Knowledge management tools for domain specific models exist, knowledge management tools for dev-ops and ML-Ops exist, but tools for analytics and model development are siloed. Information gets fragmented over time. So analysts and data scientists often have to go to experiment tools, data catalogs or ML-Ops tools to fetch information they need to develop a model. In a subsequent iteration of this model, the contextual information that informed the development of this model is often lost, and the development team, possibly with new team members, have the task of reconstructing this contextual information again. This library is a step in fixing this problem. The central idea is to organize tasks in terms of a sequence of steps that are pretty standard in data analysis work and capture knowledge in context while these tasks are performed.
+Please see [knowledge application development context](https://github.com/rajivsam/KMDS/blob/main/feature_documentation/knowledge_management_in_DS.md) for a description of a typical knowledge application development setting. Please see [the video](example_documentation/video/Knowledge_Management_for_Data_Science_comp.mp4) for a quick overview.
 
 ### How is it related to process guidelines and vocabularies for machine learning?
 Initiatives such as [CRISP DM](https://www.datascience-pm.com/crisp-dm-2/) provide guidelines and processes for developing data science projects. Projects such as [Open ML](https://openml.github.io/openml-python/main/index.html) provide semantic vocabulary standardization for machine learning tasks. These are excellent tools. However, the guidelines they provide are task focussed. The gap between a conceptual idea and the final, or, even candidate data science tasks for a project is filled with many assumptions and experimental evaluations. The information in these assumptions and experimental evaluations is what this tool aims to capture. There is also an ordering to these assumptions and experimental evaluations. This is also what this tool aims to capture.
@@ -29,20 +25,16 @@ Initiatives such as [CRISP DM](https://www.datascience-pm.com/crisp-dm-2/) provi
 This tool is for data scientists and data analysts.
 
 ### How do you use this tool?
+This version of the tool takes all the recent advances (as of early 2026) into consideration in how this tool is used. This is a python package. It is assumed that you have an API key to a provider. The basic usage scenario is as follows:
+1. Install the python package in the environment where you intend to experiment and do your data science analysis.
+2. Work through your analysis plan for your model development or experiment. The tool will not offer help with how your analysis or experiment will be done. It assumes you are the expert and you know how to do this. Of course, you can use __Jupyternaut__ or a similar generative AI tool to generate your code for you.
+3. As you work through your exploratory data analysis, data representation and modeling phases, log your findings to ```kmds```
+4. Run a report to fetch the details of your design rationale as needed. To communicate your findings to your team or management, simply export your knowledge base. Point a generative AI tool such as __NotebookLM__ to the export and generate your report, video or other documentation artifact.
 
-1. You install this library along with the other python dependecies you need for your analysis task
-2. Review the [basic recipe](https://github.com/rajivsam/KMDS/blob/main/examples_of_use/workflow_recipe.md) for capturing your observations.
-3. Review [the templates section](https://github.com/rajivsam/KMDS/blob/main/examples_of_use/workflow_recipe.md) to find the example relevant to you. For analytics projects, review the analytics template. For machine learning projects, review the machine learning template.
-4. Start using the tool in your projects using the information from your review of the above two steps.
-
-
-_Note:_ 
-1. The examples are based on using the files in the package, but it is quite straight forward to connect to S3 storage to get your data files, see [the connection notes document](https://github.com/rajivsam/KMDS/blob/main/examples_of_use/connection_notes.md) for details. Minio provides a sandbox where you can try this
-   
-2. _Please the [wiki pages](https://github.com/rajivsam/KMDS/wiki/KMDS-Design-Perspectives) section of the repository for design perspectives and documentation_. This is work in progress.
-
-
+### Examples of use
+The repository contains two examples of use. One example is from analytics, the other is from machine learning. The notebooks for analytics are in [the analytics example](examples_of_use/analytics) and the notebooks for machine learning are in [the machine learning example](examples_of_use/machine_learning). The analytics example evaluates the effectiveness of ticket resolution help desk. Using ticket resolution data for a particular quarter, Q2 2016, the example illustrates how effectiveness of the organization can be evaluated. The reader can explore the notebooks to see the details of the implementation and details of how findings in each phase of the model building cycle are logged. The findings from the resulting knowledge base can be exported to create materials to communicate the details of the project to team members and management, see [this video](examples_of_use/analytics/Help_Desk_Analytics%20_comp.mp4) and this [infographic](examples_of_use/analytics/usecase_overview_mindmap.png)
 
+The machine learning example illustrates how Principal Components Analysis can be used to summarize the sales activity in an online store for a particular quarter. The reader can view the notebooks under [the machine learning example](examples_of_use/machine_learning) for details of the implementation. As with the analytics example, generative AI tools (Notebook LM in this case) can be used to communicate the findings and results from the knowledge base, see [this infographic](examples_of_use/machine_learning/ml_infographic_kmds.png).
 
 ### Licensing and Feature Questions
 

build_instructions/build_instructions.md

@@ -1,5 +1,5 @@
 ### Building KMDS
-This package is built with poetry. Install poetry, then run ``` poetry build```
+This package is built with poetry. Install poetry, then run ``` uv pip build ```
 
 ### Updating Documentation
 This is the sequence of steps to follow to make changes to the documentation.

design_considerations/Knowledge_Mgmt_for_Data_Science_comp.mp4

@@ -0,0 +1,4 @@
+1. The context in which development is done for most enterprise projects is provided in [this document](../feature_documentation/knowledge_management_in_DS.md  
+2. A basic recipe for capturing observations is provided in [this document](../feature_documentation/km_app_pipeline.md
+3. The need to use an ontology for knowledge capture is provided in [this document](../feature_documentation/ontology_management.md)
+4. A glossary of observation types is provided in [this document](../feature_documentation/glossary_observation_types.md)