Local llm #83

mahdibayouli · 2025-07-20T21:33:54Z

✅ Enable Drop-in Local LLM (LM Studio) Support & Robust Test Isolation

✨ Summary

This PR adds first-class LM Studio support as a drop-in OpenAI replacement and overhauls the test suite for fully isolated, fast testing. All changes are backwards-compatible with OpenAI cloud.

🚀 Features

🔁 Drop-in LLM Support

Switch via .env:

LLM_PROVIDER=lmstudio
OPENAI_API_BASE=http://host.docker.internal:1234/v1
OPENAI_API_KEY=local-key
OPENAI_MODEL=qwen1.5-7b-chat

No new env vars needed
Works with OpenAI-compatible models (e.g. qwen1.5-7b-chat)

📦 Structured Output Fallback

OpenAI-style structured output fallback:
- openai_parser → function_calling → strict_json_prompt

🧠 Local Embeddings

Uses langchain_huggingface + sentence_transformers
All dependencies included in Docker images

🧪 Test Suite Overhaul

Fully mocked external services (LLM, Weaviate, etc.)
In-process FastAPI tests (no network calls)
Centralized config via pytest.ini
Test strategy documented in tests/README.md

🛠️ Changed Files (Highlights)

File	Description
`llm_service.py`	Adds robust fallback for structured output
`embedder_service.py`	Switches to local HuggingFace embeddings
`requirements.txt`, `Dockerfile`	Ensures all deps for local LLM/embedding
`tests/`, `pytest.ini`	New isolated test infra
`.env.example`, `README.md`	Updated usage guides and configs

⚙️ Usage

🖥️ LM Studio (Local)

Run LM Studio.
Set .env as shown above.
Rebuild & restart GenAI container.

☁️ OpenAI (Cloud)

Use regular OpenAI env settings.

🧪 Run Tests

pytest -q

📝 Notes

Switching embedding provider may require Weaviate schema reset.
Structured output with LM Studio may have some limitations (see README).

…ider logic

Copilot

Pull Request Overview

This PR adds drop-in local LLM support via LM Studio as an OpenAI-compatible alternative and implements comprehensive test isolation infrastructure. The changes maintain full backward compatibility with OpenAI cloud services while enabling local development workflows.

Key changes include:

LM Studio integration with robust structured output fallback mechanisms
Local HuggingFace embeddings as an alternative to OpenAI embeddings
Comprehensive test suite with mocked external dependencies for fast, isolated testing

Reviewed Changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`llm_service.py`	Adds LM Studio provider support and multi-tier structured output fallback
`embedder_service.py`	Implements provider-aware embedding selection (OpenAI vs HuggingFace)
`requirements.txt`	Adds langchain_huggingface dependency for local embeddings
`Dockerfile`	Installs sentence_transformers and langchain-huggingface packages
`pytest.ini`	Configures test discovery and exclusion patterns
`genai/tests/*`	New isolated test suite with comprehensive mocking
`.env.example`	Updates configuration examples for LM Studio setup
`README.md`	Adds detailed LM Studio setup and usage documentation

Copilot · 2025-07-20T21:35:26Z

genai/src/services/llm/llm_service.py

+            openai_api_base=os.getenv("OPENAI_API_BASE", "http://127.0.0.1:1234/v1"),
+        )
+
    if provider in ("openai", "llmstudio", "local"):


The conditional check includes 'llmstudio' but this case is already handled by the previous condition on line 25. This creates unreachable code for the 'llmstudio' case in this block.

Suggested change

if provider in ("openai", "llmstudio", "local"):

if provider in ("openai", "local"):

Copilot · 2025-07-20T21:35:27Z

genai/tests/test_embedder.py

+            assert client is not None
+            return object()
+
+    monkeypatch.setenv("LLM_PROVIDER", "local")


This line duplicates the setenv call from line 7. The duplicate environment variable setting is unnecessary.

Suggested change

monkeypatch.setenv("LLM_PROVIDER", "local")

Copilot · 2025-07-20T21:35:27Z

genai/tests/test_api.py

@@ -0,0 +1,42 @@
+# tests/conftest.py


The file comment indicates this is 'tests/conftest.py' but the actual file is 'test_api.py'. This misleading comment should be corrected.

Suggested change

# tests/conftest.py

# genai/tests/test_api.py - Test cases and fixtures for API testing

Copilot · 2025-07-20T21:35:28Z

genai/Dockerfile

+# Install curl & the new embeddings lib
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends curl \
+    && pip install --no-cache-dir sentence_transformers langchain-huggingface \


Installing packages directly in the runtime stage bypasses dependency management. Consider adding these to requirements.txt and installing via the existing pip install command for consistency.

Copilot · 2025-07-20T21:35:28Z

genai/src/services/llm/llm_service.py

+        }
+
+        resp = client.chat.completions.create(
+            model=os.getenv("OPENAI_MODEL"),


The model parameter could be None if OPENAI_MODEL environment variable is not set, which may cause the OpenAI client to fail. Consider providing a default value.

Suggested change

model=os.getenv("OPENAI_MODEL"),

model=os.getenv("OPENAI_MODEL", "gpt-4o-mini"),

Copilot · 2025-07-20T21:35:28Z

genai/src/services/embedding/embedder_service.py

+
+    if provider == "openai":
+        embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")
+    else:  # lmstudio → local HF model


The comment suggests this path is only for 'lmstudio' but the condition catches any non-openai provider. Consider making the condition more explicit or updating the comment to reflect the broader scope.

Suggested change

else: # lmstudio → local HF model

else: # Handles all non-"openai" providers (e.g., lmstudio → local HF model)

…iate dependencies

GravityDarkLab

tests works - tested with act - can skip piplines as they taking long

mahdibayouli added 14 commits July 20, 2025 23:31

Add pytest.ini to centralize test discovery and config

9d5de8e

Update .env.example for LM Studio and OpenAI provider clarity

e0980c5

Add langchain_huggingface to requirements for local embeddings

56bbbb6

Document local LM Studio setup and clarify model instructions

517423a

Install sentence_transformers and langchain-huggingface in runtime image

fcdb134

Add empty __init__.py to mark tests as a package

3e335ae

Stub external services and set test env defaults in conftest

4971618

Clarify test coverage, mocking strategy, and usage in README

311500c

Fully mock app startup and add FastAPI integration tests

5d27cac

Mock all dependencies and assert course post-processing

a5d5b41

Refactor crawler test to use asyncio.run and mock HTTP

c1432cb

Fully mock embedder dependencies and test chunking logic

e663d6a

Add robust OpenAI-compatible function-calling for structured output

468cb16

Switch to langchain_huggingface for local embeddings and improve prov…

a29c042

…ider logic

mahdibayouli requested review from GravityDarkLab and Copilot July 20, 2025 21:34

mahdibayouli self-assigned this Jul 20, 2025

Copilot AI reviewed Jul 20, 2025

View reviewed changes

chore: add conda environment config with FastAPI, LangChain, and Weav…

12b7267

…iate dependencies

GravityDarkLab approved these changes Jul 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Local llm #83

Local llm #83

Uh oh!

mahdibayouli commented Jul 20, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 20, 2025

Uh oh!

Copilot AI Jul 20, 2025

Uh oh!

Copilot AI Jul 20, 2025

Uh oh!

Copilot AI Jul 20, 2025

Uh oh!

Copilot AI Jul 20, 2025

Uh oh!

Copilot AI Jul 20, 2025

Uh oh!

GravityDarkLab left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	if provider in ("openai", "llmstudio", "local"):
	if provider in ("openai", "local"):

	# tests/conftest.py
	# genai/tests/test_api.py - Test cases and fixtures for API testing

	model=os.getenv("OPENAI_MODEL"),
	model=os.getenv("OPENAI_MODEL", "gpt-4o-mini"),

	else: # lmstudio → local HF model
	else: # Handles all non-"openai" providers (e.g., lmstudio → local HF model)

Local llm #83

Are you sure you want to change the base?

Local llm #83

Uh oh!

Conversation

mahdibayouli commented Jul 20, 2025

✅ Enable Drop-in Local LLM (LM Studio) Support & Robust Test Isolation

✨ Summary

🚀 Features

🔁 Drop-in LLM Support

📦 Structured Output Fallback

🧠 Local Embeddings

🧪 Test Suite Overhaul

🛠️ Changed Files (Highlights)

⚙️ Usage

🖥️ LM Studio (Local)

☁️ OpenAI (Cloud)

🧪 Run Tests

📝 Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 20, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 20, 2025

Choose a reason for hiding this comment

Uh oh!

GravityDarkLab left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants