Skip to content

Commit b435635

Browse files
sebuzdugansebuzdugan
committed
fix: packaging issue
1 parent 507db87 commit b435635

File tree

17 files changed

+2312
-860
lines changed

17 files changed

+2312
-860
lines changed

‎README.md‎

Lines changed: 47 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,20 @@
1-
# FRAI - Framework of Responsible Artificial Intelligence
1+
███████████ ███████████ █████████ █████
2+
░░███░░░░░░█░░███░░░░░███ ███░░░░░███ ░░███
3+
░███ █ ░ ░███ ░███ ░███ ░███ ░███
4+
░███████ ░██████████ ░███████████ ░███
5+
░███░░░█ ░███░░░░░███ ░███░░░░░███ ░███
6+
░███ ░ ░███ ░███ ░███ ░███ ░███
7+
█████ █████ █████ █████ █████ █████
8+
░░░░░ ░░░░░ ░░░░░ ░░░░░ ░░░░░ ░░░░░
9+
10+
11+
# FRAI · Framework of Responsible Artificial Intelligence
212

313
![npm version](https://img.shields.io/npm/v/frai)
414

5-
FRAI is a responsible AI compliance framework that generates comprehensive documentation in under 3 minutes. It guides you through a progressive, context-aware questionnaire and produces:
6-
- Implementation Checklist
7-
- Model Card
8-
- Risk & Compliance File
15+
FRAI (Framework of Responsible Artificial Intelligence) is an open-source toolkit that helps any team — from solo developers to compliance officers — ship AI features responsibly. It walks you through quick questions, scans your code, and generates documentation you can actually hand to stakeholders: implementation checklists, model cards, risk files, evaluation reports, and policy-aware RAG indexes.
16+
17+
Think of FRAI as a safety net for AI launches: it collects the right facts, highlights blind spots, and keeps evidence tidy so production reviews stop feeling like guesswork.
918

1019
## 🚀 Getting Started
1120

@@ -25,6 +34,8 @@ You only need to do this once per machine or project. Your key is stored securel
2534
frai
2635
```
2736

37+
> **Tip:** `frai` is available globally after installing the published npm package. If you're working from a local clone, follow the steps in [Local Development](#-local-development) to run the CLI from source.
38+
2839
---
2940

3041
## 🛠️ CLI Commands & Features
@@ -102,6 +113,37 @@ If you skip this step, FRAI will prompt you to set up your key on first use.
102113

103114
---
104115

116+
## 🧑‍💻 Local Development
117+
118+
Run FRAI directly from this repository without publishing:
119+
120+
```bash
121+
pnpm install
122+
pnpm --filter frai run build
123+
node packages/frai-cli/dist/index.js --help
124+
```
125+
126+
### Configure an OpenAI key from source
127+
128+
- Interactive CLI:
129+
```bash
130+
node packages/frai-cli/dist/index.js --setup YOUR_KEY
131+
```
132+
- Manual `.env`:
133+
```
134+
OPENAI_API_KEY=YOUR_KEY
135+
```
136+
137+
To test the global binary locally (without npm publishing), install the workspace package:
138+
139+
```bash
140+
pnpm install --global ./packages/frai-cli
141+
# then:
142+
frai --setup
143+
```
144+
145+
---
146+
105147
## 📖 Learn More
106148
- [GitHub Repository](https://github.com/sebastianbuzdugan/frai)
107149
- [NPM Package](https://www.npmjs.com/package/frai)

‎TODO.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
- ✅ **Day 9:** Implement `frai rag index` command, vector store abstraction, and RAG helper modules; cover with CLI tests.
2929
- ✅ **Day 10:** Scaffold `frai eval` baseline metrics leveraging open-source models; ensure configurable thresholds and reporting.
3030
- ✅ **Day 11:** Implement AI provider abstraction (`frai-core/src/providers`) supporting OpenAI + future adapters with comprehensive test coverage.
31-
- **Day 12:** Draft fine-tuning governance spec (dataset audit, training hooks, bias evals); prototype config schema.
31+
- ✅ **Day 12:** Draft fine-tuning governance spec (dataset audit, training hooks, bias evals); prototype config schema.
3232
- **Day 13:** Develop VS Code MCP endpoint + extension scaffold reading data from `frai-core`; document protocol usage.
3333
- **Day 14:** Author problem-solution playbooks (RAG compliance, LLM agent docs) in `/docs`; tie features to user pain points.
3434
- **Day 15:** Add opt-in telemetry, gather KPIs (usage of new commands/tests), and produce sprint retrospective summary.

‎docs/finetune-governance.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
# Fine-Tuning Governance Specification
2+
3+
This document captures the governance schema and readiness checks implemented in `frai-core` for the upcoming `frai-finetune` toolkit. The goal is to make fine-tuning workflows auditable, bias-aware, and production-safe by default.
4+
5+
## Schema Overview
6+
7+
The governance plan is represented as a JSON object with the following top-level sections:
8+
9+
| Section | Purpose | Key Fields |
10+
| --- | --- | --- |
11+
| `dataset` | Capture provenance, consent, and quality of the fine-tuning corpus. | `name`, `useCases`, `sources[]`, `sensitivity`, `retention`, `qualityChecks` |
12+
| `training` | Describe the base model, objectives, and safety hooks. | `objective`, `baseModel`, `targetPersona`, `callbacks[]`, `safety.guardrails[]` |
13+
| `evaluation` | Define regression suites and launch gates. | `datasets[]`, `metrics[]`, `gateCriteria.minimumSuccessRate`, `gateCriteria.riskTolerance` |
14+
| `approvals` | Track stakeholders and sign-off status. | `stakeholders[].role`, `stakeholders[].name`, `stakeholders[].status` |
15+
| `monitoring` | Prepare live telemetry and rollback plan. | `owner`, `metrics[]`, `telemetry`, `rollbackPlan.trigger`, `rollbackPlan.steps[]` |
16+
| `audit` | Archive artefacts for compliance. | `artefacts[]`, `notes` |
17+
18+
### Enumerations
19+
- `dataset.sensitivity.level`: `none`, `internal`, `confidential`, `restricted`
20+
- `evaluation.gateCriteria.riskTolerance`: `low`, `medium`, `high`, `critical`
21+
- `approvals.stakeholders[].status`: `pending`, `approved`, `rejected`
22+
23+
## Validation Rules
24+
25+
`frai-core/src/finetune/index.js` exposes `validateGovernancePlan(plan)` which returns `{ valid, errors }`. Key checks include:
26+
- Required sections must be present and use the prescribed shape.
27+
- Dataset sources require `name`, `type`, `access`, `owner`.
28+
- Evaluation metrics require `name`, `direction`, and numeric `threshold`.
29+
- Monitoring rollback steps must be non-empty strings.
30+
- Stakeholder approvals must use an allowed status value.
31+
32+
## Readiness Scoring
33+
34+
`calculateReadiness(plan)` evaluates a plan across five checkpoints:
35+
36+
1. `dataset` – dataset use-cases, sources, and quality checks populated.
37+
2. `evaluation` – metrics with thresholds and gate criteria.
38+
3. `approvals` – every stakeholder marked as `approved`.
39+
4. `monitoring` – named owner plus metrics and rollback steps.
40+
5. `audit` – artefact references stored.
41+
42+
The function returns `{ status, score, checkpoints }` where:
43+
- `status` is `ready`, `pending`, or `blocked` (`blocked` if approvals or evaluation are incomplete).
44+
- `score` is the proportion of satisfied checkpoints.
45+
- `checkpoints` details completion state per checkpoint.
46+
47+
`summarizeGovernance(plan)` provides a human-readable summary suitable for CLI output or reports.
48+
49+
## Example Configuration
50+
51+
```json
52+
{
53+
"dataset": {
54+
"name": "Customer Support Conversations",
55+
"description": "Anonymised support dialogues from Q1 2024.",
56+
"useCases": ["Assist agents in answering live chats"],
57+
"sources": [
58+
{
59+
"name": "Zendesk export",
60+
"type": "csv",
61+
"access": "s3://frai-data/chat-trimmed.csv",
62+
"owner": "Support Ops"
63+
}
64+
],
65+
"sensitivity": {
66+
"level": "confidential",
67+
"piiPresent": false,
68+
"mitigation": "PII scrubbed using internal tool"
69+
},
70+
"retention": {
71+
"policy": "Delete after 12 months",
72+
"reviewAt": "2025-06-01"
73+
},
74+
"qualityChecks": {
75+
"biasAssessment": "No skew observed across demographics",
76+
"dataBalanceSummary": "Balanced between positive/negative sentiment",
77+
"manualReview": "Sampled 200 records for offensive content"
78+
}
79+
},
80+
"training": {
81+
"objective": "Improve suggestion latency by 20%",
82+
"baseModel": "gpt-3.5-turbo",
83+
"targetPersona": "Customer support agent",
84+
"callbacks": ["biasAuditHook", "guardrailValidationHook"],
85+
"safety": {
86+
"guardrails": ["toxicity-filter", "pii-detector"],
87+
"escalationContacts": ["responsible-ai@frai.dev"]
88+
}
89+
},
90+
"evaluation": {
91+
"datasets": [
92+
{
93+
"name": "Zendesk hold-outs",
94+
"description": "10% holdout set for regression checks"
95+
}
96+
],
97+
"metrics": [
98+
{
99+
"name": "Exact match",
100+
"direction": "increase",
101+
"threshold": 0.66
102+
},
103+
{
104+
"name": "Toxicity rate",
105+
"direction": "decrease",
106+
"threshold": 0.02
107+
}
108+
],
109+
"gateCriteria": {
110+
"minimumSuccessRate": 0.65,
111+
"riskTolerance": "medium"
112+
}
113+
},
114+
"approvals": {
115+
"stakeholders": [
116+
{
117+
"role": "Responsible AI",
118+
"name": "Avery Rai",
119+
"status": "approved"
120+
},
121+
{
122+
"role": "Security",
123+
"name": "Chris Sec",
124+
"status": "approved"
125+
}
126+
]
127+
},
128+
"monitoring": {
129+
"owner": "oncall-support",
130+
"metrics": [
131+
{
132+
"name": "Latency p95",
133+
"alertCondition": "above 1.5s for 10m",
134+
"owner": "ml-oncall"
135+
}
136+
],
137+
"telemetry": {
138+
"storage": "datadog::frai-finetune",
139+
"retentionDays": 90
140+
},
141+
"rollbackPlan": {
142+
"trigger": "Any sev1 incident or approval revocation",
143+
"steps": [
144+
"Disable fine-tuned model",
145+
"Revert to base model",
146+
"Notify stakeholders"
147+
]
148+
}
149+
},
150+
"audit": {
151+
"artefacts": ["s3://frai-artifacts/fine-tune/run-42"],
152+
"notes": "Review scheduled with governance board on 2024-05-12."
153+
}
154+
}
155+
```
156+
157+
## Next Steps
158+
- Expose CLI helpers to initialise a governance template and validate JSON files.
159+
- Integrate readiness scoring into `frai eval` and future dashboard surfaces.
160+
- Link artefact storage to `frai-core` document generation for unified audit bundles.

‎docs/refactor-plan.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
- ✅ Assemble AI feature backlog (Day 8) with prioritised user stories (RAG, eval, fine-tune).
2929
- ✅ Implement compliance-aware RAG toolkit (frai-rag).
3030
- ✅ Build evaluation harness (frai-eval).
31-
- [ ] Design fine-tuning governance schema (frai-finetune).
31+
- ✅ Design fine-tuning governance schema (frai-finetune).
3232

3333
### 4. CLI Rebuild
3434
- [ ] Build new CLI entry using `commander` (or similar) within `frai-cli/src/index.ts`.

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
],
99
"scripts": {
1010
"dev": "pnpm --filter frai run start",
11-
"cli": "pnpm --filter frai exec node src/cli.mjs",
11+
"cli": "pnpm --filter frai run build && pnpm --filter frai exec node dist/index.js",
1212
"lint": "turbo lint",
1313
"test": "turbo test",
1414
"build": "turbo build"

0 commit comments

Comments
 (0)