Skip to content

Commit 0a6079d

Browse files
codekilncodekiln
committed
feat: Add Prompt Ops platform pages and reorganize PromptFooAI namespace
- Add pages for PromptLayerAI, BraintrustAI, HumanloopAI, and PromptFooAI - Create AI/LLM/Observability/Platform and AI/LLM/Ops/Platform concept pages - Create AI/Security/Attack/Prompt/Injection page with alias - Move PromptFooAI to its own top-level namespace (like LangSmith) - Tag all Prompt Ops platforms with observability and ops platform tags - Update journal entry with PromptFooAI reference
1 parent f53e3fd commit 0a6079d

File tree

8 files changed

+117
-5
lines changed

8 files changed

+117
-5
lines changed

‎journals/2025_11_06.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
1-
## [[LangSmith Evals]]
1+
## Langsmithery
22
- [[LangSmith/Eval/Q/Can an Annotation Queue be attached to an Evaluator]]
33
- Discovered that Custom Output Rendering is possible on a [[LangSmith/Annotation/Queue]]; see [[LangSmith/Docs/Custom Output Rendering]]
44
- [[LangSmith/Eval/Idea/Store One Annotation Queue Per Behavior]]
55
- [[LangSmith/Resource/Tag/Q/What types of resource tags are there]]
66
- [[LangSmith/Annotation/Queue/Q/What goes in the default dataset]]
7-
- ## The GitHub World
7+
- ## GitHubbery
88
- [[GitHub/Codespace/Q/Can OIDC Grant Access to AWS Bedrock as in GitHub Actions for Claude Code]]
9-
- ## [[AWS]] Resources
9+
- ## AWSery
1010
- [[AWS/Solutions Library/Sample/Guidance for Claude Code with Amazon Bedrock]] created to document the AWS Solutions Library sample repository that demonstrates secure enterprise authentication for Amazon Bedrock using OIDC identity providers and AWS services
1111
- [[AWS/IAM/Identity Center/Technique/Long-Lived Temporary Credentials]] - Notes on using AWS IAM Identity Center as an alternative to role chaining for longer-lived temporary credentials (1-12 hours vs 1 hour limit)
12-
- ## [[JIRA]] Resources
13-
- [[JIRA/How To/Customize Email Notifications by Space]] - Documented that `/jira/settings/personal/notifications` is the path where one can customize email notification preferences by space in Atlassian products
12+
- ## AI Security
13+
- [[PromptFooAI]] - Testing framework for LLM applications that helps detect prompt injection attacks and other security vulnerabilities

‎pages/AI___LLM___Observability___Platform.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
tags:: [[AI/LLM/Observability]]
2+
3+
- # LLM Observability Platform
4+
- Platforms that provide observability, monitoring, and tracing capabilities for LLM applications
5+
- Key features typically include:
6+
- Real-time monitoring of LLM usage
7+
- Latency and performance tracking
8+
- Cost tracking and analysis
9+
- Log aggregation and search
10+
- Trace visualization
11+
- Error tracking and debugging
12+
- ## Platforms
13+
- [[PromptLayerAI]] - Prompt management, evaluations, and LLM observability
14+
- [[BraintrustAI]] - AI observability platform with real-time monitoring
15+
- [[HumanloopAI]] - LLM evaluation platform with observability features
16+
- [[LangSmith]] - LLM observability and evaluation platform
17+
- [[PromptFooAI]] - Testing framework with evals for LLM applications
18+

‎pages/AI___LLM___Ops___Platform.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
tags:: [[AI/LLM/Ops]]
2+
3+
- # LLM Ops Platform (Prompt Ops)
4+
- Platforms that help with LLM operations, including prompt engineering, evaluations, versioning, and deployment
5+
- Also known as "Prompt Ops" tools
6+
- Key capabilities typically include:
7+
- Prompt management and versioning
8+
- Prompt engineering workflows
9+
- Evaluation and testing frameworks
10+
- A/B testing and experimentation
11+
- Regression testing
12+
- Dataset management
13+
- Prompt deployment and rollback
14+
- Collaboration tools for non-technical stakeholders
15+
- ## Platforms
16+
- [[PromptLayerAI]] - Your workbench for AI engineering with prompt management, evaluations, and observability
17+
- [[BraintrustAI]] - AI observability platform with prompt engineering and batch testing
18+
- [[HumanloopAI]] - LLM evaluation platform with prompt management
19+
- [[PromptFooAI]] - Testing framework for detecting prompt injection attacks and security vulnerabilities
20+
- [[LangSmith]] - LLM observability and evaluation platform
21+

‎pages/AI___Security___Attack___Prompt___Injection.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
tags:: [[AI/Security/Attack]]
2+
alias:: [[Prompt Injection]]
3+
4+
- # Prompt Injection
5+
- A security vulnerability where malicious input is crafted to manipulate or override the intended behavior of an LLM application
6+
- Attackers inject malicious instructions into prompts to:
7+
- Bypass safety measures
8+
- Extract sensitive information
9+
- Manipulate the AI's behavior
10+
- Gain unauthorized access to systems
11+
- ## Types
12+
- **Direct Prompt Injection**: Malicious input is directly inserted into user prompts
13+
- **Indirect Prompt Injection**: Malicious content is embedded in data sources that the AI processes (e.g., web pages, documents, GitHub issues)
14+
- ## Detection and Testing
15+
- [[PromptFooAI]] - Testing framework for detecting prompt injection attacks
16+
- ## Related Concepts
17+
- [[AI/Security/Attack/Toxic Agent Flow]] - Use of indirect prompt injection to trigger malicious tool use sequences
18+

‎pages/BraintrustAI.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
tags:: [[AI/Security]], [[AI/LLM/Observability/Platform]], [[AI/LLM/Ops/Platform]]
2+
3+
- # BraintrustAI
4+
- [Braintrust](https://www.braintrust.dev/) - AI observability platform
5+
- Platform for building quality AI products
6+
- Provides infrastructure for evaluating AI applications, monitoring performance, and ensuring reliable outputs
7+
- Features include:
8+
- Prompt engineering
9+
- Batch testing
10+
- Real-time monitoring
11+
- Automated and human scoring
12+
- Scalable log ingestion
13+
- Helps teams iterate, evaluate, and deploy AI applications effectively
14+

‎pages/HumanloopAI.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
tags:: [[AI/Security]], [[AI/LLM/Observability/Platform]], [[AI/LLM/Ops/Platform]]
2+
3+
- # HumanloopAI
4+
- [Humanloop](https://humanloop.com/) - LLM evaluation platform
5+
- Development platform for LLM applications
6+
- Focused on enabling the safe and rapid adoption of AI
7+
- Features include:
8+
- Prompt management
9+
- Evaluation
10+
- Observability
11+
- In 2025, Humanloop joined Anthropic to further their mission of enabling the safe and rapid adoption of AI
12+

‎pages/PromptFooAI.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
tags:: [[AI/Security]], [[AI/Security/Attack/Prompt/Injection]], [[AI/LLM/Observability/Platform]], [[AI/LLM/Ops/Platform]]
2+
3+
- # PromptFooAI
4+
- [PromptFoo](https://www.promptfoo.dev/) - Testing framework for LLM applications
5+
- PromptFoo is a testing framework for LLM applications that helps detect prompt injection attacks and other security vulnerabilities
6+
- Has evals (like [[LangSmith]])
7+
- ## Mentions
8+
- [[Person/Shawn @swyx Wang]] mentioned PromptFoo in [[Latent Space/Pod]] episode about Sander Schulhoff's The Prompt Report
9+
- [Consistently use the OpenAI playground for prompting. (50sec)](https://share.snipd.com/snip/8f14b85f-3deb-4640-83f3-00c201541caf)
10+
- swyx called out specialists in this area: **[[PromptLayerAI]], [[BraintrustAI]], [[PromptFooAI]], [[HumanloopAI]]**
11+

‎pages/PromptLayerAI.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
tags:: [[AI/Security]], [[AI/LLM/Observability/Platform]], [[AI/LLM/Ops/Platform]]
2+
3+
- # PromptLayerAI
4+
- [PromptLayer](https://www.promptlayer.com/) - Your workbench for AI engineering
5+
- Platform for prompt management, evaluations, and LLM observability
6+
- Enables teams to version, test, and monitor prompts and agents with robust evals, tracing, and regression sets
7+
- Empowers domain experts to collaborate in the visual editor
8+
- Features include:
9+
- Prompt management and versioning
10+
- Visual prompt editor (no-code)
11+
- A/B testing
12+
- Evaluations and regression tests
13+
- Observability and monitoring
14+
- Dataset management
15+
- Prompt chaining
16+
- Model agnostic - one prompt template for every model
17+
- SOC 2 Type 2 compliant, HIPAA compliant
18+

0 commit comments

Comments
 (0)