Add LangSmith automation notes, GitHub project templates, and AI coding workflow resources

codekiln · codekiln · commit c1bef3fdd562 · 2025-02-21T07:30:01.000-05:00
diff --git a/assets/Image_from_iOS_1740132023763_0.jpg b/assets/Image_from_iOS_1740132023763_0.jpg
diff --git a/assets/image_1740131905264_0.png b/assets/image_1740131905264_0.png
diff --git a/assets/image_1740132047647_0.png b/assets/image_1740132047647_0.png
diff --git a/journals/2025_02_20.md b/journals/2025_02_20.md
@@ -1,3 +1,3 @@
--
 - #LangSmith
-	-
+	- #Filed #Watched
+		- [[LangChain/YT/PL/Get Started LangSmith/06 Automations n Online Evaluation]]
diff --git a/journals/2025_02_21.md b/journals/2025_02_21.md
@@ -1,20 +1,29 @@
 - [[AI/Coding]]
 	- #Filed
-		- #Learned about the term "GitHub App" - [[GitHub/App]]
+		- #Created
+			- [[GitHub/codekiln/project/ai-coding-template]], a project template that has github statuses for issues inspired by the workflow in [[Person/Harper Reed/Blog/25/02/My LLM codegen workflow atm]]
+			- [[GitHub/codekiln/project/ai-coding]], a [[GitHub/Project]] for managing AI coding todos. Created some custom Github project statuses
+				- https://github.com/users/codekiln/projects/2/settings/fields/Status
+		- #Learned
+			- #Term "GitHub App" - [[GitHub/App]]
+			- [[GitHub/Issue/Draft]]
 		- #Question
 			- [[GitHub/CLI/Q/Is it possible to scope gh CLI permissions by directory or repo]]
 	- [[GitHub/CoPilot]]
 		- #[[JetBrains]]
-			- #Learned that JetBrains IDEs with CoPilot now have a codebase #AI/RAG - [Boost your productivity with GitHub Copilot in JetBrains IDEs, Introducing @project context, AI-generated commit messages, and other updates - GitHub Changelog](https://github.blog/changelog/2025-02-19-boost-your-productivity-with-github-copilot-in-jetbrains-ides-introducing-project-context-ai-generated-commit-messages-and-other-updates/) [[Blog/post]]
-				- `@project` context: Ask questions about your entire project and get intelligent references to relevant files and symbols.
+			- #Learned JetBrains IDEs w/ CoPilot now have codebase #AI/RAG - [Boost your productivity with GitHub Copilot in JetBrains IDEs, Introducing @project context, AI-generated commit messages, and other updates - GitHub Changelog](https://github.blog/changelog/2025-02-19-boost-your-productivity-with-github-copilot-in-jetbrains-ides-introducing-project-context-ai-generated-commit-messages-and-other-updates/) [[Blog/post]]
+				- `@project` context: Ask questions about your entire project and get intelligent references to relevant files and symbols. - [[GitHub/CoPilot/@/project]]
 					- #Question how it compares with [[CursorAI/@/Workspace]]
 				- [[git/commit/message]] with #AI - finally!
-	- #Filed
-		- [[AI/Coding/Todos]]
+	- #Updated
+		- [[Person/Harper Reed/Blog/25/02/My LLM codegen workflow atm]]
+			- #Filed
+				- [[AI/Coding/Todos]] for learning the tools Harper Reed referenced
 - GitHub
 	- #Filed
-		- [[GitHub/Project/Template]] - presumably a particular configuration of swimlanes, issue fields, etc
+		- [[GitHub/Project/Template]] - a particular configuration of swimlanes, issue fields, etc
+		- [[GitHub/Project/Field/Iteration]]
 		- [[GitHub/CLI/project]]
-			- can create #[[GitHub/Projects]]
+			- can create #[[GitHub/Project]]
 			-
 	-
diff --git a/pages/AI___Coding___Todos.md b/pages/AI___Coding___Todos.md
@@ -1,5 +1,6 @@
+- [[GitHub/codekiln/project/ai-coding]]
 - [[Todos/Follow-Up]] from [[Person/Harper Reed/Blog/25/02/My LLM codegen workflow atm]]
-	- TODO install, configure and try [[mise]]
+	- TODO install, configure and try [[mise]] - [gh issue](https://github.com/users/codekiln/projects/3/views/1?pane=issue&itemId=98892630)
 	- TODO install, configure and try [[aider]]
 	- TODO install, configure and try [[llm-cli]]
 	- TODO install, configure and try [[Repomix]]
diff --git a/pages/Create.md b/pages/Create.md
@@ -0,0 +1 @@
+alias:: Created
diff --git a/pages/CursorAI___Idea___Proj Rules GitHub CLI AI SLDC.md b/pages/CursorAI___Idea___Proj Rules GitHub CLI AI SLDC.md
@@ -19,7 +19,7 @@ tags:: [[Idea]]
 	- When using [[CursorAI/Agent Mode]], the agent can read [[CursorAI/Project Rule]] to craft CLI command invocations for the GitHub CLI, `gh`. A lot of information about this CLI is likely already in the training data, but Project Rules may let users focus the AI's attention on the details about how to use it in an AI-assisted [[Software/Engineering/Development Life Cycle (SLDC)]] and specifically within the context of Cursor.
 	- A centerpiece of this approach is that rather than explicitly saying what should be done in the composer chat, the [[Product Requirements Document]] should most likely be specified in a external project management software like a GitHub issue.
 	- ### Defining the [[PRD]] in the [[GitHub/Issue]]
-		- A project rule `gh-issue-update.mdc` rule could be created to help with the ideating phase of the issue. Perhaps a [[GitHub/Projects]] `status` could be applied to the issue to signal that it is in a design phase.
+		- A project rule `gh-issue-update.mdc` rule could be created to help with the ideating phase of the issue. Perhaps a [[GitHub/Project]] `status` could be applied to the issue to signal that it is in a design phase.
 		- A draft issue can be created as a markdown issue, and [[GitHub/CLI/gh/issue/create]] can create an issue when it's time, or [[GitHub/CLI/gh/issue/edit]] could edit an issue if it needs to be updated.
 	- ### Checking out a [[GitHub/Issue]] to work on it
 		- [[GitHub/CLI/gh/issue/develop]] can check out a branch from another branch, and link the branch to the issue. The new branch will be configured as the base branch for pull requests created using  [[GitHub/CLI/gh/pr/create]].
diff --git a/pages/GitHub___CLI.md b/pages/GitHub___CLI.md
@@ -0,0 +1,14 @@
+alias:: gh
+tags:: [[CLI/Tool]]
+
+- # `gh` command
+	- [Manual | GitHub CLI](https://cli.github.com/manual/)
+	- [announced in early 2020](https://github.blog/2020-02-12-supercharge-your-command-line-experience-github-cli-is-now-in-beta/)
+	- ## [Installation](https://github.com/cli/cli#installation)
+		- `brew install gh`
+		- ### Authentication
+			- `gh auth login` - quick start, global
+			- [[GitHub/CLI/EnvVar/GH_TOKEN]]
+				- `Alternatively, populate the GH_TOKEN environment variable with a GitHub API authentication token.`
+				- See also [[GitHub/CLI/Q/Is it possible to scope gh CLI permissions by directory or repo]]
+		-
diff --git a/pages/GitHub___CLI___Q___Is it possible to scope gh CLI permissions by directory or repo.md b/pages/GitHub___CLI___Q___Is it possible to scope gh CLI permissions by directory or repo.md
@@ -1,6 +1,15 @@
-# TODO is it possible to scope the `gh` command so that the permissions it has in one directory are different than the permissions it has in another directory?
+# DOING is it possible to scope the `gh` command so that the permissions it has in one directory are different than the permissions it has in another directory?
+:LOGBOOK:
+CLOCK: [2025-02-21 Fri 05:19:05]
+:END:
 	- ## Impetus
-		- [[CursorAI/Agent Mode]] can invoke the [[GitHub/CLI]] when instructed to do so in a [[CursorAI/Project Rule]]. It would be great if the `gh` cli permissions could be locked down to modifying just the assets associated with a certain repo.
+		- [[CursorAI/Agent Mode]] can invoke the #gh when instructed to do so in a [[CursorAI/Project Rule]]. It would be great if the `gh` cli permissions could be locked down to modifying just the assets associated with a certain repo.
 	- ## Follow-up #Questions
-		- TODO Is it possible that the GitHub CLI access token could source an [[EnvVar/.env]] file in the current repo to obtain a [[GitHub Personal Access Token]] that's scoped to the current repository?
-		- TODO how might we quickly provision scoped, temporary [[GitHub Personal Access Token]]s?
+		- ### DOING Is it possible that the GitHub CLI access token could source an [[EnvVar/.env]] file in the current repo to obtain a [[GitHub Personal Access Token]] that's scoped to the current repository?
+		  :LOGBOOK:
+		  CLOCK: [2025-02-21 Fri 05:19:16]
+		  :END:
+			- #Theoretical - this could be possible with [[GitHub/CLI/EnvVar/GH_TOKEN]]
+				- we could create a wrapper script that implements behavior akin to [[Pyenv/.python-version]]
+		- TODO how might we quickly provision scoped, temporary [[GitHub Personal Access Token]]s?
+			-
diff --git a/pages/GitHub___Issue___Draft.md b/pages/GitHub___Issue___Draft.md
@@ -0,0 +1,12 @@
+alias:: [[GitHub Draft Issue]], [[GitHub Project Item]], [[GitHub/Project/Item]]
+
+- [Adding items to your project - GitHub Docs](https://docs.github.com/en/issues/planning-and-tracking-with-projects/managing-items-in-your-project/adding-items-to-your-project#creating-draft-issues)
+- ## Limitations
+	- it's not possible to trigger GitHub Actions directly from draft issue status changes
+	- draft issues have limited functionality compared to repository issues:
+		- no labels
+		- no milestones
+		- no assignees (until converted to repository issue)
+		- only visible within project view
+	- draft issues cannot be referenced using #issue-number notation
+	- draft issues cannot be mentioned in other issues/PRs using #issue-number notation	
diff --git a/pages/GitHub___Project.md b/pages/GitHub___Project.md
@@ -0,0 +1 @@
+alias:: [[GitHub Projects]], [[GitHub Project]]
diff --git a/pages/GitHub___Project___Field___Iteration.md b/pages/GitHub___Project___Field___Iteration.md
@@ -0,0 +1,18 @@
+alias:: [[GitHub Project Iteration Field]]
+tags:: [[Agile/Sprint]]
+
+- [About iteration fields - GitHub Docs](https://docs.github.com/en/issues/planning-and-tracking-with-projects/understanding-fields/about-iteration-fields)
+	- ## overview
+		- **associate items with specific repeating blocks of time**
+			- like [[Agile/Sprint]]
+		- can be set to any length of time
+		- can include breaks
+		- group by iteration to visualize the balance of upcoming work
+		- use filters to focus on a single iteration
+		- sort by iteration
+	- ## To add a new iteration of the same duration, click Add iteration.
+	- ![add iteration](https://docs.github.com/assets/cb-17133/mw-1440/images/help/projects-v2/add-iteration.webp)
+	- ## Inserting a break
+	- You can insert breaks into your iterations to communicate when you are taking time away from scheduled work. The duration of a new break defaults to the length of the most recently created iteration.
+		- ![insert break](https://docs.github.com/assets/cb-36151/mw-1440/images/help/issues/iteration-insert-break.webp)
+	- Hover over the dividing line above an iteration, then click Insert break.
diff --git a/pages/GitHub___Project___Template.md b/pages/GitHub___Project___Template.md
@@ -1,4 +1,13 @@
 alias:: [[GitHub Project Template]]
 
-- [Managing project templates in your organization - GitHub Docs](https://docs.github.com/en/issues/planning-and-tracking-with-projects/managing-your-project/managing-project-templates-in-your-organization)
-	-
+- # GitHub Project Template
+	- ## #notes
+		- only available for github organizations, not personal accounts. Personal accounts can create a project and then "copy" it.
+	- ## [Managing project templates in your organization - GitHub Docs](https://docs.github.com/en/issues/planning-and-tracking-with-projects/managing-your-project/managing-project-templates-in-your-organization)
+		- If you have admin permissions for a project in your organization, you can set the project as a template and make it available for others in your organization to use.
+		- Navigate to your project.
+		- In the top-right, click  to open the menu.
+		  
+		  ![Screenshot showing a project's menu bar. The menu icon is highlighted with an orange outline.](https://docs.github.com/assets/cb-789/images/help/projects-v2/open-menu.png)
+		- In the menu, click ** Settings** to access the project settings.
+		- In the "Templates" section, next to "Make template", select the switch to toggle it to **On**.
diff --git a/pages/GitHub___Projects___Q___What is the difference between Github Milestones and Github Projects.md b/pages/GitHub___Projects___Q___What is the difference between Github Milestones and Github Projects.md
@@ -1,3 +1,3 @@
 - [[StackOverflow/16/09/What is the diff between Github Projects and Github Milestones]]
-	- [[GitHub/Milestones]] are about [[Release/Management]] and are more for [[Scrum]]. They deal with concrete deadlines. Each issue can be a part of at most one milestone, while a single issue may appear in many [[GitHub/Projects]].
-	- [[GitHub/Projects]] are about [[Kanban]]. They are more of a "going concern," and can even span multiple [[GitHub/Repo]]s. Projects can contain *Notes* that can be converted to issues, so as to prevent spamming the issue tracker with nascent ideas.
+	- [[GitHub/Milestones]] are about [[Release/Management]] and are more for [[Scrum]]. They deal with concrete deadlines. Each issue can be a part of at most one milestone, while a single issue may appear in many [[GitHub/Project]].
+	- [[GitHub/Project]] are about [[Kanban]]. They are more of a "going concern," and can even span multiple [[GitHub/Repo]]s. Projects can contain *Notes* that can be converted to issues, so as to prevent spamming the issue tracker with nascent ideas.
diff --git a/pages/GitHub___codekiln___project___ai-coding-template.md b/pages/GitHub___codekiln___project___ai-coding-template.md
@@ -0,0 +1,5 @@
+tags:: [[GitHub/Project/Template]], [[Template]]
+
+- [View 1 · ai-coding-template](https://github.com/users/codekiln/projects/2)
+	- has [Status fields](https://github.com/users/codekiln/projects/2/settings/fields/Status) that are inspired by [[Person/Harper Reed/Blog/25/02/My LLM codegen workflow atm]]
+	-
diff --git a/pages/GitHub___codekiln___project___ai-coding.md b/pages/GitHub___codekiln___project___ai-coding.md
@@ -0,0 +1,3 @@
+tags:: [[GitHub Project]]
+
+- # [View 1 · ai-coding](https://github.com/users/codekiln/projects/3)
diff --git a/pages/LangChain___YT___PL___Get Started LangSmith___06 Automations n Online Evaluation.md b/pages/LangChain___YT___PL___Get Started LangSmith___06 Automations n Online Evaluation.md
@@ -0,0 +1,39 @@
+# DONE [Getting Started with LangSmith (6/7): Automations & Online Evaluation - YouTube](https://www.youtube.com/watch?v=xj2R3lBgihs)
+	- notes
+		- Contains examples of chaining automation rules with online evaluation
+			- ![image.png](../assets/image_1740131905264_0.png)
+			- ![image.png](../assets/image_1740132047647_0.png)
+		- Example of you using structured output with LLM as a judge
+			- ![Image from iOS.jpg](../assets/Image_from_iOS_1740132023763_0.jpg)
+		-
+	- vid
+		- {{video https://www.youtube.com/watch?v=xj2R3lBgihs}}
+			- {{youtube-timestamp 0}} in this video we're going to talk about setting up automations in link Smith automations are executed live on traces as they come in from your application today we'll take a look at the different types of automations that we can set up
+			- {{youtube-timestamp 14}} uh one of which is onlo evaluations cool first let's take a look at these two traces which I ran recently there are two questions that I asked to our parent named paully and we can see that for this one paully does a pretty
+			- {{youtube-timestamp 29}} good job she likes playing soccer this was in her list of facts however for this one I asked do you like animals and paully actually says that she likes animal crackers this was kind of a tricky one on purpose paully has a fact
+			- {{youtube-timestamp 41}} that says that she likes animal crackers and we can see in this case that she kind of uh hallucinates and doesn't really answer the right question Cool let's take a look at automations uh to do that we'll click on this rules
+			- {{youtube-timestamp 56}} button in the upper right hand side and we can see two existing automations that I've already set up uh but let's go ahead and click add Rule and sort of walk through the different kinds of automations we can set up and how we can set them
+			- {{youtube-timestamp 70}} up so first we can name our new automation rule then we want to set a series of run filters and the Run filters are applied to determine which runs in this table are actually applicable for this automation you can filter on a lot of different things uh a
+			- {{youtube-timestamp 87}} common one might be status and maybe you only care about traces where status is Success uh so you can filter on that and the fact that it's a root run there's also this uh sliding bar that allows you to set the sampling rate
+			- {{youtube-timestamp 102}} so for some automations you'll want to sample every time but for other more expensive ones such as L as a judge you might only want to sample for some subset of the automations and this basically just allows you to tweak
+			- {{youtube-timestamp 115}} that we also have this button that allows you to apply this new rule onto past runs and so if I do toggle this on I just need to select a date to backfill from and it will start this rule automation at that date and uh apply it
+			- {{youtube-timestamp 131}} there cool let's walk through our actions uh there are a few basic ones like we can automatically push a trace to an annotation queue and similarly we can automatically push a run to a data set we can also specify LM as a judge
+			- {{youtube-timestamp 148}} evaluators or custom Cod evaluators to conduct online evaluation online evaluation uh like these automations is applied to your run at runtime live as it comes in and so this is a really great way to get feedback on your application immediately
+			- {{youtube-timestamp 165}} uh whether through an llm or through a code evaluator setting up the evaluator looks very similar to what we did for offline evaluation we can uh use an off-the-shelf evaluator or we can create a prop from scratch and we just wire up
+			- {{youtube-timestamp 179}} various components from our application to the actual prompt that we pass the llm the same thing goes for the custom code evaluator except in this case we only have access to the Run object uh and we can do different things like
+			- {{youtube-timestamp 194}} check that particular part of your run matches and expected value cool you can also trigger web hooks from uh these automations a pretty common flow might be if you get a certain type of error you might want to trigger a web hook that will create a
+			- {{youtube-timestamp 213}} Duro ticket or uh ping someone to take a closer look this is a great way uh to do that through automations cool we're not actually going to add a new automation here let's take a closer look at these two automations that I have set up one is
+			- {{youtube-timestamp 231}} called score answer relevance and the other one is called add relevant answers to annotation que so the first uh evaluator here uses an llm as a judge online to score the relevance of the answer that we provided for the user's
+			- ### examples of chaining automation rules with online evaluation
+				- ![image.png](../assets/image_1740131905264_0.png)
+					- Score Answer Relevance adds relevance, then Add Relevant Answers to Annotation Queue adds only the highly scored Relevance traces to the queue
+			- {{youtube-timestamp 248}} question the second one takes this feedback score and if it is high enough it will automatically push that run into an annotation CU for a human to look at this is pretty neat because these two evaluators are chained in a sense first
+			- {{youtube-timestamp 263}} we need this online LM as a judge to give us a feedback score and then based on this feedback score we might add it to an annotation CU there's nothing special that we did here to allow them to be chained in this sense for our
+			- {{youtube-timestamp 276}} evaluators that generate feedback Those runs with the new feedback are then run through these rules again to make sure that any uh rules that are triggered off of those SC pieces of feedback will fire cool let's take a closer look at
+			- {{youtube-timestamp 292}} score answer relevance uh if I go into this rule edit it I can see it's an LM as judge evaluator and if I click into the pro I can see all that I've done is I've mapped in my input and output and I've asked the llm to return me a
+			- {{youtube-timestamp 307}} structured output score of relevance uh this is going to be a number from 1 to 10 and so all of my answers are being scored uh from 1 to 10 on relevance as my sampling right here is one I can then take a look uh at my logs
+			- {{youtube-timestamp 326}} for this particular Automation and I can see it recently ran on these two traces right here so for my question of do you like soccer the answer was quite relevant if I take a look at the Run um once again we can immediately see uh
+			- {{youtube-timestamp 342}} that yes I like soccer is is pretty relevant however uh this other run here which we also just took a look at where we ask if it likes animals and paully says that she likes animal crackers is not super relevant and so this got a
+			- {{youtube-timestamp 356}} feedback score of three note that this feedback that we get is available under the auto evaluator source and the key here is provided as well as the score in this feedback pain in the trace so you can always go back to the trace to see this
+			- {{youtube-timestamp 372}} feedback cool now let's make sure that our Auto evaluate ref fired as our final step here if we navigate over to our annotation queue poly correctness analysis and we wait for it to load we can see that we did in fact get this do you like soccer
+			- {{youtube-timestamp 390}} uh question here and now a human can come in evaluate it potentially edit it and then potentially add it to a data set thanks for following along
diff --git a/pages/Person___Harper Reed___Blog___25___02___My LLM codegen workflow atm.md b/pages/Person___Harper Reed___Blog___25___02___My LLM codegen workflow atm.md
diff --git a/pages/Pyenv___.python-version.md b/pages/Pyenv___.python-version.md
diff --git a/pages/StackOverflow___16___09___What is the diff between Github Projects and Github Milestones.md b/pages/StackOverflow___16___09___What is the diff between Github Projects and Github Milestones.md
