Merge branch 'main' into tmoreau89/android

Author: tmoreau89

recipes/use_cases/agents/langchain/README.md

@@ -1,61 +1,70 @@
 # LangChain <> Llama3 Cookbooks
 
-LLM agents use [planning, memory, and tools](https://lilianweng.github.io/posts/2023-06-23-agent/) to accomplish tasks.
+LLM agents use [planning, memory, and tools](https://lilianweng.github.io/posts/2023-06-23-agent/) to accomplish tasks. Agents can empower Llama 3 with important new capabilities. Here, we will show how to give Llama 3 the ability to perform web search, as well as multi-modality: image generation (text-to-image), image analysis (image-to-text), and voice (text-to-speech) tools!
 
-LangChain offers several different ways to implement agents.
+LangChain offers several different ways to implement agents with Llama 3:
 
-(1) Use [AgentExecutor](https://python.langchain.com/docs/modules/agents/quick_start/) with [tool-calling](https://python.langchain.com/docs/integrations/chat/) versions of Llama 3.
+(1) `ReAct agent` - Uses [AgentExecutor](https://python.langchain.com/docs/modules/agents/quick_start/) with [tool-calling](https://python.langchain.com/docs/integrations/chat/) versions of Llama 3.
 
-(2) Use [LangGraph](https://python.langchain.com/docs/langgraph), a library from LangChain that can be used to build reliable agents with Llama 3.
+(2) `LangGraph tool calling agent` - Uses [LangGraph](https://python.langchain.com/docs/langgraph) with [tool-calling](https://python.langchain.com/docs/integrations/chat/) versions of Llama 3.
+
+(3) `LangGraph custom agent` - Uses [LangGraph](https://python.langchain.com/docs/langgraph) with **any** version of Llama 3 (so long as it supports structured output).
+
+As we move from option (1) to (3) the degree of customization and flexibility increases:
+
+(1) `ReAct agent` using AgentExecutor is a great for getting started quickly with minimal code, but requires a version of Llama 3 with reliable tool-calling, is the least customizable, and uses higher-level AgentExecutor abstraction.
+  
+(2) `LangGraph tool calling agent` is more customizable than (1) because the LLM assistant (planning) and tool call (action) nodes are defined by the user, but it still requires a version of Llama 3 with reliable tool-calling.
+  
+(3) `LangGraph custom agent` does not require a version of Llama 3 with reliable tool-calling and is the most customizable, but requires the most work to implement. 
+
+![langgraph_agent_architectures](https://github.com/rlancemartin/llama-recipes/assets/122662504/5ed2bef0-ae11-4efa-9e88-ab560a4d0022)
 
 ---
 
-### AgentExecutor Agent
+### `ReAct agent`
 
-AgentExecutor is the runtime for an agent. AgentExecutor calls the agent, executes the actions it chooses, passes the action outputs back to the agent, and repeats.
+The AgentExecutor manages the loop of planning, executing tool calls, and processing outputs until an AgentFinish signal is generated, indicating task completion.
 
 Our first notebook, `tool-calling-agent`, shows how to build a [tool calling agent](https://python.langchain.com/docs/modules/agents/agent_types/tool_calling/) with AgentExecutor and Llama 3.
 
-This shows how to build an agent that uses web search, text2image, image2text, and text2speech tools.
-
 --- 
 
-### LangGraph Agent
+### `LangGraph tool calling agent`
 
 [LangGraph](https://python.langchain.com/docs/langgraph) is a library from LangChain that can be used to build reliable agents.
 
-LangGraph can be used to build agents with a few pieces:
-- **Planning:** Define a control flow of steps that you want the agent to take (a graph)
-- **Memory:** Persist information (graph state) across these steps
-- **Tool use:** Modify state at any step
+Our second notebook, `langgraph-tool-calling-agent`, shows an alternative to AgentExecutor for building a Llama 3 powered agent. 
+
+--- 
 
-Our second notebook, `langgraph-agent`, shows an alternative way to AgentExecutor to build a Llama 3 powered agent in LangGraph. 
+### `LangGraph custom agent`
 
-It discusses some of the trade-offs between AgentExecutor and LangGraph. 
+Our third notebook, `langgraph-custom-agent`, shows how to build a Llama 3 powered agent without reliance on tool-calling. 
 
 --- 
 
-### LangGraph RAG Agent
+### `LangGraph RAG Agent`
 
-Our third notebook, `langgraph-rag-agent`, shows how to apply LangGraph to build advanced Llama 3 powered RAG agents that use ideas from 3 papers:
+Our fourth notebook, `langgraph-rag-agent`, shows how to apply LangGraph to build a custom Llama 3 powered RAG agent that use ideas from 3 papers:
 
 * Corrective-RAG (CRAG) [paper](https://arxiv.org/pdf/2401.15884.pdf) uses self-grading on retrieved documents and web-search fallback if documents are not relevant.
 * Self-RAG [paper](https://arxiv.org/abs/2310.11511) adds self-grading on generations for hallucinations and for ability to answer the question.
 * Adaptive RAG [paper](https://arxiv.org/abs/2403.14403) routes queries between different RAG approaches based on their complexity.
 
 We implement each approach as a control flow in LangGraph:
-- **Planning:** The sequence of RAG steps (e.g., retrieval, grading, and generation) that we want the agent to take
-- **Memory:** All the RAG-related information (input question, retrieved documents, etc) that we want to pass between steps
-- **Tool use:** All the tools needed for RAG (e.g., decide web search or vectorstore retrieval based on the question)
+- **Planning:** The sequence of RAG steps (e.g., retrieval, grading, and generation) that we want the agent to take.
+- **Memory:** All the RAG-related information (input question, retrieved documents, etc) that we want to pass between steps.
+- **Tool use:** All the tools needed for RAG (e.g., decide web search or vectorstore retrieval based on the question).
 
 We will build from CRAG (blue, below) to Self-RAG (green) and finally to Adaptive RAG (red):
 
-![Screenshot 2024-05-03 at 10 50 02 AM](https://github.com/rlancemartin/llama-recipes/assets/122662504/ec4aa1cd-3c7e-4cd1-a1e7-7deddc4033a8)
+![langgraph_rag_agent_](https://github.com/rlancemartin/llama-recipes/assets/122662504/ec4aa1cd-3c7e-4cd1-a1e7-7deddc4033a8)
 
 --- 
+ 
+### `Local LangGraph RAG Agent`
 
-### Local LangGraph RAG Agent
-
-Our fourth notebook, `langgraph-rag-agent-local`, shows how to apply LangGraph to build advanced RAG agents using Llama 3 that run locally and reliably.
+Our fifth notebook, `langgraph-rag-agent-local`, shows how to apply LangGraph to build advanced RAG agents using Llama 3 that run locally and reliably.
 
-See this [video overview](https://www.youtube.com/watch?v=sgnrL7yo1TE) for more detail.
+See this [video overview](https://www.youtube.com/watch?v=sgnrL7yo1TE) for more detail on the design of this agent.

recipes/use_cases/agents/langchain/langgraph-agent.ipynb renamed to recipes/use_cases/agents/langchain/langgraph-custom-agent.ipynb

@@ -18,21 +18,6 @@
     "! pip install -U langchain_groq langchain tavily-python replicate"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "745f7d9f-15c4-41c8-94f8-09e1426581cc",
-   "metadata": {},
-   "source": [
-    "# Tool calling agent with Llama 3\n",
-    "\n",
-    "[Tool calling](https://python.langchain.com/docs/modules/agents/agent_types/tool_calling/) allows an LLM to detect when one or more tools should be called.\n",
-    "\n",
-    "It will then respond with the inputs that should be passed to those tools. \n",
-    "\n",
-    "LangChain has a general agent that works with tool-calling LLMs. In this notebook we'll use the following tools with Agent Executor:\n",
-    "\n"
-   ]
-  },
   {
    "attachments": {
     "6af2e76d-6f3d-44c9-a128-c19e70a400e9.png": {
@@ -43,6 +28,14 @@
    "id": "3fcd455a-6255-4c7c-9e8e-2aaf1b323c05",
    "metadata": {},
    "source": [
+    "# Tool calling agent with Llama 3\n",
+    "\n",
+    "LLM-powered agents combine planning, memory, and tool-use (see [here](https://lilianweng.github.io/posts/2023-06-23-agent/), [here](https://www.deeplearning.ai/the-batch/how-agents-can-improve-llm-performance/)).\n",
+    "\n",
+    "LangChain's [agent executor](https://python.langchain.com/docs/modules/agents/agent_types/tool_calling/) offers a simple way to quickly get started with agents.\n",
+    "\n",
+    "Here, we will show how to augment a tool-calling version of Llama 3 with various multi-modal capabilities using an agent. \n",
+    "\n",
     "![image.png](attachment:6af2e76d-6f3d-44c9-a128-c19e70a400e9.png)"
    ]
   },
@@ -51,11 +44,11 @@
    "id": "8baac811-22b0-4bcb-a7d4-6eb87a1b61d3",
    "metadata": {},
    "source": [
-    "## Tools\n",
+    "## Introduce the tools that we want our agent to use\n",
     "\n",
     "### 1. Custom Function\n",
     "\n",
-    "We'll define a few tools that our agent will use."
+    "This is a custom function that we want our agent to utilize."
    ]
   },
   {
@@ -85,8 +78,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
-   "id": "07dc6530-abb0-4bda-a071-1163239ab2c6",
+   "execution_count": null,
+   "id": "75c46af4-f03e-4448-a49e-8bfe0e644c2a",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -105,23 +98,15 @@
    "source": [
     "### 2. Web Search\n",
     "\n",
-    "Let's use [Tavily](https://tavily.com/#api) for web search."
+    "We'll use [Tavily](https://tavily.com/#api) for web search."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
-   "id": "16569e69-890c-4e0c-947b-35ed2e0cf0c7",
+   "execution_count": null,
+   "id": "4c6be142-757c-4ee4-983e-0b5a54fafff9",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdin",
-     "output_type": "stream",
-     "text": [
-      " ········\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "# Ensure API key is set\n",
     "import os\n",
@@ -148,25 +133,17 @@
    "source": [
     "### 3. Text-2-Image\n",
     "\n",
-    "We'll use Replicate, which [hosts an open DALL-E model](https://replicate.com/lucataco/open-dalle-v1.1/versions/1c7d4c8dec39c7306df7794b28419078cb9d18b9213ab1c21fdc46a1deca0144).\n",
+    "We'll use [Replicate](https://replicate.com/), which offers free to try API key and hosts an [open DALL-E model](https://replicate.com/lucataco/open-dalle-v1.1/versions/1c7d4c8dec39c7306df7794b28419078cb9d18b9213ab1c21fdc46a1deca0144).\n",
     "\n",
-    "Test the code before converting it to a tool (this may take 1-2 minutes to run):"
+    "Test the code (this may take 1-2 minutes to run):"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
-   "id": "c130481d-dc6f-48e0-b795-7e3a4438fb6a",
+   "execution_count": null,
+   "id": "2fb8879e-24e2-4370-b89e-10644ce1ddfc",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdin",
-     "output_type": "stream",
-     "text": [
-      " ········\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "# Ensure API key is set\n",
     "REPLICATE_API_TOKEN = getpass()\n",
@@ -282,7 +259,7 @@
    "source": [
     "### 4. Image-2-Text\n",
     "\n",
-    "We'll use Replicate, which [hosts llava-13b](https://replicate.com/yorickvp/llava-13b).\n",
+    "We'll use Replicate, which hosts [llava-13b](https://replicate.com/yorickvp/llava-13b).\n",
     "\n",
     "Test the code before converting it to a tool:"
    ]
@@ -363,7 +340,7 @@
    "source": [
     "### 5. Text-2-Speech\n",
     "\n",
-    "We'll use Replicate, which [hosts text-2-speech](https://replicate.com/cjwbw/seamless_communication).\n",
+    "We'll use Replicate, which hosts [text-2-speech](https://replicate.com/cjwbw/seamless_communication).\n",
     "\n",
     "Test the code before creating yet another custom tool (this may take a couple of minutes to run):"
    ]
@@ -498,7 +475,7 @@
     "\n",
     "This can be accomplished via prompt engineering (e.g., see [here](https://replicate.com/hamelsmu/llama-3-70b-instruct-awq-with-tools)) or fine-tuning (e.g., see [here](https://huggingface.co/mzbac/llama-3-8B-Instruct-function-calling) and [here](https://huggingface.co/mzbac/llama-3-8B-Instruct-function-calling)).\n",
     "\n",
-    "We can review LLMs that support tool calling [here](https://python.langchain.com/docs/integrations/chat/) and Groq is included.\n",
+    "We can review LangChain LLM integrations that support tool calling [here](https://python.langchain.com/docs/integrations/chat/) and Groq is included.\n",
     "\n",
     "[Here](https://github.com/groq/groq-api-cookbook/blob/main/llama3-stock-market-function-calling/llama3-stock-market-function-calling.ipynb) is a notebook by Groq on function calling with Llama 3 and LangChain."
    ]
@@ -836,7 +813,7 @@
    "source": [
     "We can see that the agent correctly decides which tool to call for different query using AgentExecutor. \n",
     "\n",
-    "In the next [notebook](langgraph-agent.ipynb), we will show an alternative way to implement this agent using LangGraph."
+    "In the next [notebook](langgraph-tool-calling-agent.ipynb), we will show an alternative way to implement this agent using LangGraph."
    ]
   }
  ],
@@ -856,7 +833,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.9"
+   "version": "3.10.14"
   }
  },
  "nbformat": 4,

recipes/use_cases/agents/langchain/langgraph-tool-calling-agent.ipynb

@@ -1342,9 +1342,12 @@ sha
 tmoreau
 toolchain
 wifi
+AgentFinish
+ReAct
+customizable
 Kaggle
 SalesBot
 Weaviate
 MediaGen
 SDXL
-SVD
+SVD