[Docs Agent] Docs Agent release version 0.3.6 (#523)

* [docs-agent] Clean up Docs Agent's README.md files.

* [Release] Docs Agent version 0.3.1

What's changed:

- Bug fixes in the Docs Agent web app UI.
- Added more templates for the Docs Agent web app: widget and experimental
- A new custom splitter added: FIDL (.fidl) file specific splitter.

* [docs] Update the git clone instructions on Docs Agent's CLI setup
guide.

* [docs] Minor edits in Docs Agent's CLI setup guide.

* [Docs Agent] Release of Docs Agent v 0.3.2.

What's change:

- Support the new Gemini 1.5 models in preview.
- Add new experimental CLI features to interact with Gemini models
  directly from a Linux terminal: `agent tellme` and `agent helpme`.
- Better handle uploading of text chunks using the Semantic Retrieval API.
- Add a new chat UI feature to provide a page for viewing logs.
- Update the Google Generative AI SDK version to `0.5.0`.
- Refactor the pre-preprocessing module (in progress).
- Remove unused code and type mismatch errors.
- Bug fixes.

* [docs-agent] Release of Docs Agent v.0.3.3

What's changed:

- Added a new post-processing feature to delete existing chunks
  in databases if they are no longer found in source dataset.
- Updated the pre-processing module to make sure text chunks
  are not bigger than 5 KB for generating embeddings.
- Added a feature to display the distribution of chunk sizes
  after running `agent chunk`.
- Added more `agent` commandlines and options.
- Updated the "Rewrite" button to the "Feedback" button by default.
- Bug fixes.

* [Docs Agent] Release of Docs Agent v.0.3.4

What's changed

-  **Enhanced file processing:** Process an entire directory with the
   `agent helpme` command and export results to a `responses.yaml` file.
-  **Improved session control:** Manage conversation context using the
   `--new` and `--cont` flags with `agent helpme`.
-  **Debugging tools:** Capture detailed logs for troubleshooting and
   export them to CSV with the new `agent write-logs-to-csv` command.
-  **Documentation expansion:** Explore updated configuration references,
   recent release notes, and a new concept doc on chunking.
-  **Bug fixes:** Added fixes for the FIDL splitter, file handling, and
   blank results, and updates to various README files.

* [Docs Agent] Bug fix - Fix the "index out of range" errors in Docs
Agent's chat web app.

* [Docs Agent] Release of Docs Agent v 0.3.5.

What's changed:

- **`agent runtask` command:** Run tasks defined in `.yaml` files, for example,
  generating release notes based on commit messages.
- **Session Management:** Start and maintain sessions using `--new` and `--cont`
  flags with the `helpme` and `tellme` commands.
- **CLI Command Refactoring:** The CLI is more modular and easier to extend
  with new commands.
- **Web App Mode Updates:** A new `widget-pro` mode is added as an
  iFrame-friendly template.
- **Bug Fixes:**  Various bug fixes related to flags, UI, and file handling.

* [cli] Update the pyproject.toml file missed in the previos commit for
0.3.5 release.

* [Docs Agent] Remove unused CSS files from the previous 0.3.5 version
commit.

* [Docs Agent] Release of Docs Agent v 0.3.6.

What's changed:

- Runtask enhancement: The `runtask` command allows specifying
  a custom input string using the `--custom_input` flag.
- ExtractWorkflows task: This new task extracts workflows and
  commandlines from a source document.
- HelpPolishPrompts task: This task helps users polish prompts
  in a task file.
- IndexPageGenerator task: This task generates an index page
  describing files in a given directory.
- Chat app bug fix: Fixed an issue where + characters were
  removed from the page title.

Author: kyolee415

examples/gemini/python/docs-agent/README.md

@@ -7,9 +7,9 @@ Docs Agent provides a set of easy-to-use self-service tools designed to give you
 your team access to Google's [Gemini API][genai-doc-site] for learning, experimentation,
 and project deployment.
 
-## Overview
+## Docs Agent web app
 
-Docs Agent apps use a technique known as Retrieval Augmented Generation (RAG), which
+Docs Agent uses a technique known as **Retrieval Augmented Generation (RAG)**, which
 allows you to bring your own documents as knowledge sources to AI language models.
 This approach helps the AI language models to generate relevant and accurate responses
 that are grounded in the information that you provide and control.
@@ -18,13 +18,24 @@ that are grounded in the information that you provide and control.
 
 **Figure 1**. Docs Agent uses a vector database to retrieve context for augmenting prompts.
 
-Docs Agent apps are designed to be easily set up and configured in a Linux environment.
-If you want to set up and launch the Docs Agent chat app on your host machine, check out
-the [Set up Docs Agent][set-up-docs-agent] section below.
+The Docs Agent chatbot web app is designed to be easily set up and configured in a Linux
+environment. If you want to set up and launch the Docs Agent chat app on your host machine,
+check out the [Set up Docs Agent][set-up-docs-agent] section below.
 
-### Summary of features
+## Docs Agent tasks
 
-The following list summarizes the tasks and features supported by Docs Agent:
+Docs Agent's `agent runtask` command allows you to run pre-defined chains of prompts,
+which are referred to as **tasks**. These tasks simplify complex interactions by defining
+a series of steps that the Docs Agent will execute. The tasks are defined in `.yaml` files
+stored in the [`tasks`][tasks-dir] directory of your Docs Agent project. The tasks are
+designed to be reusable and can be used to automate common workflows, such as generating
+release notes, updating documentation, or analyzing complex information.
+
+To set up and run the `agent runtask` command, see [Set up Docs Agent CLI][cli-readme].
+
+## Summary of features
+
+The list below summarizes the tasks and features supported by Docs Agent:
 
 - **Process Markdown**: Split Markdown files into small plain text chunks. (See
   [Docs Agent chunking process][chunking-process].)
@@ -60,9 +71,9 @@ The following list summarizes the tasks and features supported by Docs Agent:
   [Set up the Docs Agent CLI][cli-readme] to make requests to the Gemini models
   from anywhere in a terminal.
 - **Support the Gemini 1.5 models**: Docs Agent works with the Gemini 1.5 models,
-  `gemini-1.5-pro-latest` and `text-embedding-004`. The new ["1.5"][new-15-mode] web app
-  mode uses all three Gemini models to their strength: AQA (`aqa`), Gemini 1.0 Pro
-  (`gemini-pro`), and Gemini 1.5 Pro (`gemini-1.5-pro-latest`).
+  `gemini-1.5-pro`, `gemini-1.5-flash`, and `text-embedding-004`. The new
+  [`full`][new-15-mode] web app mode uses all three Gemini models to their strength:
+  AQA (`aqa`), Gemini 1.0 Pro (`gemini-pro`), and Gemini 1.5 Pro (`gemini-1.5-pro`).
 - **Complete a task using the Docs Agent CLI**: The `agent runtask` command allows you
   to run pre-defined chains of prompts, which are referred to as tasks. These tasks
   simplify complex interactions by defining a series of steps that the Docs Agent will

examples/gemini/python/docs-agent/config.yaml

@@ -17,15 +17,15 @@
 configs:
   - product_name: "Fuchsia"
     models:
-      - language_model: "models/aqa"
+      - language_model: "models/gemini-1.5-flash-latest"
         embedding_model: "models/embedding-001"
         api_endpoint: "generativelanguage.googleapis.com"
         embedding_api_call_limit: 1400
         embedding_api_call_period: 60
     docs_agent_config: "normal"
     markdown_splitter: "token_splitter"
     log_level: "NORMAL"
-    db_type: "google_semantic_retriever"
+    db_type: "chroma"
     db_configs:
       - db_type: "chroma"
         vector_db_dir: "vector_stores/chroma"

examples/gemini/python/docs-agent/docs/cli-reference.md

@@ -273,8 +273,29 @@ For example:
 agent runtask --task DraftReleaseNotes
 ```
 
+### View the list of available Docs Agent tasks
+
 To see the list of all tasks available in your project, run
-`agent runtask` without any arguments.
+`agent runtask` without any arguments:
+
+```sh
+agent runtask
+```
+
+### Ask the model to run a task using custom input
+
+If a task script has a `<INPUT>` placeholder, you can provide
+a custom input string to the task:
+
+```sh
+agent runtask --task <TASK> --custom_input <INPUT_STRING>
+```
+
+For example:
+
+```sh
+agent runtask --task IndexPageGenerator --custom_input ~/my_example/docs/development/
+```
 
 ## Managing online corpora
 

examples/gemini/python/docs-agent/docs_agent/agents/docs_agent.py

@@ -359,7 +359,11 @@ def add_custom_instruction_to_context(self, condition, context):
 
     # Return true if the aqa model used in this Docs Agent setup
     def check_if_aqa_is_used(self):
-        if self.config.models.language_model == "models/aqa":
+        if (
+            self.config.models.language_model == "models/aqa"
+            or self.config.app_mode == "full"
+            or self.config.app_mode == "widget-pro"
+        ):
             return True
         else:
             return False
@@ -525,16 +529,21 @@ def ask_content_model_with_context_prompt(
         try:
             response = ""
             if model == "gemini-pro":
-                response = self.gemini_pro.generate_content(contents=new_prompt)
+                response = self.gemini_pro.generate_content(
+                    contents=new_prompt, log_level=self.config.log_level
+                )
             elif model == "gemini-1.5":
-                response = self.gemini_15.generate_content(contents=new_prompt)
+                response = self.gemini_15.generate_content(
+                    contents=new_prompt, log_level=self.config.log_level
+                )
             else:
-                response = self.gemini.generate_content(contents=new_prompt)
-        except:
+                response = self.gemini.generate_content(
+                    contents=new_prompt, log_level=self.config.log_level
+                )
+        except Exception as e:
+            print("Error in generate_content()")
+            print(e)
             return self.config.conditions.model_error_message, new_prompt
-        # for chunk in response:
-        #     if str(chunk.candidates[0].content) == "":
-        #         return self.config.conditions.model_error_message, new_prompt
         return response, new_prompt
 
     # Use this method for talking to a Gemini content model
@@ -549,9 +558,6 @@ def ask_content_model_to_use_file(self, prompt: str, file: str):
             response = self.gemini.generate_content(contents=new_prompt)
         except google.api_core.exceptions.InvalidArgument:
             return self.config.conditions.model_error_message
-        # for chunk in response:
-        #     if str(chunk.candidates[0].content) == "":
-        #         return self.config.conditions.model_error_message
         return response
 
     # Use this method for asking a Gemini content model for fact-checking.