-
Notifications
You must be signed in to change notification settings - Fork 81
IBX-8689: AI Actions documentation #2473
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
2188724
IBX-8689: AI Actions documentation
dabrt 5db48c2
Change wording
dabrt 9eaf572
PG update
dabrt 351fe83
Tyypo fix
dabrt 9abac00
Remove unnecessary article stems
dabrt af41b70
Implement changes after the release
dabrt 5f3dde3
Update guide diagram
dabrt 938fa1e
Add a release notes section
dabrt 341e838
Apply suggestions from code review
dabrt 72927f7
Implement formatting changes
dabrt fd4b0b5
More formatting changes
dabrt 4dfaf1b
Update docs/ai_actions/ai_actions_guide.md
dabrt 31c4482
Fix a link in index file
dabrt 89fc0d5
Apply suggestions from code review
dabrt 90629bc
Implement reviewer changes
dabrt 53b6713
Update docs/ai_actions/ai_actions_guide.md
dabrt 028cc1f
Merge branch 'master' into IBX-8689
dabrt 5c66139
Merge branch 'master' into IBX-8689
dabrt b4714ef
Implement reviewer comments
dabrt ba1c07b
Implement reviewer comments
dabrt c8ee8ed
IBX-8689: Format add_ai_actions.sql (#2536)
adriendupuis 1213f01
Update code_samples/ai_actions/config/postgresql/add_ai_actions.sql
dabrt File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| --- | ||
| description: AI Actions help editors by automating repetitive tasks. | ||
| page_type: landing_page | ||
| --- | ||
|
|
||
| # AI Actions | ||
|
|
||
| Enhance the usability and flexibility of [[= product_name =]] by installing the AI Actions add-on that can automate various tasks, for example, generate alt text for images. | ||
| Potentially, you can use AI for transforming text passages, generating illustrations for your articles based on a prompt, and so on. | ||
|
|
||
| [[= cards([ | ||
| "ai_actions/ai_actions_guide", | ||
| "ai_actions/install_ai_actions", | ||
| "ai_actions/extend_ai_actions" | ||
| ], columns=4) =]] | ||
|
|
||
| ## Alt text generation | ||
|
|
||
| [[= cards([ | ||
| "ai_actions/alt_text/enable_alt_text_generation", | ||
| "ai_actions/alt_text/configure_alt_text_generation", | ||
| "ai_actions/alt_text/customize_alt_text_generation" | ||
| ], columns=4) =]] | ||
|
|
||
| ## Explore AI Actions API | ||
|
|
||
| ... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| --- | ||
| description: AI Actions add-on helps editors by automating repetitive tasks. | ||
| --- | ||
|
|
||
| # AI Actions product guide | ||
|
|
||
| ## What are AI Actions | ||
|
|
||
| In the evolving landscape of digital experience, artificial intelligence becomes more and more important by enhancing user interaction and automating complex tasks. | ||
| To harness AI's potential, [[= product_name =]] introduces th AI Actions add-on, which you can install and use to automate various time-consuming tasks, for example, generating alt text for images. | ||
|
|
||
| Built upon the PHP framework, AI Actions offer an extensible solution for integrating services provided by AI engines into your workflows, all managed through a user-friendly interface. | ||
|
|
||
| You can extend the add-on and make it do different things, for example, transform text passages, or generate illustrations for your articles based on a prompt. | ||
| The possibilities are endless. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Availability | ||
|
|
||
| AI Actios are available as an add-on to the v.4.6.x version of [[= product_name =]], regardless of its edition. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| To use it's capabilities you must install the package and perform initial configuration. | ||
|
|
||
| ## How it works | ||
|
|
||
| ### The core concepts | ||
|
|
||
| #### Actions | ||
|
|
||
| Actions are tasks or functions that the AI is set to perform. | ||
| Each Action is a combination of an action type and an action configuration. | ||
| Action types define what kind of task the AI will perform, while the action configuration specifies how the task should be executed. | ||
| This clear separation allows for a flexible system where actions can be easily created, managed, and customized. | ||
|
|
||
| #### Action types | ||
|
|
||
| Action types are predefined by developers within YAML configuration. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| Each action type defines the structure and nature of the task that the AI engine performs. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| Action types could be designed to generate alt text for images, summarize a passage of text, or even translate content into another language. | ||
| By defining action types in YAML, developers can create a wide range of functionalities that can be easily deployed within the application. | ||
|
|
||
| #### Action configurations | ||
|
|
||
| AAction configurations let administrators employ action types to generate actions. | ||
| Action configurations are managed in the [Admin Panel](admin_panel.md), and allow administrators to customize and fine-tune the behavior of each action. | ||
| This might involve setting parameters for the AI, defining specific inputs, or configuring how the output should be handled. | ||
| By adjusting these configurations, administrators can ensure that the actions are tailored to meet their requirements. | ||
|
|
||
| #### Handlers | ||
|
|
||
| Once an action is defined and configured, it must be executed, and this is where handlers come into play. | ||
| Handlers are pieces of PHP code that are responsible for resolving an action. | ||
| Each handler is designed to work with a specific AI engine and action type pair. | ||
| Handlers may include hardcoded prompts for conversational AI engines like ChatGPT, or they may operate without prompts for other types of AI. | ||
| Handlers take configurations defined in the action type and action configuration, combine it with user input and any predefined settings or prompts, and pass this information to the AI engine for processing. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ### Triggering actions from the UI | ||
|
|
||
| Among other components, AI Actions add-on brings action management in Admin Panel and alt-text generation feature in image management UI. | ||
| Both these areas are user-friendly and well integrated with the application’s UI. | ||
| Administrators can manage atcion configurations with ease, while end-users can trigger actions by simply clicking a button. | ||
| Processes are straightforward and intuitive, ensuring that users can quickly and easily achieve their desired outcomes. | ||
|
|
||
| ### Triggering actions programmatically | ||
|
|
||
| AI Actions add-on exposes an API that allows for programmatic execution of actions. | ||
| With the API, developers can automate tasks and execute actions on batches of content by integrating them into workflows. | ||
| By issuing commands through the API, developers can trigger actions based on external events: | ||
| ... | ||
|
|
||
| ## Capabilities and use cases | ||
|
|
||
| ### Management | ||
|
|
||
| ... | ||
|
|
||
| ### Extensibility | ||
|
|
||
| Built-in action types offer a good starting point, but the real power of AI Actions lies in extensibility. | ||
| Extending AI Actions opens up new possibilities for content management and editing. | ||
| Developers can extend the feature by adding new action types that use existing AI engines or even integrating additional AI services. | ||
| This involves defining a new action type in YAML, writing a handler that communicates with the new service or engine, and creating a form for configuring the options that extends the action configuration form shown in the Admin Panel. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| For example, a developer could write a handler that uses an AI service to shorten or expand text passages, or generate illustrations based on a body of an article. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ### Use cases | ||
|
|
||
| #### ... | ||
|
|
||
| #### Generating Alt Text | ||
|
|
||
| Out of the box, the [[= product_name_base =]] AI Actions add-on comes with the generating alt text for images capability. | ||
| Organizations can benefit from it's use by improving accessibility and SEO. | ||
| Once the feature is configured, editors can generate alt text for images they upload to the system by clicking one button, while administrators can use the API to run a batch process against a larger collection of illustrations. | ||
| Before you can start generating alt text, some preliminary setup is required, which includes configuring access to an AI engine. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| --- | ||
| description: Configure alt text generation to match your needs. | ||
| --- | ||
|
|
||
| # Configuree alternative text generation | ||
|
|
||
| Once you've enabled alt text generation, you can fine-tune the output to match technical requirements of your application or your company's voice and style. | ||
| You can modify the settings of individual action configurations by selecting the preferred language, tone, and temperature, along with other customizable settings to ensure the generated alt text meets your specific needs. | ||
|
|
||
| ## Configure action configurations | ||
|
|
||
| ... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| --- | ||
| description: Add custom features to alternative text generation for images. | ||
| --- | ||
|
|
||
| # Customize alternative text generation | ||
|
|
||
| If you need more control over the alt text generation feature, you can customize, for example, by integrating a different AI engine, and modifying the default behavior. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Add custom glossaries | ||
|
|
||
| You can enhance the relevance of alt text or apply jargon specific for your company. | ||
| One way to do it is by creating lists of specific keywords or phrases that should either always be included (whitelist) or avoided (blacklist) in the generated descriptions. | ||
|
|
||
| ... | ||
|
|
||
| ## Integrate a different AI engine | ||
|
|
||
| If the default AI engine doesn’t meet your requirements, you can connect to a different AI service. | ||
| To do it, you must configure API endpoints, input formats, and authentication details for the new engine. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ... | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| --- | ||
| description: Configure your application to enable generating alternative text for images. | ||
| --- | ||
|
|
||
| # Enable alternative text generation | ||
|
|
||
| With this feature you can enhance accessibility and streamline your content creation process by generating alternative text for images in `ezimage` and `ezimageasset` fields by using the image analysis AI. | ||
| Before you can use the feature, you must enable automated alt text generation by setting up a connection with an external AI service. | ||
|
|
||
| ## Get authentication parameters | ||
|
|
||
| First, either you or another [[= product_name_base =]] user responsible for managing the [[= product_name =]] instance must ... | ||
|
|
||
| ## Ensure you account has credits | ||
|
|
||
| ... | ||
|
|
||
| ## Set up API key | ||
|
|
||
| When you receive the credentials, add them to your configuration. | ||
| In the root folder of your project, edit the `<ADD FILE NAME>` file by adding the following lines with your customer ID and license key: | ||
|
|
||
| ``` | ||
| ... | ||
| ... | ||
| ``` | ||
|
|
||
| !!! note "Configuring user credentials for different engines" | ||
|
|
||
| If your installation supports [multiple AI engines](extend_ai_actions.md) that perform the same AI action, you must provide credentials for each of these engines. | ||
| Otherwise some of the AI Actions will remain inactive. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## ... | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| description: Extend AI Actions by connecting to other engines and adding new capabilities. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| --- | ||
|
|
||
| # Extend AI Actions | ||
|
|
||
| By extending AI Actions you can enhance the attractiveness of mundane content management and editing tasks and make them less burdensome. | ||
| You can start from adding new action types that use the existing AI engine, and evolve to integrating additional AI services with completely new areas of application. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| For example, you could write a handler that connects to a translation model and use it to translate your website on-the-fly, or generate illustrations based on a body of an article. | ||
|
|
||
| ## ... | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| description: Install the AI Actions add-on. | ||
| --- | ||
|
|
||
| # Install AI Actions | ||
|
|
||
| AI Actions are available as an add-on to the v.4.6.x version of [[= product_name =]] or higher, regardless of its edition. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| To use this feature you must first install the add-on and then configure it YAML configuration. | ||
|
|
||
| ## Install AI Actions | ||
|
|
||
| Ibexa Engage is available from v4.6.x of [[= product_name =]]. | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| Run the following command to install the bundle: | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| `composer require ibexa/connector-ai` | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| This command adds the code, Twig templates and configuration files required for using AI Actions, and modifies the permission system to account for the new functionality. | ||
|
|
||
| To check for the presence of AI Actions in your application, run the following command: | ||
|
|
||
| `composer show | grep "ibexa/connector-ai"` | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Prepare configuration files | ||
|
|
||
| When you install AI Actions add-on, it modifies the YAML configuration files by adding configuration keys similar to this example: | ||
dabrt marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ```yaml | ||
| ... | ||
|
|
||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.