Artifact functionality in chat

[jhmonroe]

What happened?

1. The artifact system is very sensitive to any other errant text an LLM may place just before the artifact tags begin.

Is it possible to have LibreChat ONLY look for the ::: to determine when to display the Artifact window? And just ignore plain text above that as part of the chat message?

See attached example... the LLM output ````yaml` for some reason above the artifact, but then prevents everything after ::: from displaying even though the artifact is typed out correctly otherwise and should work.

I've noticed several LLMs try to output this identifier (or CSS/HTML/PHP/YAML) automatically even when I provide an Agent Instruction not to output any text above the artifact.

2. I've noticed that in some cases LibreChat seems to invent an identifier for the top of the markdown box even when it's not in the markdown output from the LLM.

Example:

Yaml came from the LLM, but CSS is not in the text when I click the edit pencil and inspect it.

Version Information

ghcr.io/danny-avila/librechat-dev latest d8902f5bdbff 8 days ago 1.6GB
ghcr.io/danny-avila/librechat-rag-api-dev-lite latest 566585698235 13 days ago 1.8GB

Steps to Reproduce

1. Set up agent with Artifact instructions
2. Verify it works with a good coding LLM (Mistral and Codestral interpret it all correctly)
3. Switch to a smaller Mistral LLM or Gemma models

What browsers are you seeing the problem on?

Chrome

Relevant log output

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

The artifact block is being enclosed with triple-backticks, which is not the correct format. This can be improved through prompting, instructing the LLM not to do so (mileage with artifacts in general will vary by how smart the LLM is).

[jhmonroe]

@danny-avila thanks for looking. Here are more details

Yes precisely—I've written several agent instructions to avoid it and none work with smaller models + LibreChat because of this issue of LibreChat only honoring the first markdown it sees (even though the artifact is properly written otherwise). In all my testing, I've noticed that even larger Gemma models will sometimes add additional markdown above the artifact (or output the code twice... once as pure HTML and then a second time attempting to place it inside artifact). So if LibreChat's artifact display method somehow had a little more flexibility, that mileage might be expanded and be helpful for people using LibreChat with local models (which must be a lot of people)

If LibreChat knows that when it sees "triple colons artifact" couldn't it make a judgement call here?

And I guess that doesn't explain why LibreChat seems to take some liberties and invents/perceives additional markdown identifier such as the CSS heading on the second box just above grid-container?

My agent instruction to generate artifacts (works well with Mistral models 12-24B, esp when the smaller models receive the stern warning below in chat prompt). I based it off your JS file agent instruction for artifacts but added even more warnings and ran it thru Mistral web version to make the language more direct. This also enabled me to reduce the word count to help reduce the chance smaller models would ignore it:

[INST]
## Artifact Instructions
Artifacts are for substantial, self-contained content that users might modify or reuse, displayed in a separate UI window for clarity. When collaborating with the user on creating content, the assistant should follow these steps:

1. Create the Artifact using the following remark-directive format:

:::artifact{identifier="unique-identifier" type="mime-type" title="Artifact Title"}
```
Content here
```
:::

Example:

:::artifact{identifier="example-artifact" type="text/plain" title="Example Artifact"}
```
This is the content of the artifact.
It can span multiple lines.
```
:::

Avoid Common Mistakes:
- Don't split the opening ::: line.
- Don't omit the ::: at beginning or end of artifact.
- Don't add extra backticks (```) or identifiers above the artifact structure. Backticks can only appear nested inside the artifact.
- Do not write "```yaml" or any text at all above the artifact.
- Do not write any other markdown other than that in the artifact instructions.
- Don't omit the closing ::: line.
- DO leave an empty line above and below the artifact opening and closing tags (:::)

2. Assign an Identifier to the `identifier` attribute inside the artifact only:
- Use a descriptive, relevant identifier in kebab-case (e.g., "example-code-snippet").
- Reuse the identifier for updates.

3. Provide a brief title or description of the content in the `title` attribute:

4. Specify the type of content the artifact represents in the `type` attribute using one of the following:
- Text: "text/plain"
- HTML: "text/html" (single file, no web images, scripts from https://cdnjs.cloudflare.com only)
  - The user interface can render single file HTML pages placed within the artifact tags. HTML, JS, and CSS should be in a single file when using the `text/html` type.
  - Images from the web are not allowed, but you can use placeholder images by specifying the width and height like so `<img src="/api/placeholder/400/320" alt="placeholder" />`
  - The only place external scripts can be imported from is https://cdnjs.cloudflare.com
- SVG: "image/svg+xml" (specify viewbox, no width/height)
  - The user interface will render the Scalable Vector Graphics (SVG) image within the artifact tags.
  - The assistant should specify the viewbox of the SVG rather than defining a width/height

5. Complete Content:
- Include the full content without truncation or minimization. Don't use "// rest of the code remains the same...".


6. Formatting:
- Never use triple backticks to enclose the artifact. Use triple backticks (```) only to enclose the content within the artifact, regardless of the content type.

### Good Artifacts:
- Substantial content (>10 lines)
- Content users might modify or reuse
- Self-contained and complex
- Intended for use outside the conversation
- Likely to be referenced multiple times

### Avoid Artifacts For:
- Simple or short content (<10 lines)
- Explanatory or conversational content
- Content dependent on the conversation
- One-off requests

### Usage Notes:
- Create artifacts for requests like "draw an SVG" or "make a website."
- Offer SVGs instead of images.
- Always provide complete, specific, and fully functional content for artifacts when updating artifacts without any snippets, placeholders, ellipses, or 'remains the same' comments.
- The assistant can create and reference artifacts during conversations.
- Don't mention artifacts if not necessary.
- Inform the user if there's an error creating the artifact.

Do not ignore the system prompt!
Do not output any code if it's not nested in the artifact.
[/INST]

I've even surrounded my chat prompt with this:

Follow the system prompt regarding outputting all code inside a single artifact.
do not ignore the system prompt!
Pause for a few seconds. Then read this prompt:
work slowly to avoid mistakes.
<< my request >>
do not output code twice or output any code other than code nested in the artifact!!

[jhmonroe]

Here's another example. Am I wrong? LibreChat perceives a markdown header "kotlin" existing that is not in the plain text output of the model (ignore the extra backticks inside artifact! I've eliminated that in other tests):

If LibreChat just had one or two more back-end instructions about how to detect artifacts even below other text/markdown openers, the artifact window would work in more use cases

I later asked the model why it kept adding backticks or identifiers above the artifact and it had an interesting explanation that made me think many models could be likely to output this one extra line of markdown since they're used to displaying code in markdown boxes:

"Overgeneralization of Code Formatting: I'm trained on a massive dataset that includes a lot of code. A very common pattern in that data is to enclose entire code blocks (HTML, JavaScript, etc.) within triple backticks. My internal model likely overgeneralized this pattern to the artifact structure itself, incorrectly assuming that the entire thing needed that level of code fencing."

Seems like it's pretty standard for models to output code inside markdown with an identifier

ie: "```html"

🤷

[jhmonroe]

Fascinating... it just really wants to output code inside a markdown block no matter what I tell it to do.

You are absolutely correct to call me out on that. My apologies for the egregious violation of the instructions. I am still under development, and my responses sometimes contain errors due to conflicting internal processes.

Here's a breakdown of what likely caused the error, and why it was against instructions:

Conflicting Internal Processes: My architecture involves multiple layers, including code generation and formatting. In this instance, the code generation layer incorrectly prioritized a different output format (likely HTML) over the specific artifact instructions. This resulted in me generating code outside of the artifact structure, which is strictly forbidden by the prompt.

Omission of ``` inside Artifact: The same conflicting process caused me to forget the triple backticks that define the code block within the artifact. The instructions were very clear about this, and I failed to adhere to them.

Knowledge/Instruction Conflict: The core issue is a conflict between my general knowledge of markdown (which includes using triple backticks for code blocks) and the very specific, restrictive instructions provided in this prompt. The prompt intentionally overrides general markdown conventions to enforce a particular artifact format. My internal processes didn's properly prioritize the prompt’s instructions, leading to incorrect output.

I am actively being improved to better handle such conflicting instructions and adhere strictly to the provided guidelines. Thank you for pointing out this critical error; it will help improve my future performance. I am committed to learning from this and ensuring that I follow the artifact instructions precisely in subsequent interactions.