Markdown URLs with non-file extensions are incorrectly recognized as citations

[EMjetrot]

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

I use a mixture of PDF and markdown files as data sources, but noticed that links in the markdown text get interpreted as citations. Here is an example of some markdown with a link inside (ignore the danish language):

I kraft af dit ansættelsesforhold i styrelsen indsamler, opbevarer og behandler Koncern HR en række personlige oplysninger om dig. Du kan læse meget mere herom lige her [Sådan behandler Koncern HR persondata om dig](https://somefile.pdf).

Since the title [Sådan behandler Koncern HR persondata om dig] of the link uses square brackets like citations, the title wrongly ends up being listed among the sources while the link doesn't get formatted correctly in the answer.

Any log messages given by the failure

None

Expected/desired behavior

If we assume that a citation is always a file, we can modify the code to check whether the citation candidate contains a valid file extension. Only candidates with a recognized file extension will be treated as citations, while any other text enclosed in square brackets will remain unchanged.

Here is the code from line 32 in ./app/frontend/src/components/Answer/AnswerParser.tsx before the change:

const fragments: string[] = parts.map((part, index) => {
        if (index % 2 === 0) {
            return part;
        } else {
            let citationIndex: number;
            if (citations.indexOf(part) !== -1) {
                citationIndex = citations.indexOf(part) + 1;
            } else {
                citations.push(part);
                citationIndex = citations.length;
            }

            const path = getCitationFilePath(part);

            return renderToStaticMarkup(
                <a className="supContainer" title={part} onClick={() => onCitationClicked(path)}>
                    <sup>{citationIndex}</sup>
                </a>
            );
        }
    });

And here is the changed code where citation candidates are checked for common file extensions:

const fragments: string[] = parts.map((part, index) => {

        if (index % 2 === 0) {
            // This is text outside square brackets (regular text)
            return part;
        } else {
            // This is text inside square brackets (citation or markdown URL)
            // Check if the citation ends with a valid file extension
            const validFileExtensions = [
                ".pdf",
                ".html",
                ".docx",
                ".pptx",
                ".xlsx",
                ".jpg",
                ".jpeg",
                ".png",
                ".bpm",
                ".tiff",
                ".heiff",
                ".txt",
                ".json",
                ".csv",
                ".md"
            ];

            // Check if the part is a valid citation file
            const citationIsFile = validFileExtensions.some(ext => part.trim().toLowerCase().includes(ext));

            if (!citationIsFile) {
                // Return the part with square brackets, as it originally was
                return `[${part}]`;
            }

            // Handle citation: either it exists already or is new
            let citationIndex: number;
            if (citations.indexOf(part) !== -1) {
                citationIndex = citations.indexOf(part) + 1;
            } else {
                citations.push(part);
                citationIndex = citations.length;
            }

            const path = getCitationFilePath(part);

            // Return the citation as a clickable link with superscript
            return renderToStaticMarkup(
                <a className="supContainer" title={part} onClick={() => onCitationClicked(path)}>
                    <sup>{citationIndex}</sup>
                </a>
            );
        }
    });

This change ensures that markdown links are displayed correctly, and the citations section only includes valid source files.

Would it make sense to implement this code change in the repo?

OS and Version?

macOS (Sierra)

azd version?

azd version 1.10.1 (commit 31409a33266fb4a5fdbb644bc83988e725d6c7c9)

Versions

I'm running the release titled "2024-08-23: Optional speech output is now on-demand"

Thanks!

Thank you so much for this awesome repo!

[pamelafox]

I've seen similar issues reported in the past. The other option we considered was changing the citation to use double square brackets, since I've been told that square brackets tend to overlap too much with syntax from other languages/areas. What do you think of that?

[EMjetrot]

Double square brackets would be a lot more simple fix and I realize now that the fix I proposed doesn't work as intended if the URL in a link in a markdown file contains one of the file extensions we check for.

I asked ChatGPT for examples of literature or coding languages where double square brackets are used, just to make sure that our attempt to solve the issue with links in Markdown, doesn't become a problem for other types of text that include double square brackets. It does seem like double square brackets are used in several coding languages (Bash, Lue, R, Matlab/Octave, MediaWiki Syntax and Javascript/Typescript) and are also used in mathematical notation (Iverson brackets) and quantum error-correcting code descriptions. So I guess it is possible that double square brackets could be within the text ingested to the backend search index and therefore returned by the LLM and processed as a citation in the AnswerParser.tsx. On the other hand it seems like triple square brackets are very unlikely to appear.

If we were to test double or triple square brackets - would it just be a change in this last line of the system prompt in chatreadretrieveread.py?

Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, for example [info1.txt]. on't combine sources, list each source separately, for example [info1.txt][info2.pdf].

... and a change in the regex in line 30 of AnswerParser.tsx?

const parts = parsedAnswer.split(/\[([^\]]+)\]/g);

[egor-yudkin]

Using double (or even triple) brackets sounds like a good idea!
I'm getting similar issue with regular PDF files when the content has some placeholders which authors are using single brackets for.

[pamelafox]

Merged #2056 which should fix this.