How to properly use include-read

[marin-jurjevic]

Hi,

I have a project where we have following structure

.
├── README.md
├── doc
│   ├── conf.py
│   ├── doctrees
│   ├── Doxyfile
│   ├── _doxygen
│   ├── _extensions
│   ├── getting_started
│             └── index.rst

Inside getting_started/index.rst is following

.. _getting_started:

Getting Started
###############

.. include:: ../../README.md
   :parser: myst_parser.sphinx_

We have README.md which is markdown + some raw html for aligning images etc. I want to keep this as is since all images and paths are properly set so it can be rendered in GitLab without errors.
The problem is that when sphinx processes README.md, all images have incorrect paths since they are relative in Markdown.

We've been using sphinx 5.3.0. I've tried enabling app.connect("source-read", process_source_read) to preprocess links and correct them by replacing path prefix. When I do this I only get raw RST (Markdown seems to not be processed yet).
After that I found that sphinx 7.2.5 added include-read hook, which works the same way as source-read.
When I connect the include-read hook it doesn't execute.
But, when I remove :parser: myst_parser.sphinx_ the hook works and I get it to execute, but then README.md is processed as regular RST.

My question is, why does :parser: myst_parser.sphinx_ break include-read functionality and what's the best way to solve this. To sum up and avoid XY problem:

What is the correct way to include existing README.md file without changing it so that it can get properly rendered on GitLab web interface and correctly processed by sphinx in resulting HTML documentation?

[marin-jurjevic]

After debugging this for couple of hours I found out that docutils never executes part of code for emiting include-read event while processing Include directive.

Code responsible for this is located in docutils package:

https://github.com/docutils/docutils/blob/f725d4a2742f997a95b9de53553642044df475ab/docutils/docutils/parsers/rst/directives/misc.py#L187-L205

Due to 'parser' option being set, code never reaches bottom lines which would call patched _insert_input defined in Sphinx:
https://github.com/sphinx-doc/sphinx/blob/bb74aec2b6aa1179868d83134013450c9ff9d4d6/sphinx/directives/other.py#L368-L417.

As I don't have full understand of the way parsing logic is implemented, I'm not sure if this should be corrected. I see two possible solutions:

1. Leave docutils package as is and emit event directly from MystParser.parse() function
2. Contribute to docutils and remedy this by calling state_machine.insert_input when parser option is set

[chrisjsewell]

Thanks for looking into it, I guess (2) is the better option, but would maybe consider (1)