Normalize Markdown in `.pot` files

[mgeisler]

When mdbook-xgettext extracts translatable text, it would be great if it could normalize the strings. This would make it possible for us to reformat the entire course without fearing that the translations get destroyed while doing so.

The normalization would take Markdown like this

# This is a heading

This is another heading
=======================

A _little_
paragraph.

```rust,editable
fn main() {
    println!("Hello world!");
}
```

* First
* Second

and turn it into these messages in the .pot file:

• "This is a heading" (atx heading is stripped)
• "This is another heading" (setext heading is stripped)
• "A _little_ paragraph." (soft-wrapped lines are unfolded)
• "fn main() {\n println!("Hello world!");\n}" (info string is stripped, we should instead use a #, flag)
• "First" (bullet point extracted individually)
• "Second"

Like in google/comprehensive-rust#318, we should do this in a step-by-step fashion and make sure to apply the transformations to the existing translations. It would also be good if we have a way to let translators update their not-yet-submitted translations.

[mgeisler]

When writing this, it occured to me that we can get to the end-state of nicely formatted Markdown by running a formatter on

• The .md files
• The msgid fields in all .po files
• The msgstr fields in all .po files

If we do this in lock-step, we ought to end up with a lossless process. I was experimenting with this over the weekend: the msgfilter program can process the msgstr fields, but we need our own little helper to do the same with the msgid fields.

I was using https://dprint.dev/ for the formatting since it seems fast and extensible. It can format code blocks inside Markdown which is something I really want 😄

[jooyunghan]

Hey Martin,
While updating Korean last week, i found a few missing statements in msgstr entries. Is it from any recent updates of xgettext?

[mgeisler]

While updating Korean last week, i found a few missing statements in msgstr entries. Is it from any recent updates of xgettext?

A few entries were lost and changed as part of google/comprehensive-rust#449 — @djmitche would know the details 😄

[djmitche]

Yeah, that was a somewhat lossy process. Hopefully those appear in fuzzy or old messages?

[djmitche]

@mgeisler I'm not sure what you mean about the formatter. Can you describe that in more detail?

[mgeisler]

Can you describe that in more detail?

Sure! My thinking is that we can safely format the Markdown files if we we know that it won't create more/fewer entries in the PO files when mdbook-xgettext is executed on the formatted files.

To avoid losing translations, we'll then run the same formatting on all msgid fields.

We can already safely run the formatter on the msgstr entries today with msgfilter and that might be something we should encourage translators to do to get smaller diffs.

[djmitche]

I don't think I'm going to get a chance to work on this issue soon.

[simonsan]

I was using https://dprint.dev/ for the formatting since it seems fast and extensible. It can format code blocks inside Markdown which is something I really want 😄

https://rust-lang.github.io/mdBook/format/mdbook.html?highlight=code#inserting-runnable-rust-files

I'm still thinking if it wouldn't be better to use rustfmt in that case and just refactor the markdown files to embed the code from rs files. This would make it possible to have a crate next to the documentation/tutorial and be able to check, fix, fmt and test it. Need to think about further implications. But it would at least separate the code from the text – with its own dis-/advantages for sure - which could make it easier to translate the text that is needed. because right now I see the code blocks showing up as well in the translations.

[djmitche]

I see the appeal, but I'm also wary of making it too easy to add too much code to a slide. I think a wide-open .rs file in a text editor invites adding extra lines.

[simonsan]

Yesterday, I started the translation of rust-unofficial/patterns to German with cloud-translate. And I think the initial state could be much better with markdown normalization. Because I often end up manually going over the entries due to \r and \n messing up the English grammatic for auto-translation. I think this feature would be really nice to have to make adoption much easier! Especially if you have set the max-line-length to 80 ... 😅