Heuristics when datafiles uses block scalar headers (aka | ) for string or direct (quoted)

[seichter]

Not really a bug report, rather a request for clarification because the documentation doesn't cover this.

When storing some longer strings (actually some markdown snippets) I found that these will be stored in YAML with the block scalar header. However, it somehow also included a block chomping indicator.

literature: |-
  Den Studierenden ....

in other parts it uses raw strings.

desc: "Die Studierenden sollen ... "

So, both have their reasons. However, it would be great to know the underlying heuristics when what is being used. Or, even better the type-annotation is extended to include instructions on how to serialize the strings.

It would be great if this could be clarified or documented somehow.

[UPDATE]

To clarify: this behavior only happens when datafile writes data into an existing YAML file where the block scalar header already exists for this item.

[jacebrowning]

In general, the goal is for Datafiles to write YAML that is optimized for storing in Git and minimize the diff to just content changes. I believe the YAML library (ruamel.yaml) will also attempt to preserve the existing structure as much as possible, which explains your updated comment.

If you can provide some examples where the initial file created by Datafiles feels in conflict with the stated goal, I can add some more test cases and attempt to coerce the YAML appropriately.

[seichter]

Thanks, same goal here: storing in git, but I would assume I can use full UTF-8 and want some readability. I currently bootstrap without datafiles. To illustrate my point a slightly modified YAML Formatter. Would you consider a PR with an API design that adds functionality to modify the behavior of the underlying YAML backend?

from io import StringIO

import ipsum
import log
from datafiles import datafile, formats, types
from datafiles.formats import Formatter
from ruamel.yaml import YAML as _YAML


def str_representer(dumper, data):
    if len(data.splitlines()) > 1:
        return dumper.represent_scalar("tag:yaml.org,2002:str", data, style="|")
    return dumper.represent_scalar("tag:yaml.org,2002:str", data)


class YAML_LongStr(Formatter):
    """Formatter for (round-trip) YAML Ain't Markup Language."""

    @classmethod
    def extensions(cls):
        return {"", ".yml", ".yaml"}

    @classmethod
    def deserialize(cls, file_object):
        yaml = _YAML()
        yaml.preserve_quotes = True  # type: ignore
        try:
            return yaml.load(file_object)
        except NotImplementedError as e:
            log.error(str(e))
            return {}

    @classmethod
    def serialize(cls, data):
        yaml = _YAML()
        yaml.register_class(types.List)
        yaml.register_class(types.Dict)

        # as an idea ... from here
        yaml.indent(mapping=2, sequence=4, offset=2)

        yaml.representer.add_representer(str, str_representer)
        # ... to here should be configurable without deriving a new Formatter

        stream = StringIO()
        yaml.dump(data, stream)
        text = stream.getvalue()

        if text.startswith("  "):
            return text[2:].replace("\n  ", "\n")

        if text == "{}\n":
            return ""

        return text.replace("- \n", "-\n")


for e in set(YAML_LongStr.extensions()):
    formats.register(e, YAML_LongStr)


@datafile("items/{self.name}.yaml")
class Item:
    name: str
    long_desc: str
    more_desc: str


def main():
    model = ipsum.load_model("en")
    p = model.generate_paragraphs(3)

    item = Item(
        name="test1",
        long_desc="\n".join(p),
        more_desc=" ".join(p),
    )

    item.datafile.save()


if __name__ == "__main__":
    main()

[jacebrowning]

Yes, I might consider a PR but for your long_desc field you may want to use the Text type, which forces the LiteralScalarString under the hood. You could also register a custom type to control the exact serialization. Do either of those options seem to accomplish what you want?