libappstream is not cross-platform anymore

[Jehan]

Hello!

I am GIMP maintainer. We are using the AppStream format as the metadata format for upcoming extension format (extensions can be things like plug-ins, or again themes, or brush sets, etc.). Until recently we were using appstream-glib then we moved to libappstream, since the former is in maintenance mode.

Unfortunately the MSYS2 project just told us that libappstream is not cross-platform anymore since you just ported your code to libfyaml which doesn't support Windows! 😱 For us, you can imagine this is a huge blocker since GIMP is heavily cross-platform and our extension format is also obviously meant to work on Windows.

Are there any plans regarding this? Are you planning to port libfyaml to Windows so that libappstream re-becomes cross-platform? (and I am not mentioning other OSes like *BSDs or others. The repository I found only mention Linux and macOS, but I for sure hope this is meant to work everywhere).

Thanks!

[lazka]

Upstream issue: pantoniou/libfyaml#10

[ximion]

Oh, interesting... I didn't even know libappstream already worked properly on Windows, I thought #541 was needed for that (unfortunately, no reply on that PR for a while, so it will likely be closed soon as it's stale).

There isn't anything inherently non-portable in libfyaml, in fact, it has been ported to Windows: pantoniou/libfyaml#67
So I guess that patch would need feedback addressed and be merged. Meson support apparently was also something thought about at libfyaml, which would likely also help.

The state of YAML C libraries is pretty sad unfortunately, libyaml itself doesn't receive much new development and leaves a lot of optimization potential on the table. Libfyaml has a much better API, performance advantages, YAML 1.2 support, much better control over the generated output, etc. However, even that library doesn't seem to have the same level of support it once had.

Either way, adapting the Windows support patch for libfyaml would already help, @pantoniou's feedback on that PR also makes a ton of sense to me and would shrink down the patch volume a lot (by not having API changes and redefining alloca on Windows (I had no idea that Windows doesn't have alloca! Wow...)

In any case, I'll help in ways I can (my Windows knowledge nowadays is rather limited, it has been a long time since I built anything specifically for that platform). Reaching out to the libfyaml maintainer may be useful though, there's quite a few PRs that would make sense to get merged...

What I'd rather want to avoid is porting to a different YAML library, as fyaml enabled quite a lot of code cleanup and simplifications in appstream.

[lazka]

I wonder if it would be possible/a good idea to make appstream build without yaml support as a temporary solution. As far as I see Gimp uses only the xml parts.

The only other downstream user we have is libadwaita, which as far as I see supports yaml, but doesn't use the APIs directly and the examples only use xml (speaking of xml, the C library state is also sad :) )

[ximion]

That's not possible, YAML support is deeply interconnected in the code with the XML functions, used for JSON parsing as well as other API features such as NEWS file conversion as well as machine-readable validator output functions and other stuff like that.

[ximion]

speaking of xml, the C library state is also sad :)

Yes indeed - hopefully we can get a solution for that... It's the age-old classical issue of foundational open source projects not receiving proportional support.

[Jehan]

Thanks. I'll help however I can too. A few years ago, I would have just have contributed the patch (I contributed to make appstream-glib build on Windows too, many years ago!), but these days I have too much to do and I'm tired! 😛

Also I don't use Windows either (again, I was still going the extra-mile, cross-building for Windows back then, but these days others do this just as well!).

Anyway I was hesitating suggesting making YAML support an option too. Too bad it's not possible. 😅

We'll be looking forward any unblocking on this topic in any case!

[ximion]

Anyway I was hesitating suggesting making YAML support an option too. Too bad it's not possible. 😅

Yeah, it would break the API hard, but more importantly, it would be an insane amount of work, I think way more work than just getting libfyaml to build, especially since a preliminary patch with feedback already exists for the latter.

I do have to say, while Windows isn't a priority for the library, it is also not ignored - I'll try my best to help the GIMP.
Also, in libappstream, there are Linux-only features, but those are mainly in the parts of the code where hardware support is tested for, and those are used in software-centers on Linux. Which isn't a concern (yet...) on Windows (and currently we can emit an error on Windows explaining this, which is a very safe fallback path).

[pantoniou]

Huh I didn't know libappstream used libfyaml.

For what is worth, libfyaml is not abandoned or anything.

I still actively develop it, but main development happens in a different branch, gearing up for a 1.0 release.

Yes, the windows issue is known, but I am not a windows developer, and it would take some amount of effort to port it.
I would be open to reviewing a PR that would provide support for it but has to go over a proper review first.

[ximion]

Huh I didn't know libappstream used libfyaml.

It only does since last month, replacing libyaml. Thanks for writing libfyaml, support for YAML 1.2 was really needed, and with fyaml we can also avoid copying memory three times for no reason. AppStream loads extremely large files on some systems, so this actually matters (although the duplicate-key detection slows down speed a lot, and YAML document emission is slower than with libyaml - but not by much).

For what is worth, libfyaml is not abandoned or anything.

That's great to hear! In hindsight, I should probably have reached out to you first, because AppStream depending on libfyaml definitely means increased scrutiny for the latter, as it will become part of the core/desktop package set in many distros and part of runtimes. So, a heads-up might have been nice ^^

In any case, it's for sure better than libyaml, which looks pretty abandoned (nearly no commits, zero feedback on PRs, ...).

On the Windows front, there were already people willing to work on it and submitting PRs! So, I'm not too worried, that's a really good sign :-)

[pantoniou]

@ximion Interesting, I know people keep using libfyaml but unfortunately no feedback. People only show up when something is broken :)

Well, if you do load large files you might like what's coming up...

Regarding windows, it is what it is, that platform is a pain in the neck, and just doing things for fun is impossible on it.
If anyone is willing to go through the trouble and make it work, I am certainly open to merging it, but, has to be done properly...

[pantoniou]

@ximion err, also have you seen this? FYPCF_ALLOW_DUPLICATE_KEYS ?

[ximion]

@ximion err, also have you seen this? FYPCF_ALLOW_DUPLICATE_KEYS ?

I did, I did a fair bit of performance profiling (libfyaml was insanely slow at reading data by comparison at first, until d4632d8).

Interesting, I know people keep using libfyaml but unfortunately no feedback. People only show up when something is broken :)

Hehe, classic experience for an open-source project maintainer. With some luck though, they'll show up with a patch out of nowhere that some company needed :-)

Windows is definitely the odd-one-out from all platforms one can support. But Cygwin and MinGW help a lot.