Some user feedback on ODK/OBO (shortened)

[matentzn]

Some feedback on our recent survey of OBO tools (shortened to essence)

• I still struggle to find the tools and documentation about the tools
• ODK documentation needs improving. It's probably all there but hard to find answers
• The OBO ontologies and development tools are vital for my work. A lot of that work is now in the form of converting the ontologies into knowledge graphs. It would great to see an OBO initiative that focused specifically on delivering "the OBO knowledge graph" as a standalone entity that integrated all the great work within each ontology. This would reduce a lot of redundant external work. Thanks for all you do OBO people!
• All of my current ontologies use ROBOT, but I would like to revamp my directory structure to make use of the ODK. However, so far I've only looked at the ODK but not used it as I was a little unclear as to the best way to swap over an existing repository. A webinar / workshop on that would be awesome.
• I'm also looking to create two forms of our release for terms4FAIRskills - one with and one without BFO / ULOs (our users don't want the ULOs in their version, but OBO Foundry would like us to produce one with ULOs to aid OBO integration, which we agree with). So, workshops or drop-ins on those subjects would be very useful. 3. Thank you for the fantastic tools!
• I see the ODK Full version has Docker; but is there an ODK lite version anymore? I really think the ODK needs a lite version that can later be converted to full: the very simple ""get going with my first ontology creation and editing project""; and the existing Docker version. One having the complexity of docker and jenkins, and the other without. One with github persona baked in, the other without. Both providing the same file structure for release and edit files, and an ability to run the quality control tests. I say this because the number of new OBOFoundry ontologies I've been involved in helping start never seem to have researchers who can handle ODK complexity as it stands. Maybe the two versions can simply be distinguished by a minimum necessary configuration file? I will try to set up the latest version with a new ontology to verify the perception that only expert developers can feel confident setting it up."
• In-depth reference for the variables in the yaml building template. List of Ontologies which lead to issues during import (foodon, ncbitaxon, chebi, go; some rectifying settings for yaml building tutorial), either due to poorly encoded formats (OBO) or due to the amount of memory required. Personally, I ran a Azure virtual machine with 32G memory to run the required build.

[gouttegd]

I see the ODK Full version has Docker; but is there an ODK lite version anymore? I really think the ODK needs a lite version that can later be converted to full: the very simple ""get going with my first ontology creation and editing project""; and the existing Docker version.

The more I think about this, the more I think it would be a good idea.

This may sound strange given that I’m the one who put in the effort to “revive” the odklite image last year, but I am less and less convinced that there’s even a point in the odklite image. It is one-third the size of odkfull, but still it’s already nearly 1 GB – so it’s not so much “lite” as it is “slightly less bloated than odkfull”. And the ODK-generated repositories are configured to use odkfull by default, so odklite is only used by people who know about its existence and who made the effort of switching to it – I’m not even sure those people even exist…

What could be more useful, I think (just daydreaming here, not pretending this is something we can realistically achieve – maybe we can, maybe we can’t), would be for “odklite” to be a native software package, that is, something that doesn’t depend on Docker. That package would mostly provide Robot, which is the one tool that most of the “standard” pipelines tend to depend on, and possibly a few Python modules (at least what’s necessary for the odk.py script itself to run) – enough stuff to initialize a ODK-generated ontology and run the standard workflows, without ever using Docker. Then, for people developing more complex ontologies (and who need more specialized tools such as Konclude, Soufflé, more Python packages, etc.), we would still provide the odkfull Docker image. Switching from using the natively installed odklite to the Docker-powered odkfull would mainly involve updating the run.sh script (when using odklite, it would call make directly; when using odkfull, it would invoke Docker).

This would obviously be a major undertaking, and I am not saying we should attempt to do it. But I think it is worth thinking about it.

[matentzn]

At least the compressed odklite image is ~ 500MB, which is I think as good as we could hope for:
https://hub.docker.com/r/obolibrary/odklite/tags

But yes, a non-docker setup would be awesome: #383.

Hard to imagine to get that platform independent though given the various uses of sed, grep etc for which the interfaces are widely different on different OS versions..

[gouttegd]

And the compressed odkfull image is ~1.15 GB, so it’s unclear to me that odklite is really worth the trouble… I am not convinced there are many cases where downloading 500 MB is fine, but downloading 1 GB is not.

As for platform independence, first thing is that we should probably exclude Windows altogether – Windows users would just have to keep using the Docker version as they do now. Sorry for them, but I just don’t think we can do a native Windows package, at least not with the team we have now (me for example: nowadays I know next to nothing about modern Windows; last time I did development on a Windows machine was with Windows XP and the .Net framework version 1.1, back in 2005 or so).

That leaves us having to deal with MacOS, GNU/Linux and other Unix-like systems. And with that, two problems to solve:

• We would need to make sure our Make rules do not rely on any GNUism or any other non-POSIX behaviour. Not straightforward, but doable.
• We would need to make sure we have all the dependencies we need. That, I think, is the biggest problem.

For GNU/Linux systems, we could focus first on Debian-based systems, and provide Deb packages. Then we could use the built-in dependency declaration mechanisms. At first those packages would be provided directly by us, and then we could try to get them merged directly into Debian (could be hard to achieve but would be great; imagine being able to start an ODK tutorial by just saying “all you have to do to get started is apt-get install odklite” 😍 ).

For MacOS, maybe we can try to provide HomeBrew or MacPorts packages. Then we would also benefit from the dependency resolutions mechanisms that are built in those systems. Of course we would not really provide native packages then – we would still need to tell people to install HomeBrew or MacPorts first, and that’s hardly better than telling them they must download Docker. But I don’t know enough about MacOS native software packaging to have any better idea for now.

[matentzn]

This sounds somehow awesome. And a bit nuts :D ODK is converging slowly slowly, so maybe we can consider it as a next step in the evolution.. I cant see it yet, but the appeal is real!

[gouttegd]

To be clear the vision is not fully formed in my mind either – far from it! But with the “standard workflows” being increasingly dependent on a smaller set of tools (mostly Robot) I do feel that a Docker-less approach is slowly coming within reach, and it’s worth to think about it.

[matentzn]

I see kind of the opposite trajectory with an increasing intricate set of python tooling interwoven with ROBOT and shell pipelines - but I still digg the idea of something w/o docker. Its an annoying overhead.

[gouttegd]

But those Python tools are not (yet?) used too much in the “standard” workflows, are they? The default, ODK-generated Makefile still mostly uses ROBOT. If I recall correctly, the only notable Python tool used in that Makefile is dosdp.

But I get your point, a ROBOT-centric approach may not be workable in the future if we start relying on more Python tools than we currently do even in the standard workflow – something that might indeed happen (sssom, oaklib, etc.).

(Very very very very crazy idea: embedding Python – or, more correctly, Jython – directly into ROBOT.)

[cmungall]

Maybe the next step in the evolution should be web based. No need for curators to install anything other than git clients and protege. Maybe not even that for template driven development. Doesn’t need to be fancy ui. No need for persistent store, the backend is GitHub driven by API calls. Maybe droid is mostly there?

…

On Sun, May 22, 2022 at 8:00 AM Nico Matentzoglu ***@***.***> wrote: This sounds somehow awesome. And a bit nuts :D ODK is converging slowly slowly, so maybe we can consider it as a next step in the evolution.. I cant see it yet, but the appeal is real! — Reply to this email directly, view it on GitHub <#573 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMMOMM2PZUNYPMW22ZBV3VLJDYNANCNFSM5QJDKFNQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jamesaoverton]

I gave a demo for the OBI developers today of a web interface for editing ontologies and ontology-driven datasets. It includes template editing (with validation), easy imports from large upstream ontologies, and will soon include Manchester editing. The demo was local, but the system is designed to work on the web. Happy to talk more about that, or show it off.

[matentzn]

@cmungall if you find us some place to host DROID we can start deploying and testing all the awesome tools @jamesaoverton has cooked up recently, but in particular, an instance of droid. The Web based solution is only blocked by us not having a clear strategy for deploying data intensive apps in Monarch!