Consider compressing all release artefacts

[gouttegd]

All release artefacts are currently published uncompressed. It is possibly to ask for the “main” release artefact to be compressed (option gzip_main), but this merely add a compressed version of the release artefact on top of the uncompressed one.

This is a considerable waste of storage space (and waste of time, considering the time needed to download the artefacts), given that most release formats are basically text-based formats (XML, JSON, OBO) and are therefore highly amenable to efficient compression. Even the binary SQLite format can be compressed quite well.

For example, with the latest standard release of CL:

Release artefact	Uncompressed size	Gzip-compressed	XZ-compressed
cl.owl	61 MB	3.7 MB (~16⨉)	2.5 MB (~24⨉)
cl.obo	16 MB	2.4 MB (~6⨉)	1.5 MB (~10⨉)
cl.json	35 MB	2.6 MB (~13⨉)	1.8 MB (~19⨉)
cl.db	491 MB	94 MB (~5⨉)	48 MB (~10⨉)

Maybe nowadays 61 MB is not so much of a big deal, but when counting all release artefacts in all formats, a full-blown CL release amounts to more than 700 MB of perfectly compressible files (and that’s before we enable the production of DB files). This could probably be cut down to less than 100 MB if release artefacts were compressed with Gzip, and maybe even less than 50 MB if they were compressed with XZ.¹

Yes, I realise that switching to compressed files is going to break a lot of existing pipelines that expect to be able to find the released version of the XYZ ontology at http://purl.obolibrary.org/obo/zyx.owl and not at http://purl.obolibrary.org/obo/xyz.owl.gz. However this is still something that should be considered. Releasing uncompressed files is nuts.

---

¹Though it must be noted that ROBOT does not support reading from XZ-compressed files, se we should probably either stick to Gzip or add XZ support to ROBOT.

[matentzn]

What you are saying is sane but it will break a lot of stuff. ROBOT is not the only tool reading from PURLs, so are dozens of python and R tools. While this wont fix the storage issue maybe we can start relieving some of the stress on the networking system by publishing the gziped release files alongside the main ontologies and starts migrating pipelines towards those.

[gouttegd]

What you are saying is sane but it will break a lot of stuff.

I know, I know. That’s why I said this is something that should be “considered”, not something that should be “done”. :)

we can start relieving some of the stress on the networking system by publishing the gziped release files alongside the main ontologies and starts migrating pipelines towards those.

That sounds like a reasonable path.

Though in my opinion the ideal solution would be for the purl.obolibrary.org server to handle the issue, if it could be configured to uncompress files on the fly when someone requests http://purl.obolibrary.org/obo/xyz.owl and the XYZ ontology only provides a http://purl.obolibrary.org/obo/xyz.owl.gz – but this requires turning that server from a machine that does nothing more than serving redirects, to an uncompressing reverse proxy. Not sure if this is compatible with how that machine is currently dimensioned.

[gouttegd]

Can we agree that SemSQL files at least should be systematically compressed?

They can be up to 10 times larger than RDF/XML files (see again CL’s example: 491 MB for uncompressed cl.db).

And there’s no way that compressing them could break an existing pipeline, since the feature to publish DB files has only existed since ODK 1.6 and AFAIK no ontology has started to use that feature yet (CL will be the first).

[matentzn]

Absolutely no objections - there is zero advantage of having the semsql files uncompressed as there is no tool that can handle semsql dbs from URLs

[gouttegd]

I am considering replacing the gzip_main option (which triggers the additional release of a compressed version of the main product) by something that would allow the project owner to configure exactly for which formats they require compression, and whether they want to publish both compressed and uncompressed versions or not.

Thinking out loud, but for now in my mind it would look like this:

export_formats:
  - owl
  - db[compressed]
  - json[compressed,uncompressed]

That is, the name of the format (owl, json, obo, db) can be followed by a list of options (enclosed within [] brackets), where for now the options would be:

• uncompressed: Make and publish uncompressed versions of the artefacts in that format
• compressed: Make and publish compressed versions of the artefacts in that format

For owl, obo, and json, the default behaviour if no option is specified would be uncompressed (same as current behaviour, so no breakage). For db, the default option would be compressed.

Later, we could switch the default options for the owl, obo, and json formats to compressed,uncompressed, so that we would by default publish both compress and uncompressed versions (so, still no breakage for pipelines that depend on uncompressed versions).

(And in an ideal world when pipelines are not left to rot and are updated by diligent people who care after them, later then we would be able to switch the default to compressed for all formats…)

[matentzn]

Super cool suggestion, fully on board.

[cmungall]

The only thing wrong with this proposal is that we didn't implement it 20 years ago.

I like how this conversation is going, let's start by making sure all core ontologies are available as gz, such that people can start converting pipelines, with a view to eventual deprecation of uncompressed. Great to have the support in ODK to help with this.

Can we make a dashboard check for gz availability?

Some specific comments:

1. uncompress on-the-fly: nice in principle but we can't commit to that, these servers are absolutely hammered by brain-dead bots that try and download the same ontology 1000s of times per hour
2. semsql: should already be .gz only there may some stale shadows, will check

[gouttegd]

uncompress on-the-fly: nice in principle but we can't commit to that, these servers are absolutely hammered by brain-dead bots that try and download the same ontology 1000s of times per hour

I agree, I think it’s an idea that is simply not compatible with the era of AI crawling bots. Better not to spend any time (nor any money!) on it.

[kltm]

I would agree that on-the-fly decompression is probably a non-starter for how we have things currently setup on purl.obolibrary.org. To give a specific example of the silly that we deal with, we had a single IP was requesting and downloading a single file 250k times a day--that's like 3 times a second, continuously for however long they had been doing it.

With actors like this floating around, I think we'd either have to commit to self-serving (using our own S3 backed delivery plus Cloudflare, monitoring, and extra overheads), partnering, or try and get people moved over to better files and accept some breakage along the way.

If the desire for data keeps increasing, we may have to revisit more assumptions about how we do things.

[gouttegd]

To give a specific example of the silly that we deal with, we had a single IP was requesting and downloading a single file 250k times a day

It’s the clever bots that are the real problem, not the silly ones.

The bots that are silly enough to do their stuff from a single IP are easy to deal with – just ban any IP that makes too many requests.

The clever bots never use the same address more than a handful of times – much more harder, if not downright impossible, to block.

Example:

A few weeks ago, my personal web server had received about ~56,000 requests over the course of ~5 hours, at a very regular rate of 3 requests per seconds, hitting every single link and in particular asking for the history of every single line on every single file of every single projects hosted on my Gitea instance. (3 requests per second may not sound like a lot, but when every request triggers a costly git blame operation, this is more than enough to bring a server to its knees.)

Those 56K requests came from 54,786 different IP addresses. 53,606 of those addresses were used only once, 1,129 were used twice, and the remaining 51 were used between 3 and 8 times.

None of those requests self-identified as a bot. They all had various user-agents made to pretend they came from legitimate humans browsing the web with legitimate browsers. Because of course it is completely credible that 54K different humans would visit every single link on my website at a regular rate of three visits per second… 🙄

Those 54K IP addresses came from 21,865 different networks belonging to 7,139 different AS. Only a handful of autonomous systems made more than a dozen of requests – most of them made 2 to 4 requests at most.

This can only be the behaviour of a botnet. That’s the only way to get so many different IPs from so many different networks and AS.

That’s what you use when you know that your “data collection” practices are highly dubious and that people would stop you if they could – you make sure they cannot stop you.

There’s even a open market for this kind of scummy services nowadays, with providers that don’t even bother to hide such as Bright Data.

[kltm]

Naturally: when dealing with this tide, both clever and silly are issues, especially as we're representing a community and enabling massive charges and/or overloads to be passed on to others. As well (on another thread), this is why I think depending on services like Cloudflare, which are better setup to detect and filter this kind of behavior, are unfortunately going to be needed more moving forward.

That said, to keep things on thread, anything that can be done to help mitigate this flood for our community, even if slowly, like starting to migrate to compressed files, I think is worthwhile.

[gouttegd]

That said, to keep things on thread, anything that can be done to help mitigate this flood for our community, even if slowly, like starting to migrate to compressed files

From the ODK’s point of view, there are two things that can be done:

(1) Producing (and publishing) compressed files (only, or in addition to uncompressed versions). This is what this issue is about.

(2) When downloading a remote ontology, (a) try to download a compressed version first, and (b) fallback to downloading an uncompressed version if we get a 404 on the URL for the compressed one. This might be better discussed in a new issue.