Restore backend-based KWIC download

[arildm]

This code was removed in c878551 because it was broken and we don't know how to test it. I think the Kielipankki team (@janiemi and others) will be looking at re-implementing it soon.

[janiemi]

Yes, I think we intend to re-implement the backend-based KWIC download, although I’m afraid that “soon” may be rather relative here.

What do you think are the merits of the backend-based KWIC download, as opposed to the frontend-based one you are using in Språkbanken’s Korp? I mean, would it be better to try to re-implement key features of the backend-based solution in the frontend?

I think you can add new formats to the backend-based KWIC download relatively easily. (At least if you know the code.) You can also customize the exact formats. And sometimes you might wish to use the download facility directly, without the frontend. However, I find the backend code rather complex, even though I’ve written it myself.

[arildm]

I don't know much about the backend-based download. I suppose it queries the Korp backend and converts the response data to a given format?

Using the download API directly would probably be useful in some cases. I imagine someone with programming skills wanting to make a large number of searches and have the result in a specific format. Perhaps one way to go would be to implement conversion in the Korp backend, and specify it with /query?...&format=nooj etc.

Another advantage with a backend-based conversion (perhaps especially as a part of the Korp backend API) would be that you could perhaps more easily offer a download of the entire result, not just the current page. (But the current solution doesn't seem to do that, if I understand correctly?)

[janiemi]

I now recall that downloading all the hits was a possible use case of yours (Språkbanken’s) for the backend-based KWIC download.

You’re right: also Kielipankki Korp frontend only offers to download the visible KWIC page. Although it may in principle possible to download all the hits of a query directly from the backend, in the current implementation it only works for modestly-sized results, as you cannot prevent timeouts in the same way as for JSON output. Our production Korp server has a one-minute timeout (from a proxy), and within that time you may be able to get some 20,000 hits, but it also depends on the complexity of the query.

I asked GitHub Copilot (with GPT-4.1) about how to avoid timeouts in this case. Based on its answer, I think a better way to implement the functionality would be to use some kind of a job queue. In this approach, one request would initiate generating the data in the background and return a response containing a job id or a URL to poll for the status of the file, and then another request would be used to check if the file is available and to eventually retrieve the file. Copilot suggested using Celery or RQ but I’m wondering if you could implement such functionality in a simpler way without either of them.

Using a job queue might also help improve the frontend user experience, as currently the user gets no indication of whether the download is being generated or not, or if the request timed out.

The old implementation accepts either the query parameters, in which case it redoes the query, or the JSON result directly as POST data. In fact, until very recently, we were only using the former, redoing the query, but passing the JSON result turned out to be significantly faster in many cases.

Although generating the result in a different format could probably be integrated into the /query endpoint, I don’t know if it would be good or not, in particular considering the job-queue approach.

I’ve implemented a proof of concept with our enhanced backend plugin facility, so that the KWIC download is a plugin implementing a new endpoint. (It required some changes to the backend code to support endpoints returning something else than JSON.) However, it is only a port of the old approach to the current Korp, so I’m yet to investigate and implement the job-queue approach.