Minimum and recommended hardware and software requirements

[WolfgangFahl]

In the qlever script #562 i am trying to gather some intelligence on the environment along what i understood are the minimum and recommmended hardware and sofware requirements. I'd love to know the "official" requirements so that the documentation might state these and the script might do checks on the feasibility of an environment.

So far I understand that a tested environment is something like:

• OS: Ubuntu 20.04 LTS
• RAM: 128 GB (recommended) - X GB (minimum)
• Disk: 4 TB (recommended maybe rotating ) - X TB (mimimum)
• ulimit -Sn set to 2^20=1024*1024=1048576 recommended - minimum depends on size of data to be indexed - 1 file as per configured number of triples per file

For the qlever script (which might need renaming to e.g. qlever-demo) the following software is needed:

needed software (Ubuntu)
docker → /usr/bin/docker ✅
top → /usr/bin/top ✅
df → /usr/bin/df ✅
jq → /usr/bin/jq ✅
lsb_release → /usr/bin/lsb_release ✅
free → /usr/bin/free ✅

needed software (MacOS)
docker → /usr/local/bin/docker ✅
top → /usr/bin/top ✅
df → /bin/df ✅
jq → /opt/local/bin/jq ✅
sw_vers → /usr/bin/sw_vers ✅

i intend to use jq for modifying the json config files via command line and included it to the list

[joka921]

I want to add that the hardware requirements highly depend on the dataset. Your recommendations mirror our experiences with indexing and running the full Wikidata. If your dataset is smaller, of course you can use less RAM and disk drive.

Concerning the OS: QLever should run natively on Ubuntu 18.04 and newer (we have Dockerfiles building on 18.04 LTS, 20.04 LTS and 21.10, so you should be able to manually perform the steps that these Dockerfiles do, to run QLever outside of Docker.

When running QLever inside Docker, also many different operating systems should work. I think there might be some limitations on a minimal version of docker and the linux kernel that is used by docker (which is the native kernel on a linux host OS, on Windows and MacOs it depends on the docker implementation, I am no expert there).

[hannahbast]

I agree that GitHub Discussions is a better place for this than GitHub Issues.

[WolfgangFahl]

When i did my first Wikidata import trials in 2018 i had a side script that would track memory and disk usage while the tdbloader indexer was running.
I also analyzed the log files using a spreadsheet tool with a script

#!/bin/bash
# 2018-01-06
# 
# get the relevant log informatin
# e.g. lazegraph.out:loading: 5876854 stmts added in 644.857 secs, rate= 9113 
# 
log() {
 grep added nohup.out
}
echo '"mill stmts" secs rate"' > stats.csv
echo "=SUMME(A3:A1000)/1000/1000;=SUMME(B3:B1000);=RUNDEN(A2/B2*1000000)" >> stats.csv
log | cut -f2,6,9 -d" " | sed 's/ /;/g' | sed 's/,//g' | sed 's/\.[[:digit:]]\+//g'  >> stats.csv
ci -l -m "auto checkin" stats.csv
scp stats.csv capri:/bitplan/Projekte/2018/Infrastruktur2018
libreoffice --calc stats.csv&

A similar script that combines the intelligence on the progress might be useful for qlever indexing.

[WolfgangFahl]

The 32 GB RAM machine behaved strangely:

creating wikidata index started at Mi 2. Feb 21:40:32 CET 2022�[0m
2022-02-02 20:40:33.292	- INFO:  �[1mQLever IndexBuilder, compiled on Jan 29 2022 20:19:12�[22m
2022-02-02 20:40:33.312	- INFO:  You specified the input format: TTL
2022-02-02 20:40:33.314	- INFO:  You specified "locale = en_US" and "ignore-punctuation = 1"
2022-02-02 20:40:33.314	- INFO:  You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files (see qlever/docs on GitHub)
2022-02-02 20:40:33.314	- INFO:  You specified "num-triples-per-partial-vocab = 10,000,000", choose a lower value if the index builder runs out of memory
2022-02-02 20:40:33.315	- INFO:  Processing input triples from /dev/stdin ...
2022-02-02 20:45:06.667	- INFO:  Input triples processed: 100,000,000
2022-02-02 20:49:23.607	- INFO:  Input triples processed: 200,000,000
...
2022-02-03 08:46:24.880	- INFO:  Input triples processed: 16,900,000,000
2022-02-03 08:48:46.031	- INFO:  Done, total number of triples read: 16,979,762,129 [may contain duplicates]
2022-02-03 08:48:46.031	- INFO:  Number of QLever-internal triples created: 10,093,310,880 [may contain duplicates]
2022-02-03 08:48:46.034	- INFO:  Merging partial vocabularies in byte order (internal only) ...
2022-02-03 08:50:11.109	- INFO:  Words merged: 100,000,000
...
2022-02-03 09:06:37.634	- INFO:  Words merged: 900,000,000
2022-02-03 09:07:07.721	- INFO:  Number of words in internal vocabulary: 938,508,130
2022-02-03 09:07:07.721	- INFO:  Merging partial vocabularies in Unicode order (internal and external) ...
2022-02-03 09:10:58.801	- INFO:  Words merged: 100,000,000
...
2022-02-03 11:15:02.417	- INFO:  Words merged: 3,200,000,000
2022-02-03 11:16:50.078	- INFO:  Number of words in external vocabulary: 2,289,905,916
2022-02-03 11:16:50.081	- INFO:  Removing temporary files ...
2022-02-03 11:17:21.520	- INFO:  Converting external vocabulary to binary format ...
2022-02-03 11:55:33.781	- INFO:  Converting triples from local IDs to global IDs ...
2022-02-03 11:56:05.539	- INFO:  Triples converted: 100,000,000
2022-02-03 11:56:32.460	- INFO:  Triples converted: 200,000,000
...
2022-02-03 14:09:23.237	- INFO:  Triples converted: 27,000,000,000
2022-02-03 14:09:45.972	- INFO:  Done, total number of triples converted: 27,073,073,009
2022-02-03 14:09:45.976	- INFO:  Building prefix tree from internal vocabulary ...
2022-02-03 14:10:07.086	- INFO:  Words processed: 100,000,000
2022-02-03 14:10:45.984	- INFO:  Words processed: 200,000,000
2022-02-03 14:11:28.436	- INFO:  Words processed: 300,000,000
2022-02-03 14:12:10.503	- INFO:  Words processed: 400,000,000
2022-02-03 14:13:01.209	- INFO:  Words processed: 500,000,000

at this point the system got unresponsive. I had to switch it off since it didn't react anymore to ssh commnands or any interaction on the console via mouse or keyboard.

[hannahbast]

@WolfgangFahl When a process uses much more RAM than a machine has, the machine first starts swapping and at some point the process either crashes (the "best" case because then you can continue to work with the machine) or the machine freezes (the "worst" case because there is not much else you can do then except turning the machine off and on again).

[WolfgangFahl]

@hannahbast thx for the quick response. I assume that monitoring the RAM from an external process or from within the indexing might be helpful as stated in my other comment. Predicting how much RAM is necessary by trial and error might be tedious and costly. In virtual machine environment such as Java the RAM to be used can be preconfigured. I don't know how this might be possible in a c++ based solution. Given that there is now an SSD available i wonder whether a large swap space on the SSD might do it if the RAM is not used very often.

[hannahbast]

@WolfgangFahl Yes, QLever will eventually also run with little RAM also for very large knowledge graphs. It's not a principle problem. It's just not our highest priority right now because if you want fast indexing and query times for a large dataset (like Wikidata), you need a reasonable amount of RAM anyway. Note that Virtuoso and Blazegraph are also very slow when you don't give them enough RAM. Also, 128 GB of RAM are pretty standard and cheap by now, and it won't be long before any reasonable PC will have at least that much RAM.