Nemo v0.9.1

This is a maintenance relaese to fix a critical bug preventing the tracing visualization in the Nemo-web frontend from working.

Notable bug fixes

• The Nemo worker no longer panics when requesting information about traced predicates and the tracing visualization in Nemo-web now works as expected. (#732)

Feedback is welcome. Users are invited to report issues or get in touch through Nemo's public community chatroom.

Full Changelog: v0.9.0...v0.9.1

Nemo v0.9.0

New Features and Improvements

• Tracing large numbers of facts is now significantly faster. (#701)
• Tracing now supports visualization with nemo-web. (#701)
• SPARQL queries now allow incremental imports with new bindings derived by the program. (#703, #709)
  See the longest winning streaks example for how this can be used.
• Some filters and value transformations in imports now avoid creating intermediate values. (#715)
• If neither @output nor @export is specified, Nemo now defaults to exporting all derived predicates. Previously, nothing would have been exported. (#704)
• CSV/TSV/DSV now handles missing columns in a row more leniently, ignoring only the affected rows. (#727)
• fstring syntax now allows {{ and }} as escape sequences for single braces, e.g., f"{{}}". (#725)

Notable bug fixes

• Calling built-in functions without any arguments no longer panics. (#723)
• Multiple anonymous variables (_) in the negated part of a rule body are no longer rejected. (#707)

Moreover, significant architectural changes have been implemented to make the codebase more maintainable in critical parts.

Feedback is welcome. Users are invited to report issues or get in touch through Nemo's public community chatroom.

Full Changelog: v0.8.0...v0.9.0

Nemo v0.8.0

New Features and Improvements

• NQuads, TriG: Support writing to the default graph (#615)
  When writing NQuads and TriG files, the IRI tag:nemo:defaultgraph is interpreted as the name of the default graph.
• New LEVENSHTEIN(x, y) builtin for computing the Levenshtein distance of two strings (#625)
• Parameterised programs (#628)
  Programs can now contain parameters of the form $name, which can be specified on the command line using --param name=value. See the RDF conversion example for a demonstration.
• SPARQL imports (#621)
  The results of a SPARQL query can now be imported as a predicate.
  See the Wikidata awards example for a demonstration.
• Import from stdin (#655)
  For at most one imported predicate, an empty resource may be specified to signify that contents should be read from the standard input.
• Program transformation pipeline (#682)
  Various program transformations and validations have been consolidated into a program transformation pipeline.
• Replace deprecated rio with oxrdfio (#626)
  The library used for reading RDF files has changed. This should improve the behaviour on malformed input.
• Container images (#691)
  We now provide OCI container images on the GitHub container registry.
• The behaviour of several builtins has been changed to be more in line with the SPARQL standard.

Full Changelog: v0.7.1...v0.8.0

Nemo v0.7.1

New Features and Improvements

• Enable transparent decompression in HTTP resources (#603)
• Add builtin functions BITSHL, BITSHR, BITSHRU (https://knowsys.github.io/nemo-doc/reference/builtins/#functions-for-numbers)
• Allow binary, octal, and hexadecimal numbers in rule files (https://knowsys.github.io/nemo-doc/reference/datatypes/)

Notable Bug Fixes

• Expose import media types through wasm bindings (#595)
• Evaluation of constant expressions in facts (#602)

Full Changelog: v0.7.0...v0.7.1

Nemo v0.7.0

New Features and Improvements

• Format string syntax added (https://knowsys.github.io/nemo-doc/reference/builtins/#format-strings)
• Attributes can be attached to rules to add meta-information (https://knowsys.github.io/nemo-doc/reference/attributes/)
• Add IRI, URIENCODE and URIDECODE builtins (https://knowsys.github.io/nemo-doc/reference/builtins/)
• Add parameters to import and export directives (https://knowsys.github.io/nemo-doc/reference/imports/)
• Add option to ignore headers in DSV imports ((https://knowsys.github.io/nemo-doc/reference/imports/)
• Nemo rule model API allows to evaluate constant expressions (#564)

Notable bug fixes

• Fixed a crash occurring when reading the strings + or - from files (#587)
• Fixed a crash occurring when computation resulted in NaN or infinity (#588)

Full Changelog: v0.6.0...v0.7.0

Nemo v0.6.0

New Features and Improvements

• Major rework of the parser, allowing for more expressive error messages
• Improved Rule API
• Added REGEX function
• Added Unicode support for String functions
• Support Trace as DAG in GraphML format
• Boolean functions can now be used on the outer level in a rule body
• More accurate reporting of memory consumption
• Improve memory consumption for storing string data
• Provide timing info in the python api

Notable bux fixes

• Fix off-by-one error in SUBSTR function (#481)

Full Changelog: v0.5.1...v0.6.0

Nemo v0.5.1

New Features and Improvements

• More builtins, including: SUM, PROD, MIN, MAX, LUKA, BITAND, BITOR, BIXOR
• Some existing functions now allow arbitrary many arguments like CONCAT
• Alternative ternary version of SUBSTR added
• Automatically upcast numeric values if needed (e.g. adding an integer and a double automatically results in a double without the need to cast the integer)

Notable bug fixes

• Fixed a bug where the occurrence of multiple datatypes in a single relation could lead to wrong results (#477, #483)
• Fixed a bug that prevented an optimization in rules which restrict variables to constants (#487)
• Fixed a bug that prevented gzipped data from working when imported from an HTTP source (#486)

Full Changelog: v0.5.0...v0.5.1

Nemo v0.5.0

Version 0.5.0 of Nemo sees the fast in-memory rule reasoner with a redesigned data model that natively supports storing multiple data types. The comman-line client nmo can be used to access all functionality (use nmo --help for a brief documentation). The online documentation of Nemo covers many of the current features.

New Features and Improvements

• All supported RDF formats are now also available as output formats
• New built-in operations isInteger, isFloat, isDouble, isIri, isNumeric, isNull, isString, ABS, SQRT, NOT, fullStr, STR, SIN, COS, TAN, STRLEN, UCASE, LCASE, ROUND, CEIL, FLOOR, DATATYPE, LANG, INT, DOUBLE, FLOAT, LOG, POW, COMPARE, CONCAT, CONTAINS, SUBSTR, AND, OR, STRSTARTS, STRENDS, STRBEFORE, STRAFTER, and REM
• The predicates to trace can now be read from a file (--trace-input-file)
• Output can be directed to the standard output by specifying an empty resource, e.g., @export predicate :- csv{resource=""}

Notable bug fixes and internal improvements

• New data model handles data types natively with less memory overhead

Breaking changes

• @source directives are no longer supported, use @import instead
• @declare directives are no longer supported; appropriate types should be inferred automatically (or specified on @import)
• The --output-dir CLI option has been renamed to --export-dir
• The --input-dir CLI option has been renamed to --import-dir
• The --save-results CLI option has been replaced by the --export option, which takes the values none (do not export anything), keep (export predicates annotated with @output; this is the default behaviour), idb (export all IDB predicates), edb (export all EDB predicates), and all (export everything)
• The --write-all-idb-predicates CLI option has been removed in favour of --export idb
• The --detailed-memory and --detailed-timing options have been replaced by the --report option, which takes the values none (do not report anything), auto (print a short report if no other results are printed, otherwise do not print anything; this is the default behaviour), short (print a short report), time (print a short report and a detailed timing report), mem (print a short report and a detailed memory usage report), and all (print all reports)

Current Functionality

• Execution (materialization) of Datalog extended with stratified negation and existential rules (tuple-generating dependencies)
• Loading input data from local or remote CSV/TSV and RDF (NT, Turtle, RDF/XML, NQ, TriG) files
• Writing results to CSV files
• Compatibility with RDF and SPARQL syntax for IRIs and literals
• Datatypes integer (whole numbers), float64 (64bit floating point numbers), string (Unicode strings), and any (union type that can represent any element; default for most contexts)
• Support for built-in predicates <=, <, >=, >, !=, and = and built-in functions +, -, *, and / for types integer and float64
• Support for built-in functions isInteger, isFloat, isDouble, isIri, isNumeric, isNull, isString, ABS, SQRT, NOT, fullStr, STR, SIN, COS, TAN, STRLEN, UCASE, LCASE, ROUND, CEIL, FLOOR, DATATYPE, LANG, INT, DOUBLE, FLOAT, LOG, POW, COMPARE, CONCAT, CONTAINS, SUBSTR, AND, OR, STRSTARTS, STRENDS, STRBEFORE, STRAFTER, and REM
• Aggregation: #count, #min, #max, and #sum

Nemo v0.5.0 is built for mid-sized computing tasks that can still be processed on a normal laptop in seconds or minutes (typically hundreds of thousands to hundreds of millions of facts). In such cases, Nemo is already quite fast – at least fast enough to outperform existing free rule engines on the tests we conducted so far. Example tasks and benchmark results can be found in our sister repository Nemo examples and benchmarks.

Note that the combination of existential quantifiers and stratified negation do not have a standard semantics, and may lead to unexpected conclusions which can be avoided by careful modeling (see Ellmauthaler, Krötzsch, Mennicke; AAAI 2022).

Known Limitations

The following features are not included in v0.5.0 yet and will be added in upcoming releases:

• Support for unary - in arithmetic expressions (current workaround: use 0-?X instead of -?X)

Moreover, the documentation, though improved, is still very limited. This will be expanded gradually.

Full Changelog: v0.4.0...v0.5.0

Nemo v0.4.0

Version 0.4.0 of Nemo extends the fast in-memory rule reasoner with support for aggregation and tracing. The command-line client nmo can be used to access this functionality (use nmo --help for a brief documentation). The online documentation of Nemo covers many of the current features.

New Features and Improvements

• Aggregates in rule heads
• tracing of atoms shows why an atom was derived: e.g., the CLI option --trace 'p(42)' shows a derivation tree for atom p(42) after reasoning
• arithmetic terms can now be used in rule bodies
• inequality operator != for arithmetic comparisons
• wildcard terms: _ in an atom always matches, but does not introduce a variable binding, e.g., q(?x, ?y) :- p(?x, _, _, ?y) is equivalent to q(?x, ?y) :- p(?x, ?z, ?zz, ?y)
• loading of RDF N-Quads and TriG files
• @import syntax for more control over input formats
• @export syntax for more control over output formats (e.g., specifying the delimiter for CSV-like outputs)

Notable bug fixes and internal improvements

• New dictionary implementation significantly reduces memory consumption on large datasets

Current Functionality

• Execution (materialization) of Datalog extended with stratified negation and existential rules (tuple-generating dependencies)
• Loading input data from local or remote CSV/TSV and RDF (NT, Turtle, RDF/XML, NQ, TriG) files
• Writing results to CSV files
• Compatibility with RDF and SPARQL syntax for IRIs and literals
• Datatypes integer (whole numbers), float64 (64bit floating point numbers), string (Unicode strings), and any (union type that can represent any element; default for most contexts)
• Support for built-in predicates <=, <, >=, >, !=, and = and built-in functions +, -, *, and / for types integer and float64
• Aggregation: #count, #min, #max, and #sum

Nemo v0.4.0 is built for mid-sized computing tasks that can still be processed on a normal laptop in seconds or minutes (typically hundreds of thousands to hundreds of millions of facts). In such cases, Nemo is already quite fast – at least fast enough to outperform existing free rule engines on the tests we conducted so far. Example tasks and benchmark results can be found in our sister repository Nemo examples and benchmarks.

Note that the combination of existential quantifiers and stratified negation do not have a standard semantics, and may lead to unexpected conclusions which can be avoided by careful modeling (see Ellmauthaler, Krötzsch, Mennicke; AAAI 2022).

Known Limitations

The following features are not included in v0.4.0 yet and will be added in upcoming releases:

• Support for RDF output
• Support for unary - in arithmetic expressions (current workaround: use 0-?X instead of -?X)
• More built-in functions, especially for string data
• More control over output formats

Moreover, the documentation, though improved, is still very limited. This will be expanded gradually.

Full Changelog: v0.3.0...v0.4.0

Nemo v0.3.0

Version 0.3.0 of Nemo extends the fast in-memory rule reasoner with support for more arithmetic features and extended data sources and formats. The command-line client nmo can be used to access this functionality (use nmo --help for a brief documentation). The online documentation of Nemo covers many of the current features.

New Features and Improvements

The following new features have been added in Nemo v0.3.0:

• Support for arithmetic operators (+, -, *, /), so far only in rule heads
• Extended arithmetic comparisons between variables (can now include two variables, not just variables and constants)
• New datatype string to load plain string data (generally faster loading than the full RDF term parsing of type any, which also can hold string data, as before)
• Support for online sources: just use URLs (with http:/https:) in @source declarations
• Support for RDF files in Turtle and RDF/XML format through the existing load-rdf method (any @base declaration from the rules file will be used as default base IRI in such formats), enabled by Oxigraph's Rio RDF library

Notable bug fixes and internal improvements:

• @output (the directive that specifies which predicate data to return) is now allowed in any place of the file (not just as the end)

Current Functionality

At this version, Nemo therefore includes support for the following features:

• Execution (materialization) of Datalog extended with stratified negation and existential rules (tuple-generating dependencies)
• Loading input data from local or remote CSV/TSV and RDF (NT, Turtle, RDF/XML) files
• Writing results to CSV files
• Compatibility with RDF and SPARQL syntax for IRIs and literals
• Datatypes integer (whole numbers), float64 (64bit floating point numbers), string (Unicode strings), and any (union type that can represent any element; default for most contexts)
• Support for built-in predicates <=, <, >=, >, and = and built-in functions +, -, *, and / (in rule heads) for types integer and float64

Nemo v0.3.0 is built for mid-sized computing tasks that can still be processed on a normal laptop in seconds or minutes (typically hundreds of thousands to hundreds of millions of facts). In such cases, Nemo is already quite fast – at least fast enough to outperform existing free rule engines on the tests we conducted so far. Example tasks and benchmark results can be found in our sister repository Nemo examples and benchmarks.

Note that the combination of existential quantifiers and stratified negation do not have a standard semantics, and may lead to unexpected conclusions which can be avoided by careful modeling (see Ellmauthaler, Krötzsch, Mennicke; AAAI 2022).

Known Limitations

The following features are not included in v0.3.0 yet and will be added in upcoming releases:

• Support for RDF output
• Support for arithmetic functions in rule bodies (currently only in heads)
• Support for unary - in arithmetic expressions (current workaround: use 0-?X instead of -?X)
• More built-in functions, especially for string data, and support for aggregates
• More control over output formats

Moreover, the documentation, though improved, is still very limited. This will be expanded gradually.

Breaking change: CSV default type

The default type for CSV data sources has changed from any to string. For existing rules files, this may require small changes in rules or in CSV files. For example, using type any as before, a quoted RDF string literal would be encoded in CSV as """Hello world!""", where the outer " delimit the value, and the inner "" encode single quotes (in CSV). The result is a string with the characters Hello world!. When using type string instead, the whole value found in CSV will be considered the contents of a string, so the above fragment would produce the string with content "Hello world!" (with the quotes as first and last symbol). To get the same as before, one would leave away the inner "" in CSV, which were only needed to mark the value as string literal in the context of RDF (where many other types of terms are allowed).

One can also revert to the old behaviour by declaring type any explicitly for CSV sources, but this will in general be slower (RDF term parsing and normalization is more work than plain string reading). Finally, one could also read a CSV file that contains RDF-formatted data as strings and work with the plain strings throughout the program. This might require changes to constant that appear in the rules file (e.g., if a former string constant "Hello world!" needs to become "\"Hello world!\"" to match the verbatim data when read as string. Note that this disables all normalization, e.g., "Hello world!" will not be equal to "Hello world!"^^<http://www.w3.org/2001/XMLSchema#string> as it would be in RDF.

Feedback and issue reports are welcome!