TripleSource as Iterator?

[KonradHoeffner]

I want to parallelize some CPU-bound processing of several GB of N-Triples using rayon, but it seems like the TripleSource I get from the NTriplesParser does not implement one of the iterator yielding traits nor offer another methods to obtain an iterator. Is that intended or could that be added?

[pchampin]

That is intended. All Rio-based parsers are focusing on limiting the number of allocations: they "deliver" a triple by calling the callback function, then reuse the buffers in which the subject/predicate/object of that triple are stored for the subject/predicate/object of the next triple. It means that the lifetime of a triple has to stop before the lifetime of the next triple starts.

This is incompatible with Iterator, that assume that all the items yielded by the iterator can coexist (think about the collect method).

A simple way to get an iterator out of NTriplesParsers (or friends) is to map each triple to a long-lived copy (e.g. [SimpleTerm<'static>; 3]), doing the extra allocations that parsers was designed to avoid. The result of the map_triples methods should be both a TripleSource and an Iterator.

[KonradHoeffner]

Thank you for the clarification! As running out of memory is a frequent problem for me with large N-Triples files it seems like going through iterators is not the correct approach after all.
I will try to find another way of increasing parallelism.

[pchampin]

Thank you for the clarification! As running out of memory is a frequent problem for me with large N-Triples files it seems like going through iterators is not the correct approach after all.

It is not necessarily wrong either... If you "consume" (i.e. free) each item soon after it is yielded by the iterator, you can avoid running out of memory...

In https://github.com/pchampin/sophia-cli, I use exactly the strategy described above: the ephemeral triples/quads generated by the different parsers are systematically converted quads of ArcTerms, which can then travel across different threads (via an MPSC queue).

I also use rayon in sophia-cli, but not on quad iterators directly...