Full Text Search

Couchbase Lite has some fairly simple but useful support for full-text search, i.e. the kind of search you do in Spotlight or Google.

As of this writing (September 21, 2013) full-text search is still experimental. You'll need to check out the fulltext branch to use it.

Indexing Text

Any view can index text instead of the regular JSON keys. To do so, you make its map block emit a special key created by the CBLTextKey() function, whose parameter is the string to be indexed.

Example:

[[db viewNamed: @"blogText"] setMapBlock: MAPBLOCK({
    if ([doc[@"type"] == "blog") {
        NSString* body = stripHTMLTags(doc[@"body"]);
        emit(CBLTextKey(body), doc[@"title"]);
    }
}) reduceBlock: NULL version: @"1"];

Note: Don't emit both full-text keys and regular JSON keys in the same view! Use separate views instead.

Searching For Text

CBLQuery has some special properties for full-text searches; they're declared in the header CBLQuery+FullTextSearch.h (which is already included by CouchbaseLite.h.)

The most important one is fullTextQuery, an NSString containing the search term(s). Setting this to a non-nil value changes the query to full-text.

The query language is defined by the SQLite Full-Text Search (FTS) extension, and is documented on the SQLite website. The gist of it is:

• Search terms are either individual words, or phrases delimited by double-quotes.
• Appending a * to a search term denotes a prefix search that matches any word beginning with that term.
• When multiple search terms are separated by spaces, all of them have to match -- it's an implicit "AND" conjunction.
• You can also put the words AND or OR (in all caps) between terms.
• The word NOT (in all caps) before a term negates it: only rows that don't include it will be returned.
• The word NEAR (in all caps) between terms is like AND but also requires that the matches be near each other.
• Multiple terms or expressions can be wrapped in parentheses for grouping.

Example:

CBLQuery* query = [[db viewNamed: @"blogText"] query];
query.fullTextQuery = @"Couchbase NEAR (Lite OR mobile)";
query.fullTextSnippets = YES;   // enables snippets; see next example

Getting The Results

A full-text CBLQuery returns its results as instances of CBLFullTextQueryRow, a subclass of CBLQueryRow with some extra accessors.

• The fullText property returns the text that was indexed.
• The matchCount property returns the number of matches that were found in the text.
• -textRangeOfMatch: returns an NSRange giving the character range in the fullText of a match.
• -termIndexOfMatch: indicates which term in the query was matched. The terms in the queries are numbered, left to right, starting at 0. (Terms that have the NOT operator applied are ignored.)
• -snippetWithWordStart:wordEnd: returns an brief substring of the full text that includes the matched terms (or as many as fit). It's intended to be shown in a compact search-results list in your app's UI. The wordStart and wordEnd strings can be used to highlight the matched terms: they're inserted before and after every appearance of a matched term. For instance, you could use [ and ], or <b> and </b> if you're displaying results as HTML. (Note: To enable snippets, you have to set the query's fullTextSnippets property.)

By default, query rows are returned in descending order of relevance (by a fairly simple/naïve definition of "relevance".) If you don't care about this ranking, you can make the search a bit faster by setting the query's fullTextRanking property to NO.

Example

for (CBLFullTextQueryRow* row in [query rows]) {
    NSLog(@"Title: %@", row.value)  // the map fn emits the post title as the value
    NSLog(@"Text:  %@", [row snippetWithWordStart: @"[" wordEnd: @"]"]);
}

Limitations

• You can't combine key-based and full-text queries in the same view. A view's emit calls should either emit regular keys or the special text objects, not some of each.
• For this reason, the key-based properties of CBLQuery have no effect in a full-text search: startKey, endKey, startKeyDocID, endKeyDocID, keys.
• Full-text queries don't support reducing. They don't call the reduce block, and the reduce-based properties have no effect: mapOnly, groupLevel.

Issues

Poor support for Asian text

Full-text search relies heavily on tokenizing -- breaking text into words -- and the tokenizer we're using only supports languages that use space characters between words; that means it can't find word breaks in Asian languages, which typically aren't written with spaces. The result is that indexing won't work, unless spaces are explicitly inserted into the text to be indexed.

(There do exist tokenizers for CJK languages; here's a good discussion.)

Can't configure language-specific stemming

Stemming means ignoring grammatical variations in words, like pluralization and verb tenses, for purposes of matching, so that a query for "dog" can match "dogs", and "searching" can match "searches". The current tokenizer we use has language-specific stemming rules for many Western languages, but the language has to be specified when the FTS data table is created, and Couchbase Lite has no API for that yet.