Skip to content

Updates to indexing #334

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 19, 2019
Merged

Updates to indexing #334

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
873d702
* Indexing on `@type` requires `@type` to be either `@id` or `@vocab`,
gkellogg Nov 15, 2019
408e31b
* Indexing on an arbitrary property, not just `@index`.
gkellogg Nov 18, 2019
10f4180
Fix lint and string interpolation issues.
davidlehn Nov 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,15 @@
- Expanding the value of a graph container which is already a graph object
generates a recursive graph object.
- Compacting multiple nodes in a graph container places them in `@included`.
- Indexing on "@type" requires "@type" to be either "@id" or "@vocab",
and defaults to "@id".

### Changed
- Default processing mode changed to json-ld-1.1. Allows a 1.1 context to be
used after non-1.1 contexts.
- `@vocab` can be relative or a Compact IRI in 1.1, resolved against either a previous `@vocab`,
`@base` or document base.
- Indexing on an arbitrary property, not just "@index".
- `@vocab` can be relative or a Compact IRI in 1.1, resolved against either
a previous `@vocab`, `@base` or document base.
- Better checking of absolute IRIs.
- Terms that begin with a ':' are not considered absolute or compact IRIs.
- Don't use terms with `"@prefix": false` or expanded term definitions to construct compact IRIs.
Expand Down
43 changes: 41 additions & 2 deletions lib/compact.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -521,7 +521,33 @@ api.compact = async ({
}
key = expandedItem['@language'];
} else if(container.includes('@index')) {
key = expandedItem['@index'];
const indexKey = _getContextValue(
activeCtx, itemActiveProperty, '@index') || '@index';
const containerKey = api.compactIri(
{activeCtx, iri: indexKey, vocab: true});
if(indexKey === '@index') {
key = expandedItem['@index'];
delete compactedItem[containerKey];
} else {
let others;
[key, ...others] = _asArray(compactedItem[indexKey] || []);
if(!_isString(key)) {
// Will use @none if it isn't a string.
key = null;
} else {
switch(others.length) {
case 0:
delete compactedItem[indexKey];
break;
case 1:
compactedItem[indexKey] = others[0];
break;
default:
compactedItem[indexKey] = others;
break;
}
}
}
} else if(container.includes('@id')) {
const idKey = api.compactIri({activeCtx, iri: '@id', vocab: true});
key = compactedItem[idKey];
Expand All @@ -545,6 +571,19 @@ api.compact = async ({
compactedItem[typeKey] = types;
break;
}

// If compactedItem contains a single entry
// whose key maps to @id, recompact without @type
if(Object.keys(compactedItem).length === 1 &&
'@id' in expandedItem) {
compactedItem = await api.compact({
Copy link
Member

@dlongley dlongley Nov 15, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just get the alias for @id here and pull out the value compactedItem[alias] without having to run compact again? Or is it the case that it could have used a different value based on the presence of @type (via scoped contexts or something?)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not easily, as if reduced to a node reference, it can be turned into a string. If not, it remains a node object. We could conceivably implement that logic there, but that's what expand is for, and it should be pretty quick in that case.

activeCtx,
activeProperty: itemActiveProperty,
element: {'@id': expandedItem['@id']},
options,
compactionMap
});
}
}

// if compacting this value which has no key, index on @none
Expand Down Expand Up @@ -1063,7 +1102,7 @@ function _selectTerm(

// determine prefs for @id based on whether or not value compacts to a term
if((typeOrLanguageValue === '@id' || typeOrLanguageValue === '@reverse') &&
_isSubjectReference(value)) {
_isObject(value) && '@id' in value) {
// prefer @reverse first
if(typeOrLanguageValue === '@reverse') {
prefs.push('@reverse');
Expand Down
34 changes: 33 additions & 1 deletion lib/context.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,7 @@ api.createTermDefinition = (

// JSON-LD 1.1 support
if(api.processingMode(activeCtx, 1.1)) {
validKeys.push('@context', '@nest', '@prefix', '@protected');
validKeys.push('@context', '@index', '@nest', '@prefix', '@protected');
}

for(const kw in value) {
Expand Down Expand Up @@ -560,6 +560,21 @@ api.createTermDefinition = (
// otherwise, container may also include @set
isValid &= container.length <= (hasSet ? 2 : 1);
}

if(container.includes('@type')) {
// If mapping does not have an @type,
// set it to @id
mapping['@type'] = mapping['@type'] || '@id';

// type mapping must be either @id or @vocab
if(!['@id', '@vocab'].includes(mapping['@type'])) {
throw new JsonLdError(
'Invalid JSON-LD syntax; container: @type requires @type to be ' +
'@id or @vocab.',
'jsonld.SyntaxError',
{code: 'invalid type mapping', context: localCtx});
}
}
} else {
// in JSON-LD 1.0, container must not be an array (it must be a string,
// which is one of the validContainers)
Expand Down Expand Up @@ -595,6 +610,23 @@ api.createTermDefinition = (
mapping['@container'] = container;
}

// property indexing
if('@index' in value) {
if(!('@container' in value) || !mapping['@container'].includes('@index')) {
throw new JsonLdError(
'Invalid JSON-LD syntax; @index without @index in @container: ' +
`"${value['@index']}" on term "${term}".`, 'jsonld.SyntaxError',
{code: 'invalid term definition', context: localCtx});
}
if(!_isString(value['@index']) || value['@index'].indexOf('@') === 0) {
throw new JsonLdError(
'Invalid JSON-LD syntax; @index must expand to an IRI: ' +
`"${value['@index']}" on term "${term}".`, 'jsonld.SyntaxError',
{code: 'invalid term definition', context: localCtx});
}
mapping['@index'] = value['@index'];
}

// scoped contexts
if('@context' in value) {
mapping['@context'] = value['@context'];
Expand Down
56 changes: 45 additions & 11 deletions lib/expand.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -610,14 +610,19 @@ async function _expandObject({
} else if(container.includes('@index') && _isObject(value)) {
// handle index container (skip if value is not an object)
const asGraph = container.includes('@graph');
const indexKey = _getContextValue(termCtx, key, '@index') || '@index';
const propertyIndex = indexKey !== '@index' &&
_expandIri(activeCtx, indexKey, {vocab: true}, options);

expandedValue = await _expandIndexMap({
activeCtx: termCtx,
options,
activeProperty: key,
value,
expansionMap,
asGraph,
indexKey: '@index'
indexKey,
propertyIndex
});
} else if(container.includes('@id') && _isObject(value)) {
// handle id container (skip if value is not an object)
Expand Down Expand Up @@ -890,7 +895,7 @@ function _expandLanguageMap(activeCtx, languageMap, options) {

async function _expandIndexMap(
{activeCtx, options, activeProperty, value, expansionMap, asGraph,
indexKey}) {
indexKey, propertyIndex}) {
const rval = [];
const keys = Object.keys(value).sort();
const isTypeIndex = indexKey === '@type';
Expand All @@ -913,15 +918,6 @@ async function _expandIndexMap(
val = [val];
}

// expand for @type, but also for @none
const expandedKey = _expandIri(activeCtx, key, {vocab: true}, options);
if(indexKey === '@id') {
// expand document relative
key = _expandIri(activeCtx, key, {base: true}, options);
} else if(isTypeIndex) {
key = expandedKey;
}

val = await api.expand({
activeCtx,
activeProperty,
Expand All @@ -931,6 +927,27 @@ async function _expandIndexMap(
insideIndex: true,
expansionMap
});

// expand for @type, but also for @none
let expandedKey;
if(propertyIndex) {
if(key === '@none') {
expandedKey = '@none';
} else {
expandedKey = _expandValue(
{activeCtx, activeProperty: indexKey, value: key, options});
}
} else {
expandedKey = _expandIri(activeCtx, key, {vocab: true}, options);
}

if(indexKey === '@id') {
// expand document relative
key = _expandIri(activeCtx, key, {base: true}, options);
} else if(isTypeIndex) {
key = expandedKey;
}

for(let item of val) {
// If this is also a @graph container, turn items into graphs
if(asGraph && !_isGraph(item)) {
Expand All @@ -944,6 +961,23 @@ async function _expandIndexMap(
} else {
item['@type'] = [key];
}
} else if(_isValue(item) &&
!['@language', '@type', '@index'].includes(indexKey)) {
throw new JsonLdError(
'Invalid JSON-LD syntax; Attempt to add illegal key to value ' +
`object: "${indexKey}".`,
'jsonld.SyntaxError',
{code: 'invalid value object', value: item});
} else if(propertyIndex) {
// index is a property to be expanded, and values interpreted for that
// property
if(expandedKey !== '@none') {
// expand key as a value
_addValue(item, propertyIndex, expandedKey, {
propertyIsArray: true,
prependValue: true
});
}
} else if(expandedKey !== '@none' && !(indexKey in item)) {
item[indexKey] = key;
}
Expand Down
15 changes: 14 additions & 1 deletion lib/util.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,8 @@ api.hasValue = (subject, property, value) => {
* an array (lists) (default: false).
* [allowDuplicate] true to allow duplicates, false not to (uses a
* simple shallow comparison of subject ID or value) (default: true).
* [prependValue] false to prepend value to any existing values.
* (default: false)
*/
api.addValue = (subject, property, value, options) => {
options = options || {};
Expand All @@ -246,6 +248,9 @@ api.addValue = (subject, property, value, options) => {
if(!('allowDuplicate' in options)) {
options.allowDuplicate = true;
}
if(!('prependValue' in options)) {
options.prependValue = false;
}

if(options.valueIsArray) {
subject[property] = value;
Expand All @@ -254,6 +259,10 @@ api.addValue = (subject, property, value, options) => {
!subject.hasOwnProperty(property)) {
subject[property] = [];
}
if(options.prependValue) {
value = value.concat(subject[property]);
subject[property] = [];
}
for(let i = 0; i < value.length; ++i) {
api.addValue(subject, property, value[i], options);
}
Expand All @@ -270,7 +279,11 @@ api.addValue = (subject, property, value, options) => {

// add new value
if(!hasValue) {
subject[property].push(value);
if(options.prependValue) {
subject[property].unshift(value);
} else {
subject[property].push(value);
}
}
} else {
// add new value as set or single value
Expand Down
31 changes: 2 additions & 29 deletions tests/test-common.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,8 @@ const TEST_TYPES = {
/compact-manifest.jsonld#ttn01$/,
/compact-manifest.jsonld#ttn02$/,
/compact-manifest.jsonld#ttn03$/,
// property-valued indexes
/compact-manifest.jsonld#tpi01$/,
/compact-manifest.jsonld#tpi02$/,
/compact-manifest.jsonld#tpi03$/,
/compact-manifest.jsonld#tpi04$/,
/compact-manifest.jsonld#tpi05$/,
/compact-manifest.jsonld#tpi06$/,
// IRI confusion
/compact-manifest.jsonld#te002$/,
// @propogate
/compact-manifest.jsonld#tc026$/,
/compact-manifest.jsonld#tc027$/,
Expand All @@ -55,10 +50,6 @@ const TEST_TYPES = {
/compact-manifest.jsonld#tin03$/,
/compact-manifest.jsonld#tin04$/,
/compact-manifest.jsonld#tin05$/,
// index on @type
/compact-manifest.jsonld#tm020$/,
/compact-manifest.jsonld#tm021$/,
/compact-manifest.jsonld#tm022$/,
// context values
/compact-manifest.jsonld#ts001$/,
/compact-manifest.jsonld#ts002$/,
Expand Down Expand Up @@ -142,18 +133,6 @@ const TEST_TYPES = {
/expand-manifest.jsonld#thc05$/,
// @type: @none
/expand-manifest.jsonld#ttn02$/,
// property index maps
/expand-manifest.jsonld#tpi01$/,
/expand-manifest.jsonld#tpi02$/,
/expand-manifest.jsonld#tpi03$/,
/expand-manifest.jsonld#tpi04$/,
/expand-manifest.jsonld#tpi05$/,
/expand-manifest.jsonld#tpi06$/,
/expand-manifest.jsonld#tpi07$/,
/expand-manifest.jsonld#tpi08$/,
/expand-manifest.jsonld#tpi09$/,
/expand-manifest.jsonld#tpi10$/,
/expand-manifest.jsonld#tpi11$/,
// misc
/expand-manifest.jsonld#te043$/,
/expand-manifest.jsonld#te044$/,
Expand Down Expand Up @@ -203,9 +182,6 @@ const TEST_TYPES = {
/expand-manifest.jsonld#tin07$/,
/expand-manifest.jsonld#tin08$/,
/expand-manifest.jsonld#tin09$/,
// index on @type
/expand-manifest.jsonld#tm017$/,
/expand-manifest.jsonld#tm020$/,
// @nest
/expand-manifest.jsonld#tn008$/,
// keywords
Expand Down Expand Up @@ -473,9 +449,6 @@ const TEST_TYPES = {
/toRdf-manifest.jsonld#tin04$/,
/toRdf-manifest.jsonld#tin05$/,
/toRdf-manifest.jsonld#tin06$/,
// index on @type
/toRdf-manifest.jsonld#tm017$/,
/toRdf-manifest.jsonld#tm020$/,
// @next
/toRdf-manifest.jsonld#tn008$/,
// keywords
Expand Down