context bait and switch

[magyarblip]

I was playing with an IOT scenario where a device out in the field is using an older version of a context, and the consuming system has a newer context with the same reference URL. Parking for a second that this would not be the best way to design this sort of thing, i wanted to see if the result degraded predictably.

import { encode, decode } from '@digitalbazaar/cborld';
import { JsonLdDocumentLoader } from 'jsonld-document-loader';

function loadContext(ctx_url, ctx) {
  const loader = new JsonLdDocumentLoader();
  loader.addStatic(ctx_url, ctx)
  return loader.build();
}

const CONTEXT_URL = 'https://example.com/my-context/v1';
const CONTEXT = {
  '@context': {
      ex: 'https://example.com/my-context#',
      type: '@type',
      Example: 'ex:Example',
      label: 'ex:label',
      content: 'ex:content'
  }
};

const myDocument = {
  '@context': CONTEXT_URL,
  type: 'Example',
  label: 'CBOR-LD',
  content: 'CBOR-LD is awesome!'
};

const myLoader = loadContext(CONTEXT_URL, CONTEXT);
const myCborldBytes = await encode({ jsonldDocument: myDocument, documentLoader: myLoader });

// This works as expected, since we are using the same context
const jsonOutput = await decode({ cborldBytes: myCborldBytes, documentLoader: myLoader });
console.log('Using the same context everything is copacetic:');
console.log(jsonOutput);


// Change content to tcontent 
const ALT_CONTEXT1 = {
  '@context': {
      ex: 'https://example.com/my-context#',
      type: '@type',
      Example: 'ex:Example',
      label: 'ex:label',
      tcontent: 'ex:content'
  }
};
const altLoader1 = loadContext(CONTEXT_URL, ALT_CONTEXT1);
const jsonOutput1 = await decode({ cborldBytes: myCborldBytes, documentLoader: altLoader1 });
console.log('Substituting tcontent for content also works:');
console.log(jsonOutput1);

/* This works as expected
{
'@context': 'https://example.com/my-context/v1',
ex: 'CBOR-LD is awesome!',
tcontent: 'CBOR-LD',
type: 'Example'
}
*/

// Change content to ucontent 
const ALT_CONTEXT2 = {
  '@context': {
      ex: 'https://example.com/my-context#',
      type: '@type',
      Example: 'ex:Example',
      label: 'ex:label',
      ucontent: 'ex:content'
  }
};
const altLoader2 = loadContext(CONTEXT_URL, ALT_CONTEXT2);
const jsonOutput2 = await decode({ cborldBytes: myCborldBytes, documentLoader: altLoader2 });
console.log('Substituting ucontent for content mangles the results:');
console.log(jsonOutput2);

/* This gets seriously mangled
{
'@context': 'https://example.com/my-context/v1',
ex: 'CBOR-LD is awesome!',
type: 'CBOR-LD',
ucontent: 100
}
*/

The names tcontent and ucontent are significant: they are alphabetically on either side of type. I would not expect the behavior to change because of that, but it apparently does.

I haven't looked far enough into the code to understand why it's doing this.

[BigBlueHat]

My guess is that the "updated"/changed context meaning isn't actually happening--which is why ucontent is coming back as a numeric value rather than it's actual value. If you switch the order of the two tests (test with tcontent second), the result is likely the same--i.e. the second context "loading" is likely ineffective--and honestly should probably throw an error as changing a context meaning mid-run would be a security issue afaict.

[BigBlueHat]

So...it's not the ordering...exactly. Digging in a bit more, I noticed that your "works as expected" example is actually incorrect:

/* This works as expected */
{
'@context': 'https://example.com/my-context/v1',
ex: 'CBOR-LD is awesome!',
tcontent: 'CBOR-LD',
type: 'Example'
}

The ex should actually be content and tcontent should actually be label...

Consequently, I took out the ex prefix and tested with absolute URLs...but weirdness persists.

Digging a bit further, there does seem to be something related to lexical ordering.

If you set ALT_CONTEXT1 to the following...

const ALT_CONTEXT1 = {
  '@context': {
      type: '@type',
      Example: 'https://example.com/my-context#Example',
      label: 'https://example.com/my-context#label',
      tcontent: 'https://example.com/my-context#content'
  }
};

Results in...

{
  '@context': 'https://example.com/my-context/v1',
  label: 'CBOR-LD is awesome!',
  tcontent: 'CBOR-LD',
  type: 'Example'
}

Note: label and tcontent are now swapped...

And...

const ALT_CONTEXT1 = {
  '@context': {
      type: '@type',
      Example: 'https://example.com/my-context#Example',
      label: 'https://example.com/my-context#label',
      acontent: 'https://example.com/my-context#content'
  }
};

Results in...

{
  '@context': 'https://example.com/my-context/v1',
  acontent: 'CBOR-LD is awesome!',
  label: 'CBOR-LD',
  type: 'Example'
}

So...something's off somewhere... Or...it's working by design, but requires that context contents never change--which means jsonld-document-loader (or any other means of passing in a context MUST prevent them from changing mid-run.

Either way...this is footgun territory for sure...

[BigBlueHat]

So...more digging and I realize you're using an entirely different document loader instance for each run, so you're essentially running a full encode/decode just with a different context value (vs. overwriting or updating and existing context mid-run). Which means changing jsonld-document-loader wouldn't help here...

[dlongley]

When using semantic compression, the context used by a producer and a consumer needs to be stable -- any changes to the context (with the exception of changing scoped-contexts that were not used in a particular payload) will cause a decoding error.

The same deterministic semantic compression dictionary has to be constructed on both ends (producer and consumer) -- this includes sorting the terms used to produce the same CBOR term IDs (integers).

[magyarblip]

it's more the inconsistency in how things degrade, rather than the degradation itself. Really, the scenario here is around devices being deployed for years without firmware upgrades, which is pretty much the norm. If both producer and consumer need to be so joined at the hip, isn't it going to make this very brittle? The CBOR part seems ok, it is more the LD that is inconsistent.

[dlongley]

it's more the inconsistency in how things degrade, rather than the degradation itself. Really, the scenario here is around devices being deployed for years without firmware upgrades, which is pretty much the norm. If both producer and consumer need to be so joined at the hip, isn't it going to make this very brittle? The CBOR part seems ok, it is more the LD that is inconsistent.

It's an interesting area to explore.

I would wonder if these devices are running the JSON-LD API to perform any transformations on the data to begin though. Or, if they are working directly with JSON based on an assumed context with simple validation rules on incoming messages. Only if they are performing expansion would I expect there to be any effect here that would be different from the problem they'd encounter with a mutable context and no CBOR-LD (just JSON-LD). In other words, I don't know that CBOR-LD would contribute anything new to the semantic confusion they'd already be experiencing.

Possible exceptions could be around "append only" contexts, whereby only new terms are added and these would be presumably "ignorable" by older devices. This can be challenging to get right in practice though -- with developers wanting to slide in "fixes" and "minor adjustments" -- that might actually cause trouble in devices that they don't even know about, based on decisions made by other disconnected developers who made different assumptions.

So, all that being said, I'd recommend the use of immutable contexts for these scenarios; an expectation of version-locked contexts would go along with the essentially locked interpretation / data expectations on the devices that aren't undergoing any firmware upgrades. With a versioning scheme / announcement of expected versions (or expected contexts) in the protocol messages / discovery, I'd expect the problems could be avoided both with JSON-LD and, by extension, CBOR-LD. Mutable contexts simply tend to cause trouble for these long-lived deployment scenarios -- even if only JSON-LD is being used (and, really, I'd expect similar challenges for any other technology used in those environments).